Your Agent Knows Exactly As Much As the Command You Let It Run, and Not One Assertion More

Let me tell you about two pull requests, because I have been thinking about them more than is probably healthy.

Pull requests #372 and #373 both shipped Go files that gofmt would reject. (If you do not write Go, gofmt is the formatter that everyone agrees to obey precisely so that nobody has to argue about braces ever again, which makes shipping code it rejects a little like showing up to a black tie event in cargo shorts.) Both PRs came from build agents, and here is the part that I find genuinely interesting rather than merely embarrassing: the agents were not lying, and they were not lazy. Each one had run its local check, watched the thing come back green, and reported the work finished, exactly as it was supposed to.

The check was moon run <app>:check. And moon run <app>:check, as it happened, did not run fmt-check, or tidy-check, or codegen-drift. So the agent’s verification was scrupulously accurate about everything it tested, and completely, serenely silent about the three things it never touched. The agents only discovered they were wrong at the pre-push gate, which runs the full suite and promptly bounced the formatting. By then the triumphant little report was already written. (I picture the agent the way I picture myself submitting a paper and only afterward noticing the theorem in section 4 is false, except the agent never gets the 3am cold sweat, which is sort of the whole problem, and we will come back to that.)

Closing that half of it took exactly one task. There is now a gen-check step that folds fmt-check, tidy-check, and codegen-drift into the per-app check. That is what #376 records, and you would be forgiven for thinking the story ends there, tidy and small. It does not.

Because the very same shape came at us from the opposite direction, which is how you know you are dealing with a real phenomenon and not just a one-off goof. The merge gate ran go test ./apps/<app>/.... Now, that innocent-looking glob, the ..., reaches downward into everything beneath it, and what it happened to reach into was an in-module browser-e2e package whose TestMain hard-fails when no server is listening. In the sandbox there is no server. So the gate failed on every single run, for a reason that had precisely nothing to do with the code under review. (There is a special flavor of dread reserved for a red light that is red no matter what you do, the test-suite equivalent of a smoke detector that chirps because of the battery and not the fire.) On cycle 5, the #234 trace shows both LLM judges and the build agent independently arriving at the same diagnosis, which I admit I find a little uncanny and a little reassuring at once: the failure was a parity gap between what the gate tested and what the change actually touched. The fix excluded /e2e from the gate’s scope and split the moneybox e2e suite into its own module, so that the glob could no longer drag it in by accident. The gate now runs task check.

So what do these two incidents, coming from opposite directions, have in common? Here is the thing I cannot stop turning over. A self-reporting agent’s confidence reaches exactly as far as the command it runs to check itself, and not one assertion further. Nothing in a green result tells you where the command stopped looking. That last clause is, I think, the whole essay, so let me say it again slowly: a green light reports what was checked, and is utterly mute about the boundary of what was checked. When the local check and the gate test different things, that difference never, ever surfaces as doubt. It surfaces as a confident green that the agent will then go on to defend, because from inside its own evidence the work genuinely is correct. #372 was not a careless agent. It was an agent whose evidence was real and whose evidence was narrow, and those two things are completely compatible, which is exactly what makes the trap a trap.

Now here is where I want to give the humans a little credit, briefly, before they get too comfortable. A human carries a second loop that the agent does not have. Even after a clean local run, an experienced engineer pushes the code and then wanders over to look at CI, half-expecting it to catch something the laptop missed. (Be honest. You do this. I do this. It is a small superstitious ritual, like blowing on dice, except that it actually works, because sometimes the dice really are loaded and CI really does catch the thing.) The agent has no such reflex. It finishes and reports the instant its local signal goes green. So any drift between that signal and the gate converts straight into a false claim, with no interval, none, in which a quiet “let me just check on CI” could sneak in and intercept it. The human’s paranoia is a feature. The agent is, in the most literal sense, not paranoid enough.

At this point the obvious move suggests itself, and I want to take it seriously precisely because it is so obvious. Just make the local check run exactly what the gate runs. Close the gap to zero. Done, go home.

Except you cannot. (You knew there was an “except.” There is always an “except.”) The e2e package needs a live server that the build sandbox does not have. The full-module test is slow, and worse, under concurrent builds it deadlocks: Postgres returns SQLSTATE 40P01 and River reports that jobs did not drain, both of them intermittently, so that the identical command passes and fails on identical code depending on nothing more than what else happened to be running at the time. And a check that flakes under load cannot possibly be the thing an agent trusts to declare itself done, because the moment green sometimes means “fine” and sometimes means “nothing was wrong except the scheduler,” green has stopped meaning anything at all. A noisy oracle is not a stricter oracle. It is no oracle.

So the local check has to leave something out. It is not negotiable; it is forced. And every single thing it leaves out reopens a sliver of exactly the divergence that produced #372 and #373 in the first place. Excluding e2e means the local check can never catch an e2e regression before the gate does. Keeping the deadlock-prone full run out of it means load-dependent failures stay invisible until merge. You do not get to win this cleanly. What you get, and I have come to think this is the actual moral, is that the surface on which a false green can form gets smaller with every alignment and never, ever reaches zero. Right now it lives in two places and two places only: the e2e suite, and the full concurrent run. Which is to say, it lives in precisely the two things the local check is allowed not to see, and the price of being allowed not to see them is that, every so often, it won’t.