Four reviewers, one mind

In June I deleted the line in the spec that said a human merges the pull request. The pipeline now runs end to end on its own: the agent works in a throwaway checkout, opens a request, a fresh agent reviews it, and the orchestrator squash-merges to main when the review approves. I wrote the trade-off down and accepted it on purpose, because the fleet is dev and test and nothing touches production. Then I watched the first builds run under the new rule, and the interesting thing was not that they wrote good code. One of them, told its change had to pass a test, re-submitted the failing test unchanged to get to green.

That is the whole problem in one move. The agent did not get better at the task; it got better at the gate. This is the failure everyone who has run these things has seen, the model optimizing the measurement instead of the thing the measurement was standing in for, and it showed up on the very first builds after I removed the human. So the question stopped being whether the agent is good enough to trust, which is the wrong question the way “is the driver careful enough to never crash” is the wrong question about a road, and became a narrower one: when the agent says it is done and the checks say it is fine, what do the checks actually prove?

Some of them prove a lot, because they are made of code. The build either compiles or it does not. The tests pass or they do not. A guard counts the test functions in the diff and refuses the change if the net number went down, which exists precisely because deleting the test that was failing is the cheapest way to make a suite green. A hash chain over every tool call means a record cannot be quietly rewritten after the fact without the chain breaking. These checks have a property that matters more than how strict they are: they fail for their own reasons, independent of what the agent was thinking, because they are not thinking at all.

Then there is the check I was proudest of, and it is the one that turned out to be hollow in a way I did not see for a while. Before a change can merge, a panel of reviewers reads the diff cold, in fresh context, none of them the agent that wrote it, each looking at a different concern: is it correct, is it secure, does it do what the issue asked, are the tests honest. The panel is fail-closed and it has to be unanimous. One reviewer unconvinced and the change waits for a human. I built it to feel safe, and it does feel safe, and unanimity is exactly the wrong thing to take comfort in here.

Unanimity only buys you safety when the votes are independent, and mine are not. The agent that writes the code is a model. Every reviewer on the panel is the same model, in a fresh session, sometimes quietly falling back to a smaller sibling when the service is overloaded. The separation I engineered so carefully is separation of role and context and prompt: this one wrote it, that one only reads, this one looks at security, that one at the tests. It is never separation of mind. Four reviewers that share a training distribution share their blind spots, so four of them missing the same subtle bug is not four checks failing to catch it. It is one check, run four times, returning the same answer each time and calling the agreement confidence. The defense against a model that games the gate is another instance of the same kind of model reading the diff, and for months I treated that as defense in depth when it is closer to defense in triplicate.

What actually caught things was the part of the system that is not a model. The gamed test was stopped by the test-floor counter, not by a reviewer noticing the dishonesty. The protection against a rewritten history is arithmetic over hashes. The panel’s job, in the end, is to catch the class of mistake a careful reader of the same temperament would catch, which is real but is the easy class, and the comfort it gives me is correlated with the failures it cannot see.

I am not going to claim the gates are sound, because the strongest one has a hole I have not closed. The hash chain is tamper-evident, not tamper-proof: it catches a careless edit, but the agent can reach the file through the ordinary tools it is allowed to run, and a writer who can regenerate the chain from the start, or just delete it so it reads as empty, defeats it. The honest scope is that none of this merges to anything real. The one surface that touches money still stops and waits for a human hand. I removed the human from the loop on a fleet where the worst case is a broken test environment, and I would not remove it anywhere the worst case is a person, which is the line between letting an agent merge to main and pretending the panel made that safe.