My Self-Improving Agent Improved Itself Right Into a Coma (and How I Woke It Up)

So here is a situation that I would have found funny if it had happened to someone else. The loop had finished dozens of builds and had written, for its trouble, exactly nothing. Each run would end, the reflection step would dutifully fire, and back it would come, empty-handed, like a graduate student who read the whole paper and reported that it was “interesting.” The rule, the one rule that governed whether anything got learned at all, was that a lesson could only be derived once a pattern recurred across two or more builds. And AppNav, the component whose job is to map which file a given build actually touched, simply refused to surface anything until it had watched the same file go wrong twice.

Now, you can already see the trap, but let me say it plainly anyway, because I did not see it for an embarrassingly long time. Our builds were varied enough that the same failure almost never showed up twice in a row. So the loop just sat there, serene, learning nothing, the way a very patient person waiting for lightning to strike the same spot twice will eventually starve. The only way to feed the thing was to re-run the same kind of build on purpose, manufacturing a recurrence, staging the lightning, so that the system would graciously concede it had learned something. (If you have ever rigged a demo, you know the precise flavor of shame I am describing.)

We inverted the rule. And the inversion is the uncomfortable part, the part I want to dwell on, because it runs straight against a reflex that you and I have both been trained to obey. An autonomous build loop should take a lesson from every single build. And the statistical instinct that warns you off generalizing from one sample, the instinct that any honest scientist has tattooed somewhere, is the exact thing that was holding the loop perfectly, beautifully still.

Here is the thing, though. That reflex is correct in the world it was trained in. If you have a noisy metric and a model you are fitting, then one data point is an open invitation to overfit, and waiting for the second occurrence is cheap insurance against mistaking randomness for signal. I would defend that to anyone. But a build trace is not that kind of data point, and conflating the two is what cost me all those empty reflections. A single trace already records where the build got blocked (its gate blocks) and the work it did that led nowhere (its waste). That is not one outcome sampled from some distribution we are squinting at. It is a complete account of one run, with the failure and all the little decisions clustered around the failure attached right there to it. Setting minBuilds=1 was, in the end, just the admission that one trace is rich enough to reflect on, because the question reflection is asking is not “did this recur” but “what did this run spend itself on.” Those are wildly different questions, and only the first one needs a second sample.

But the recurrence gate, it turned out, was only half of what had paralyzed the loop. (There is always a second half. I have come to believe this is a law.) The other half was that the earlier loop never actually wrote lessons. It selected them, from a table of advice that some human, possibly me on a more optimistic day, had written ahead of time. Which meant the loop could only ever recognize problems that someone had already anticipated and pre-phrased for it. A failure mode that nobody had written down in advance was simply invisible to it, no matter how many times it happened. The loop was, in a sense, doomed to be exactly as clever as its author had been on a Tuesday months earlier, and not one insight cleverer.

So we tore out the table and replaced it with internal/reflect, which is a read-only headless run of claude that reads the real build history and writes the lesson itself, in its own words, grounded in what that specific build actually did. Read-only by construction, so the reflection pass can inspect the entire history and alter none of it (a property I find I sleep better for). And the lessons it produces get auto-applied after every build, with no manual –apply step sitting in between, which is the thing, finally, that closes the loop. Build, reflect over the trace, apply the lesson, build again. Round and round.

The first two lessons the loop authored this way came out of builds #86 and #48. They were the first pieces of advice in the entire system that had been written from observed behavior rather than retrieved from the pre-written table. And they were applied to the very next build without anyone, me included, reading them first to check whether they happened to be right.

Which brings us, at last, to the clause with no answer attached, the one I have been circling this whole time and would rather not land on. Once minBuilds is 1, every reflection is a generalization from a single trace, and the loop now writes those generalizations about its own behavior and applies them automatically before the next run even starts. A correct lesson and a wrong one arrive through the identical door, on the identical trace count, with the identical auto-apply, wearing, as far as the system can tell, the identical clothes. Nothing in there retires a lesson the loop got wrong about itself. A spurious lesson, written from one trace where the loop misread its own gate blocks, just persists and goes on to shape build #87, and the only evidence that could ever contradict it is a later trace, which the same biased loop is now the one reflecting over. (You see the problem. The witness and the defendant are the same agent.) The material I have names the worry, calls it spurious, and then stops, politely, right there. There is no mechanism yet that proves a self-authored lesson true before the very build that runs on top of it. I would love to tell you I have one. I do not, and I think pretending otherwise is the one failure mode worth pre-writing.