April 2026
Streaming an LLM turn is not streaming text
Reconnecting to a long agent turn corrupted its answers, and the fix was to stop modeling the turn as a character stream and treat it as a structured object that survives the connection.
The first report from agentbox round 2 was that the assistant had started reading its own private reasoning aloud to the customer. A support user reconnected to a turn that had been running for about ninety seconds, and the reply that rendered had three sentences of planning, “I should check the refund window before committing,” wedged into the middle of the customer-facing answer. Nobody had touched the prompt. The reconnect did it.
We catalogued sixteen of these, A1 through A16, and they rhymed. A4 was that reasoning leak. A5 was two separate answers in one turn collapsing into a single run-on paragraph. A13 was a tool result that arrived with no call attached to it. A1 was a transcript file that stopped parsing partway down. Each looked like its own bug with its own owner. Each came from one place: code that took a turn to be a sequence of characters and concatenated them in arrival order.
A chat turn is not a string. It is a resumable, multi-block structured object, and most of these reliability bugs come from flattening it into a character stream the instant it lands.
Start with the leak. The stream coming off the model is two channels braided together, thinking deltas and prose deltas, and every event carries a tag saying which it is. Join by arrival order and the planning text drops wherever it happened to arrive, which after a reconnect is the middle of the answer. The fix has nothing to do with byte position. You read the tag on each delta and route thinking and prose into separate buffers, so the channel decides placement instead of timing (A4).
A5 is the same mistake one level up. A single turn carries several content blocks in sequence: a prose block, then a tool_use block, then a second prose block once the tool returns. The string view sees one undifferentiated stream and welds the two prose blocks into one paragraph, deleting the boundary where a tool ran. Anything downstream that counts blocks to decide what to render now sees one block where there were three.
The blocks are not independent of each other either. A tool_result block is valid only when its matching tool_use block is present in the same turn, keyed by id. On reconnect this turns sharp. If the tool_use sat in the dropped prefix and the tool_result is in the replayed suffix, you have an orphan, and shipping it to the model gets the entire turn rejected. The result has to be discarded when its call is absent, and only a representation that still knows about blocks and ids can even run that check (A13).
All of this rides on replay. Every event off the subprocess carries a monotonic seq. The turn buffers server-side and the socket is disposable; when the client reconnects it sends Last-Event-ID and the server replays everything after that seq. A connection dying is not rare. It happens several times inside a long turn, and the buffer is the thing that lives through it. So reconnect is not an error path you handle once at the edge. It is the ordinary shape of a turn, and every structure above has to survive being cut at an arbitrary seq and stitched back.
Persistence has its own version of the problem. Each turn lands in a JSONL append-log, one event per line. After a reconnect there can be two writers, the original subprocess and the resumed one, both holding the append handle and interleaving half-lines until the file no longer parses. A single-writer guard, exactly one holder of the handle per turn, is the whole difference between a transcript and a pile of bytes (A1).
Then there is the case the structure cannot close. A reconnect can leave two strands for the same turn: the buffered one from before the cut, and a fresh continuation the subprocess emits after. Both claim to be the final. A6 keeps the longer strand, betting that more tokens means it ran further before dying. That holds right up until the strands diverge in content and not just length. Picture the shorter one fixing a dollar amount the longer one got wrong, then crashing before it could write another token. Longer is not righter there. And once the subprocess is gone, no third copy of the turn exists to check against, so the ground truth that would say which strand is correct died with the process that held it.