Pointing at code you can't see

The first version of the pointing channel stored one note per session, and that was a bug before it was a feature. The idea was simple: while an agent drives a shared browser, I can point at something on the page and the gesture rides along into the agent’s next turn, so I can say “look here” by pointing instead of describing. But the implementation kept a single slot for the pending note, and writing a new one overwrote the old. So if I pointed at a button and then grabbed the controls to change the page in the same window of time, the second action silently destroyed the first, and the agent only ever saw whichever I did last. A reviewer caught it. The fix was to stop pretending “look here” and “I changed the page” were the same message: two intents, two slots, joined when the turn consumes them.

I tell that story first because it corrects the romantic version of co-presence before I can be tempted to sell it. Sharing a live view with an agent is not telepathy. It is a channel, with the same concurrency hazards as any channel, and the first thing I learned building it was that two human intents will race into one agent turn and clobber each other if you let them. The appeal is real: you see what the agent sees, you can point at the exact element instead of translating it into words, and a click carries information that a sentence fumbles. But pointing at a screen only works when there is a screen, and that turns out to be the narrow case, not the general one.

Most of what an agent does has no screen. It edits files, it calls interfaces that return no picture, it writes lines into a log. There is nothing to point at, and the part of supervision I actually wanted, reach in and say “this, here, not that,” has no surface to land on. For a while I treated the shared browser as the whole story and quietly accepted that it only covered the work that happened to be visual, which is a thin slice of what the agent spends its time on.

The thing that broke that open is small and a little ridiculous. In development, a build pass stamps every element on the page with the file, line, and column that produced it. The mark is invisible, it is stripped from production, but it is sitting on every rendered node. So when I point at a pixel, the gesture can resolve past the element to the line of source that rendered it, and the note the agent receives is not “the user pointed near coordinate 612, 240,” it is “edit this exact file at this line.” The screen stopped being a thing to look at. It became a pointer into the code the agent edits, which is the invisible work I could not reach a minute ago.

Once I had seen it, the pattern generalized past this one trick. The reason pointing at a page can reach the source is that the page is a projection of the source, and the projection was made to carry a back-reference to the thing it projects. That is the move. To build co-presence over invisible work, you do not need the work to become visible. You need it to have a visible projection that points back at itself. A rendered page that knows its own source lines is one instance. The same shape would let you point at a row in a chart and land on the query that produced it, or at a line in a log and land on the call that emitted it, as long as the projection was built to carry the address of its origin. The hard question was “how do you point at work that has no screen,” and the answer is not to find a screen, it is to make the artifact project one that remembers where it came from.

I should be precise about how narrow my one instance is, because the pattern is general and the implementation is not. The source-mapping trick only works in development, only for one kind of web app, and does nothing for a production page or anything that is not rendered from that kind of source. The pointing itself is turn-gated: the gesture does not interrupt anything, it batches onto the agent’s next turn, so between turns my pointing is inert, and grabbing the controls never yanks a wrong action out of the agent’s hands mid-flight, it only colors what the agent does next. And the shared view is a scarce resource: each live browser costs hundreds of megabytes, so by default only one runs at a time. What I have is one worked example of a pattern I am fairly sure is the right one, which is a different thing from having the pattern. The screen was never the point. The back-reference was.