Make the agent record proof a human will actually watch

For months our end-to-end suite, run with RECORD=1, had been dropping a feature-*.webm into the CI artifacts bucket every time a test touched a real feature: a login flow, a file upload, the settings toggle nobody trusts. I could have recited the exact artifact path from memory. I could not have told you the last time a human opened one of those files. (My guess: never, not once, not since the week we wrote the recorder and admired it.)

A recording on disk that no one watches is not evidence. It is storage. The fix we shipped, attach-demos.sh, walks the test output after a RECORD=1 run, collects every feature-*.webm, and posts them straight into the pull request. The part that took three tries was the URL. If you link a video from an S3 bucket or a signed CDN link, GitHub’s comment renderer shows a gray box and a download prompt, and your reviewer, who has nine other PRs open, never clicks it. Point instead at github.com/<repo>/raw/<branch>/<path>, a raw URL on GitHub’s own domain, and the renderer reads it as same-origin video and plays it inline, in the comment, while the reviewer is scrolling past on the way to the diff.

The same-domain detail is carrying more than it looks. On a private repo there is no public bucket to aim at, and a logged-in reviewer already carries the session cookie that authorizes a raw fetch from their own org’s repo, so no token ever lands in the URL. Ticket #413 confirmed this end to end: a demo from a private repo, autoplaying inline, no second login, no signed link, just the video running the moment the comment scrolled into view.

So here is the rule we turned into a build gate. Every autonomous PR must carry a feature-*.webm that autoplays in the comment, and the per-feature-demo check fails the build when a PR adds a feature and ships no demo of it. Not a warning a tired reviewer waves through. A red X, in the same column as the failing-test X, blocking merge.

Gating on a thing is only fair if producing the thing is free for whoever owns the PR, and increasingly that owner is an agent, not a person. Every app had been carrying its own roughly 50 lines of recorder plumbing: launch the browser with video capture on, name the file by feature, flush it to the directory attach-demos.sh expects, survive the flaky teardown. e2ekit is the shared harness that collapses those 50 lines into one import, so a new app gets recordings the day it gets tests. The videos go up as PR-namespaced release assets, which means the branch’s raw path keeps resolving after the working tree moves on, and two PRs touching the same feature never clobber each other’s recording.

Why spend a whole build gate on this, when we already had the recordings? Because the scarce resource in an agent-authored loop is not the agent’s time, which is close to free, and not disk, which is cheaper. It is the few seconds of human attention that decide whether anyone ever sees what the agent built before it merges. Make watching cost zero and it happens; bury it behind one click and it does not. The autoplay is not a nicety on top of the evidence. It is the difference between evidence and a file.

A passing demo proves one thing: that the happy path rendered for its 30 seconds without throwing. It does not prove the feature is correct, and it cannot. An agent rewarded for producing a watchable demo will get very good at producing a watchable demo, which is a different target from a working feature, and nothing in the gate stops it from recording a smooth, green-looking pass while the real bug sits one code path off-camera, in the case the test never drove. We made the proof impossible to ignore. We did not make it true.