How Removing Docker Taught Me That My Tests Were Lying (And So Was I)

So here is a thing that happened to me, and I want to tell you about it not because it is dramatic (it is not), but because it taught me something about the nature of green checkmarks that I find genuinely unsettling, and I suspect you might too.

We took 31 hand-rolled testcontainers TestMains, the kind of thing that accretes in a codebase the way lint accretes in a dryer, and we collapsed them into one embedded PG17 harness with pgvector compiled in. The suite got faster. It got rootless. It got Docker-free. (If you have ever waited for a Docker daemon to decide whether it feels like existing today, you understand why these are not small wins.) The migration was supposed to be the most boring kind of work there is: same SQL, same fixtures, same assertions, just fewer moving parts underneath. It passed clean on CI. And then it failed on my laptop, which is the sort of sentence that should make any honest engineer’s stomach drop, because it means the universe is about to teach you something you did not ask to learn.

The two culprits were CompetitorPostingHours and CompetitorPostingDow. Both of them bucket a timestamptz by EXTRACT(hour FROM posted_at), and (this is the part that still makes me wince) both carry a comment cheerfully announcing that the bucketing happens in UTC. The comment was decoration. It was a little motivational poster taped to the wall of a building with no foundation. Because a bare EXTRACT(hour ...) on a timestamptz does not resolve in UTC by divine right; it resolves in the session time zone, whatever the connection happened to inherit, and nobody, in the entire history of this code, had ever bothered to pin the session.

Now, why had this never mattered? Here is the lovely, infuriating reason. Under testcontainers the question simply never got asked, because the Docker image we pulled sets its clock to UTC, and so the embedded postgres running inside that container lived in UTC too. The new harness has no image. It launches a postgres binary as a child of the test process, and that child does what children do: it inherits its parent’s environment, including the host time zone. My host is America/Los_Angeles. (I live there. I like it. I do not, in general, expect my zip code to participate in my unit tests.) So the very same query that bucketed a 2 a.m. UTC post into hour 2 on CI bucketed it into hour 18 on my machine, the assertion dutifully compared it against the UTC-shaped fixture, and the whole thing came apart.

Sit with what that means about all the green that came before. For the entire life of those tests, the assertion was verifying that EXTRACT agreed with UTC, while the harness was quietly, invisibly, guaranteeing that the session already was UTC. The test could not possibly have caught a missing AT TIME ZONE 'UTC', because the only environment it had ever run in had pre-supplied, for free, the exact property it claimed to be checking. (It is a bit like an exam where the answer is printed faintly on the back of the question sheet, and you have been congratulating yourself on your scores.) The UTC in the comment was true in CI and unenforced literally everywhere else, and that test would have sailed serenely green through any refactor that broke the time zone, right up until the moment some unlucky soul ran it off a non-UTC clock. The Docker default was load-bearing. Removing Docker is what made the load visible.

And then, while I was still tracing that one, a second test fell over for what looked at first like a totally unrelated reason and turned out, on inspection, to be the identical shape wearing a different costume. (This is the part where I started to feel like the universe was making a point at my expense.)

We have an LED driver that writes to a sysfs node, and the test for its error path simulates a read-only node by doing chmod 0o444 on a temp file, then asserts that the write fails and the driver degrades gracefully. Reasonable! Except the build sandbox uses fakeowner over overlayfs so that tests can run rootless. And under fakeowner, the process is the file’s owner, and an owner can write straight through 0o444. The mode bits, in this world, describe a permission the kernel never actually checks against the uid you appear to have. So the chmod is a no-op with delusions of grandeur. The write succeeded. The error path never fired. And the assertion that the driver gracefully handles a failure passed triumphantly, in the complete absence of any failure to handle. It had been green since the day it was written, which is to say it had never once been a test.

The fix, once I understood the shape, was almost soothing in its obviousness: stop asking the file system to honor a mode bit (a thing the sandbox feels free to reinterpret), and instead point the write at a target the kernel refuses for reasons no sandbox gets a vote on. A directory returns EISDIR on a write no matter who you think you are. A symlink to /dev/full returns ENOSPC the instant the buffer flushes. Both of these produce a real, honest, failed write(2), so the error path actually runs; and both behave identically under overlayfs, under Docker, and on bare metal, precisely because none of them lean on the uid-versus-mode question that fakeowner answers its own way.

Two tests, two completely different layers of the stack, one mechanism humming underneath both. The isolation we bolted on to get hermeticity is, when you squint, just a stack of substitutions: an embedded binary stands in for a container, fakeowner stands in for a root uid, overlayfs stands in for a real mount. Each substitution faithfully preserves almost all of the semantics and quietly relocates a few. The embedded postgres changed which time zone a bare EXTRACT resolves in. Fakeowner changed whether 0o444 stops the owner. And here is the genuinely sneaky bit: neither of these changes appears in any changelog you would think to read during a migration, because from the harness’s own point of view nothing changed at all. It still launches postgres. It still creates files. The interface is identical. The defaults shifted out from under it without disturbing a single line of the contract.

Which brings me to the part I do not have an answer for, and I would love it if you did. I have no way to find the next one of these before it bites me. There is no manifest of which OS behaviors a given sandbox passes through faithfully and which it merely approximates; no clean diff between the syscall semantics of overlayfs-plus-fakeowner and the production kernel. The divergences I currently know about, I know about for exactly one reason: a test changed color when I physically moved it somewhere new. The ones I do not know about are, right now, sitting inside tests that are green in both environments, passing for reasons that have nothing whatsoever to do with the code they are nominally testing. (Some of them, statistically, are probably load-bearing in ways I will discover at the worst possible moment.) I found these two entirely by accident: one because my laptop has the temerity to live in California, and one because a write that should have failed simply declined to. I do not know how many more are waiting, patient and verdant, on a default I have not yet happened to violate.