Close enough, and too close

I built a grader for synthetic data, and the first thing I made it do was distrust its own headline number. A generated table can score well above ninety on average fidelity and still hold one column that is pure noise, because the average drowns it: ninety-some columns reproduce the real distribution beautifully, carry the mean, and the one column the table existed to preserve has quietly collapsed to garbage. An average is a machine for hiding its worst input. Watch only the average and you are training the synthesizer to do exactly that, to dress up the easy columns while the hard one rots, because the easy columns are where the cheap points are.

So the grader does not report the average. Every fidelity check runs per column, the scores get sorted, and the grader looks at the worst tenth. That bottom slice has to clear the floor on its own, and if it does not the whole run scores zero, no matter how good the mean looks. Not a deduction, not a penalty a strong showing elsewhere can buy back. Zero. The question it asks is not “is this good on the whole,” which any collapsed column can pass, but “is the worst thing in here good enough,” which is the only question whose answer survives a real query landing on the worst thing. I proved it works the honest way, with a test that takes a table passing every gate, shoves a single column far out of distribution, and checks that the run now fails. The collapse is invisible to the average and caught by the floor.

Fidelity is the easy half. The half I actually care about is privacy, and the thing I did not expect is that it is measured with the same ruler, just read from the other end. To know synthetic data is faithful, I measure how far each generated row sits from the nearest real row: land in the same neighborhoods the real data occupies and you have learned the distribution. To know it is private, I measure the exact same distance: land on top of a real row and you have memorized a person. One nearest-neighbor distance, two opposite verdicts. Close enough is the proof of quality. Too close is the proof of a leak.

That symmetry is why I stopped thinking of quality and privacy as two goals to trade against each other. They are one axis with a floor under it. A synthetic row has to sit close enough to the real data to be useful and never close enough to be a copy, and the grader enforces both ends with the same nearest-neighbor computation: a privacy gate fails the run if the synthetic rows crowd the real ones tighter than real rows already crowd each other, and a memorization gate fails it if too few of the rows are genuinely new. The clean proof of the whole stance showed up in how long I let the synthesizer train. The sweet spot is about six minutes. Train it longer and fidelity keeps creeping up while the model starts memorizing, and the privacy gate catches it. More training made the numbers prettier and the data less safe, and the same ruler that rewarded the fidelity is what flagged the leak.

The one real run I have, a genuine billion-row production table, cleared every floor, and the tightest margin in the whole scorecard was the memorization gate, which is exactly the gate I would want to be nervous about. I will not oversell that, because it is one table. The grader is general and the evidence is a single end-to-end validation, and a second and third real table will surface dtype and encoding quirks this one did not. The deeper limit is that every gate catches a failure shape I anticipated: a collapsed column, a distinguishable artifact, a copied row. A leak that is none of those walks straight through, and there is a quieter gap I have to name, because the system is supposed to fail closed: a privacy metric that breaches fails the run, but a privacy metric that could not be computed at all is reported as incomplete and does not block. Fail closed on a breach, fail open on an absence. The honest version of “fail closed” has that asterisk on it, and I would rather write the asterisk down than let the word do work it has not earned.