Inducing the schema instead of supplying it

Point a program at a stack of documents and tell it to turn them into a database you can query, and the hard part is not filling the tables. It is deciding what the tables should be. Most systems skip that part by assuming it away: a person sorts the documents into kinds, designs a table for each kind, and the machine pours the contents in. The schema is treated as the given and the pouring as the work. That is backwards. Designing the schema, what the columns are, which values are the same thing as which, how the tables join, from nothing but the documents themselves, is the part that is actually hard, and the part the assume-it framing quietly defines away.

So I built a system that designs the schema instead of being handed one, and the moment that taught me what it was really doing was a refusal. It had pulled a value off a page that fit none of the columns it had built so far, and it faced the choice the whole thing turns on: open a new column for that value, or drop it into a holding area where stray values sit unpromoted but still queryable. It makes that call by arithmetic, and the arithmetic plainly favored opening the column. It dropped the value into the holding area anyway, because the value had appeared in exactly one document, and one document does not get to add a column to the schema. That is the entire idea in a single decision: a column earns its place only by paying for itself across the whole corpus, and one confident document does not get a vote.

The arithmetic is worth slowing down on, because it is what keeps the system honest. Every value it meets that fits no existing column is a sign the schema is too small right there, and the lazy fix is to add a column for each one, which leaves you with thousands of columns that are empty almost everywhere. The disciplined fix is to make each candidate pay its way: a column costs something to carry and something to store, and that cost has to come in under the cost of just leaving the values in the holding area, all of it counted in bits, across the whole corpus. The value I refused would have shortened the corpus by 2,476 bits had I promoted it, a real saving the system is built to chase, and it said no anyway, because the bits only tell you how much you would save, never whether the column is real. A floor on how many documents a value has to appear in is what enforces the difference: a single long table in one file can want a column all it likes, and it does not get one alone.

A schema that grows has to grow without breaking what it already promised, because answers were given against earlier versions. So each step is a verified migration: old rows carry forward through a recorded map, a new column is added empty and old rows read null with a stated reason rather than a fabricated value, and before the version commits, the checks that passed against the previous schema are re-run against the new one and must still pass. A step that would break a prior commitment is blocked and quarantined automatically. I should be honest about what that check is, though: it proves a migration did not damage what was already accepted, not that the new schema is good. Those are different claims, and treating non-regression as quality is how teams convince themselves a churning schema is improving. The other invariant is that every materialized cell can replay: it carries a derivation back to the source span, document, page, box, character offset, and re-reading the source confirms it. On one audited build that replay succeeded on 85 of 85 columns, including the held-aside tier, and on a blind battery of aggregate questions the answers came back 11 of 15 value-correct with every cell tracing to a real source cell and nothing invented, where the retrieval baselines that could not compute over rows simply made the numbers up. The payoff of inducing real structure is that you can answer by running SQL over it, and a system asked to onboard 9 documents it had never seen built a schema and answered grounded questions in 23.9 seconds, with no configuration and no domain code.

Then there is the rock, and it sits underneath all of it. The lever that would make the schema genuinely parsimonious is merge: recognizing that two columns are the same concept under different names and folding them into one. I built it, and I could not make it sound. An embedding-based proposer merged a count of drivers in alcohol-related crashes into a count of the people those crashes killed, scored it a confident gain, and did it because the two names share the word alcohol, while in the same run it refused a true synonym pair that happened to share no words. Embedding distance ranks name overlap, which is anti-correlated with concept identity, and no threshold I tried separated the wrong twin from the real alias. I tried a model to judge it instead, and on one corpus it was high-recall and then folded two columns with an identical header that were measuring different things, fatal-crash rates and injury-crash rates, because the distinction lived in a row band the column had dropped.

The deepest version of the failure is that even a perfect merger would barely help where it matters. On one corpus of financial filings, 93 percent of the line-item labels appeared in exactly one filing, so there is nothing to align them against, and the 7 percent that recur are already identical strings that need no merging at all. The synonym problem lives entirely in the bespoke tail, one filing each, where unification has no second instance to unify with. I can split a same-shaped pair into two columns soundly, because two paths that disagree about the same source cell are plainly different measurements. Unifying two truly-same concepts that never co-occur is the open part, and I do not have it. The discipline of making a column pay for itself in bits is the part I am sure of. Knowing when two columns are one concept is the part the whole field, me included, is still standing in front of.