The contaminated corpus

Our own research corpus is assembled by machine, and in February we took a spade to it. The cleanup ledger records 13,880 lines removed across 197 files, 84 of them orphaned backup files left behind by an earlier automation that had failed partway through its own repair. The number that matters is smaller and worse. Twenty-three files carried what the record calls cross-topic concatenation, meaning an ingestion pipeline had joined two unrelated query results into one document with nothing between them. Research on human connection ran to its natural close, and then, with no heading, no separator, not so much as a blank line’s hesitation, an analysis of an unrelated market began in the same file under the same title.

Nothing catches this. The frontmatter is valid. The markdown parses. The headings nest correctly, the links resolve, the file opens cleanly in every tool that was ever going to open it, and a search returns it as one document because that is precisely what it is. A concatenation of two unrelated documents is well formed at every level a machine checks, so the only instrument that detects the defect is a reader who already knows what both halves were supposed to be about, and assembly at machine scale is exactly the condition under which no such reader exists.

Then the second failure, which is the one worth publishing. The first audit of the damage estimated that roughly 37 percent of 451 files might be contaminated, something near 170 of them, and the remediation plan was sized against that figure. The second audit found 23, which the record puts at six to eight percent. The overcount was not carelessness and it was not caution either. Around 350 files in that corpus carry a deliberate two-tier shape, a short summary above a fuller treatment of the same subject, and the first pass could not distinguish two passages worded differently about one topic from two passages about two topics. That is the same faculty the ingestion pipeline had been missing when it made the mess. The instrument that measured the damage was blind in the way the process that caused it was blind, and it got the answer wrong by a factor of five in the direction that made the corpus look worse than it was.

We keep the ledger, wrong first estimate included, because a corpus that cannot describe its own damage is asking to be trusted rather than checked, and trust is not a property anyone can audit. The disciplines that follow from it are unglamorous and they are all about joins. Validate at the seam rather than at the file, since the file is not where assembled knowledge breaks. Require an ingestion step to say which query produced which block, so a document with two parents says so before a person has to notice. And read every automated damage estimate as a measurement taken by an instrument that may share a blind spot with the thing it is measuring, which is a reason to grade the auditor rather than a reason to stop auditing. Every seam we found is now a place the next pass knows to look, and a structure that can point at its own bad joins gets sounder each time it is doubted.