Twelve of twelve came back corrected
When we built our own newsroom we did the thing everybody says they do and almost nobody does. Before the first twelve articles went out, each one was handed to a reviewer whose only job was to re-fetch the source and check every number, name and date against it. Not a proofread. A hostile read, looking for the failure modes we already knew a language model has.
Twelve of twelve came back corrected. None passed untouched.
Three of the findings are worth naming, because they are the three shapes this goes wrong in.
A fabricated quote. Not a distorted one. A sentence attributed to a publication that never printed it, in an article that was otherwise accurate and well sourced. This is the failure everybody knows about, and knowing about it does not help, because it arrives wearing the same clothes as a real quote.
A truncated quote that changed the meaning. The words were real and the attribution was correct. The sentence had been cut at the point that reversed what the speaker was arguing. This one is worse than the fabrication, because every automatic check passes: the string is genuinely in the source.
A regulatory date that was wrong and mattered. Not a typo in a sentence nobody reads. A date somebody could have planned around, on exactly the point the source had warned was widely misread.
We publish this for two reasons. The first is that it is the honest reason the human gate exists. If the drafts were reliably right the gate would be a formality, and somebody would eventually remove it to go faster. They are not, so it is not, and nobody will. The second is that a vendor telling you their AI does not hallucinate is telling you they have not looked. We looked, in our own copy, and wrote down what we found. That is the standard we would want applied to us, so we do not soften it into a paragraph about rigorous quality assurance.
What changed afterwards was not the model. It was the format. Every claim links to a source you can open. Every response closes by saying what it does not claim. Every piece is approved by a named person on both language versions before it exists publicly. The errors did not stop happening. They stopped shipping.
Thirty-seven published items later, thirty of them sourced responses, the same review runs on every one. How it works