How to stop AI hallucinations in citations and numbers

Chris Rudge, a health law researcher at the University of Sydney, was reading a report the Australian government had published in July 2025 when he found a colleague's name on work she had never written. The report was Deloitte's review of the framework and IT system that penalize job seekers in the welfare system when they miss obligations, and its references cited reports by Lisa Burton Crawford, a real professor at the university's law school, that do not exist. It also quoted paragraphs 25 and 26 of a Federal Court judgment by Justice Jennifer Davies in the robodebt case Amato v Commonwealth, a judgment with no such paragraphs.

The department posted a corrected version in October 2025. On page 58 of its 273 pages, it now said Deloitte had used "a generative AI large language model (Azure OpenAI GPT-4o) based tool chain". Of the 141 sources in the original reference list, 127 were left. Deloitte agreed to repay the final instalment of a contract worth about A$440,000.

A department spokesperson said Deloitte "has confirmed some footnotes and references were incorrect" and that the substance of the review was retained, with no change to its recommendations. I can see why a reader would accept that, since footnotes feel like the trim on the work. Rudge pointed at what the fix itself showed. Where one invented reference had been, the new version had "like five, six or seven or eight in their place," he told the Guardian, which suggests "the original claim made in the body of the report wasn't based on any one particular evidentiary source." The sentence came first, and a source was found for it afterwards. Among the review's own findings, by the Guardian's account, was a lack of "traceability" between the rules of the framework and the legislation behind it.

Why the model guesses the quote

A language model writes in that same order. It produces the likeliest next words, and a citation is just more words. In September 2025, researchers from OpenAI and Georgia Tech argued in Why Language Models Hallucinate that training and grading reward guessing over saying "I don't know", the way an exam with no penalty for wrong answers rewards a student who always answers. Their sharpest case is the fact that appears only once in the training data. If 20% of birthday facts appear exactly once, they write, expect a base model to hallucinate on at least 20% of them. One paragraph of one judgment is that kind of fact, and so is the exact wording of one section of a statute.

I build AI for regulated work, and our own model did this in July 2026. Asked to record when BC tenancy law treats a mailed or posted notice as received, it returned the right day counts and then invented text for section 90 of the Residential Tenancy Act, a seven-day mail rule and a regulation that changed it, none of which exist. Our grading step caught it before anything shipped.

Catching it once is not a design, so we changed where the value comes from. In Perch, the assistant we are building for BC property managers, a legal value such as a deposit limit or a notice date comes from a small typed lookup. It returns the value with its section and official link, worked out by code from a stored copy of the law that carries the date we retrieved it. The model gets an id for that fact and cites the id. Code puts the value and the quoted section on the page. A scheduled check compares our copy with BC Laws, and when a section has changed or cannot be read, the lookup returns "This section is being re-checked against BC Laws; Perch does not supply its text." instead of a guess. If the model writes a number into its own sentence anyway, a plain check compares it with the passage it cites and marks it red when it contradicts the passage, yellow when the number is not there.

This does not make the model right about which rule applies. It can still reach for the wrong section, and a person still reads the draft before anything is sent. What it removes is the error Rudge found: a value or a quote that exists only in the model's output, with a citation that makes it look checked. I think that is worse than no answer at all, because the citation is what tells a busy reviewer they can stop checking.

Take one AI workflow you run today and print a real output from it. Circle every number, date, case name, section and quoted sentence. For each one, ask where it came from: did code read it from a source you can open, or did the model type it? The ones the model typed are the next thing to move into a lookup.