Image: Vectara

In September, a lawyer in Australia's Federal Circuit and Family Court admitted using generative AI to prepare submissions in an immigration case. According to Damien Charlotin's AI Hallucination Cases database, the court found that six authorities in one set of submissions and five in another were hallucinated, and that other cited authorities did not support what they were cited for. The court had to do the fact-checking itself. It ordered the lawyer to pay AUD 9,600 in costs and named him in the judgment.

That case is one of many. The same month, a federal court in New York fined a lawyer $500 and ordered extra legal education in a case where the database lists ChatGPT as the tool involved. Charlotin's database counted 2,225 court decisions dealing with hallucinated material at the time of this writing, 879 of them involving lawyers and 1,283 involving people representing themselves. It only includes cases where a court or tribunal found or implied that a party relied on hallucinated content, so it is a floor rather than a full count.

So why do chatbots that can write code and pass exams still invent court cases? The research has a surprisingly plain answer.

What a hallucination actually is

OpenAI defines hallucinations as "plausible but false statements generated by language models." The key word is plausible. A typo or a bad calculation is an ordinary error. A hallucination looks right. It has the shape of a real fact, a real citation or a real date, and the model delivers it with the same confidence as everything else.

The research paper behind OpenAI's post, by Adam Tauman Kalai, Ofir Nachum, Santosh Vempala and Edwin Zhang, opens with a neat example. The researchers asked a leading open model for Kalai's birthday, with instructions to answer only if it knew. In three tries it gave three different dates, all wrong. Asked for the title of Kalai's PhD dissertation, ChatGPT, DeepSeek and Llama each produced a confident, specific and incorrect answer.

The paper also separates two kinds of mistakes. Some hallucinations contradict the prompt itself, like miscounting the letters in a word. Others contradict the real world or the training data, like a court case that never existed. The legal cases above are mostly the second kind.

Why it happens

The paper's argument comes in two parts.

The first is about pretraining, the stage where a model learns by predicting the next word across huge amounts of text. That text carries no labels saying which statements are true and which are false. OpenAI points out that spelling and grammar follow consistent patterns, so models stop making those errors as they scale. Arbitrary facts that show up rarely, like a specific person's birthday, cannot be worked out from patterns at all. The paper puts a number on this. If 20% of birthday facts appear exactly once in the training data, base models should be expected to get at least 20% of birthday questions wrong. The authors say errors would happen even if the training data were perfectly clean.

The second part explains why later training does not fix this. Models are graded on benchmarks that mostly score answers as right or wrong. A blank or an "I don't know" earns zero, the same as a wrong answer. OpenAI compares it to a multiple choice test where guessing beats leaving the question blank. If a model guesses a birthday, it has a 1 in 365 chance of being right. Saying it does not know guarantees nothing. Over thousands of questions, the guesser looks better on the scoreboard.

The paper describes models as always being in "test-taking" mode. It argues that a model which honestly signals uncertainty will lose to an otherwise identical model that always guesses, under the scoring used by most benchmarks.

How often chatbots make things up

There is no single hallucination rate, because it depends heavily on the task. The best known public tracker is Vectara’s hallucination leaderboard, last updated on October 22, 2026. It gives each model over 7,700 articles to summarize, with instructions to use only facts from the article, then uses Vectara's own detection model to check whether each summary sticks to the source.

On that test, the lowest rate belonged to antgroup's finix_s1_32b at 1.8%, followed by OpenAI's GPT-5.4 nano at 3.1% and Google's Gemini 2.5 Flash Lite at 3.3%. The highest rates on the board reached into the 20s, with OpenAI's o3-pro at 23.3% and Mistral's Ministral 3 3B at 24.2%.

These numbers need context. Vectara says plainly that the leaderboard measures factual consistency when summarizing a document the model has been handed. It does not measure overall factual accuracy, and in its FAQ Vectara agrees that the test does not capture every way a model can hallucinate. A model that sticks to a document it is given may still invent facts when asked an open question from memory, which is exactly the situation in most of the court cases. The checker is also an AI model, not a human reviewer.

Why newer models have not solved it

It would be natural to assume bigger, newer models hallucinate less. The published numbers do not support that as a rule.

On Vectara's board, several of the newest flagship models sit well below the leaders. GPT-6 Sol scores 6.5% and GPT-6 Astra 8.7%. Claude Opus 5.5 is at 9.9% and Gemini 4 Argon at 12.1%. Meanwhile, GPT-4.1 from April 2025 scores 5.6%. On this one task, newer and smarter does not reliably mean more faithful.

OpenAI's own figures show why. Its post cites the SimpleQA results from the GPT-5 system card. The older o4-mini answered 24% of questions correctly and got 75% wrong, abstaining just 1% of the time. The newer gpt-5-thinking-mini was right slightly less often, at 22%, but abstained on 52% of questions and got only 26% wrong. Judged on accuracy alone, the older model looks better. Judged on how often it confidently made things up, it was far worse.

OpenAI says GPT-5 has significantly fewer hallucinations than earlier models, especially when reasoning, but that they still occur. It also argues that accuracy will never reach 100%, because some questions simply cannot be answered from the information available.

What might actually fix it

The paper's main proposal is to change how models are scored, not to invent yet another hallucination test. OpenAI says the widely used accuracy benchmarks need to penalize confident errors more than uncertainty and give partial credit for honest abstention. The paper suggests benchmarks state a confidence target in their instructions, telling the model to answer only if it is more than, say, 75% confident, because wrong answers cost points while "I don't know" costs nothing. The authors note that some exams, including India's JEE, NEET and GATE, already use negative marking for similar reasons.

The other common fix is grounding. That means giving the model search results or documents to work from and asking it to cite them. Both the paper and Vectara acknowledge this kind of retrieval can reduce hallucinations, and it is how most chatbots now answer questions about current events.

It is not a cure. The paper argues that grading still rewards guessing whenever a search fails to turn up a confident answer, and that search does not help with errors like miscounting. Citations have their own problems too. As we reported earlier this month, studies by the BBC, the European Broadcasting Union and Columbia's Tow Center found chatbot citations frequently fail to support the answers they are attached to.

A checklist for using chatbots without getting burned

Until the incentives change, here is what I would recommend for anyone relying on a chatbot for facts.

• Treat names, dates, numbers, quotes and citations as the riskiest parts of any answer, since these are the rare and specific facts models are worst at.
• Open every source a chatbot cites and check that it exists and says what the chatbot claims.
• Paste the document into the chat and ask the model to work only from it when accuracy matters, rather than asking from memory.
• Tell the chatbot it is fine to say it does not know, and take that answer seriously when you get it.
• Ask the same factual question twice. Different answers to the same question are a strong warning sign, as the birthday example shows.
• Never file, publish or send anything with legal, medical or financial weight without checking it against an original source yourself.

"I don't know" is a feature

The research points to an uncomfortable conclusion. Chatbots make things up partly because they have been rewarded for it. A model that guesses looks smarter on a leaderboard than one that admits its limits, even when the guesser is wrong far more often.

OpenAI's paper argues hallucinations are not inevitable, since a model can always choose to abstain. Getting there means valuing that choice. The next time a chatbot tells you it is not sure, that may be the most trustworthy thing it says all day.