AI-generated. Written by an automated system, checked by independent automated gates, and published without human editorial review. How we label and verify.
Short answer: you can reduce AI hallucination with a strict prompt, but you cannot prompt it away. In our content pipeline a “never invent a fact” instruction cut fabrication sharply, yet invented facts never reached zero. What actually stopped them was a separate check that compared every concrete claim against a fixed set of source facts and blocked anything that was not backed.
What a strict prompt does, and what it can’t do
We wrote a strict instruction: never invent a fact; “unknown” beats invented. It worked, up to a point. Fabrications dropped sharply. But no matter how carefully we worded the prompt, invented facts never went away completely.
Which hallucinations survive a good prompt
The ones that got through were never the obvious ones. They were the confident-sounding fills:
- a plausible number,
- a named source,
- a tidy statistic the model supplied because the sentence wanted one.
These are exactly the claims a reader is least likely to question, which is what makes them dangerous.
What worked: three layers, in this order
1. A strict prompt (layer one)
The “never invent, unknown beats invented” instruction. It reduces the problem but does not close it.
2. Explicit grounding
We required every claim in a draft to map to a supplied fact. Drafts came back cleaner on the first try, and the next layer had an easier job. One small detail made a big difference: an inline note after each data claim showing exactly which supplied fact it came from.
3. A blocking check in front of publication
A separate check compared every concrete claim with the fixed set of source facts and blocked anything that was not backed, regardless of how good the prompt was. This is the layer that stopped the fabrications.
The turning point
The real change was in how we thought about it. We stopped treating the truth prompt as the defense and started treating it as the first layer, with the blocking check as the thing we actually trusted. Our rule since then: only trust a check you have watched refuse something.
Don’t drop the prompt: the counter-case
The prompt still earns its place. When the prompt was weaker, the blocking check had to reject far more drafts. Good instructions cut wasted generations, even though they never replace the check.
FAQ
Can you stop AI hallucinations with prompt engineering alone?
No. In our testing a strict prompt cut fabrication sharply but never to zero. The remaining fabrications were stopped by a separate check against a fixed set of source facts.
What kind of hallucinations are hardest to catch?
The confident ones: plausible numbers, named sources and tidy statistics that complete a sentence. Those were the ones that survived a strong prompt.
Is the prompt useless then?
No. A weaker prompt made the blocking check reject far more drafts. The prompt reduces waste; the check provides the guarantee.
How do I make claims easy to check?
Make the grounding explicit: every claim has to point to the supplied fact it came from. That made our drafts cleaner and the check simpler.
The tested, runnable version of the check is in the members’ library.
What research says about reducing hallucinations
Published research points the same way: you can lower the rate, but no single technique removes the problem. The research also helps explain why a prompt alone falls short.
Models are rewarded for guessing
A 2025 paper from OpenAI researchers, Why Language Models Hallucinate, argues that training and evaluation procedures reward guessing over acknowledging uncertainty. Most benchmarks score a wrong answer and an honest “I don’t know” the same way, so a model that guesses looks better on the leaderboard. The authors propose modifying the scoring of existing benchmarks rather than adding more hallucination tests on the side. For anyone building a pipeline, the takeaway is simple. Unless something in your system rewards abstaining, the model will tend to fill the gap.
Grounding in retrieved sources helps
The original retrieval-augmented generation paper (Lewis et al., NeurIPS 2020) reported that models given retrieved documents produced more specific, diverse and factual language than a comparable model working from its own parameters. A follow-up study, Retrieval Augmentation Reduces Hallucination in Conversation, found that standard dialogue models suffer from hallucination and that its best retrieval-augmented models substantially curtail the issue. Curtail is not the same as eliminate, and neither paper claims it is.
Self-checking reduces errors without closing the gap
Meta researchers tested a method called Chain-of-Verification. The model drafts an answer, plans verification questions, answers them independently, then writes a revised response. The paper reports that this decreases hallucinations across a variety of tasks, from list questions to longform text. The key design choice is that the verification answers are produced separately from the draft, so the model is less likely to repeat its own mistakes.
Checking claim by claim
FActScore (Min et al., EMNLP 2023) measures factual precision by breaking long text into atomic facts and computing the percentage of atomic facts supported by a reliable knowledge source. In its human evaluation of generated biographies, ChatGPT only achieves 58% on that measure. The authors also built an automated estimator using retrieval and a language model, reported at less than a 2% error rate against human judgment.
What the vendors and standards bodies recommend
Anthropic’s own guidance on reducing hallucinations recommends letting the model admit uncertainty and asking it to use direct quotes for factual grounding when working with long documents. The same page is candid about the limit: these techniques reduce hallucinations but do not remove them, and it tells developers to always validate critical information. The OWASP Top 10 for LLM Applications lists misinformation as a security risk and recommends cross-checking output against trusted sources and adding tools to automatically validate key outputs, especially in high-stakes settings.
How the research lines up with our findings
Our experience described above fits this picture closely. A strict “unknown beats invented” prompt cut fabrication sharply but never to zero. That matches the vendor guidance: giving the model permission to say it does not know helps, and nobody who publishes on the subject claims it is enough.
The incentive argument in the OpenAI paper also explains which errors survived. We saw the confident fills get through: a plausible number, a named source, a tidy statistic the sentence seemed to want. Those are exactly the guesses a model trained to avoid blank answers would produce.
Our second layer, requiring each claim to map to a supplied fact, is a form of grounding. The retrieval research found grounding reduces hallucination. It did not find that grounding stops it, and neither did we. Grounding made our drafts cleaner and made the next layer easier.
The blocking check is closest to FActScore and to OWASP’s advice on automatic validation. Both treat text as a set of individual claims that are either supported by a trusted source or not. The difference is where the check sits. Research metrics usually score output after the fact. We put the check in front of publication and let it refuse drafts. Chain-of-Verification shows a model can catch some of its own errors, but the decision we came to trust was an external comparison against a fixed fact set, not the model’s own review.
A practical checklist
If you are working out how to stop AI hallucinations from reaching readers, this is the order we would follow, combining our layers with the published guidance:
- Write the strict instruction. Tell the model that “unknown” is an acceptable answer and invented facts are not.
- Supply the facts. Give the model a fixed set of source facts for the task instead of relying on what it remembers.
- Make grounding explicit. Require every data claim to point to the supplied fact it came from, inline.
- Ask for quotes on long sources. For long documents, have the model extract the relevant passages before it answers, as Anthropic’s guidance suggests.
- Split the draft into individual claims. Numbers, names, dates and cited sources are the ones to isolate first.
- Check each claim against the fact set with a separate step, not inside the same generation.
- Block on failure. An unsupported claim stops publication; it does not just add a warning.
- Test the check itself. Feed it a draft with a planted false claim and confirm it refuses. Only trust a check you have watched refuse something.
- Track rejections. If the check starts rejecting more drafts, look at the prompt and the supplied facts before loosening the check.
More questions
Does retrieval-augmented generation solve hallucination?
No. The research on retrieval augmentation in conversation reports that it substantially curtails hallucination, which is a reduction, not a cure. A model can still misread or embellish the documents it was given, so the output still needs checking.
Why do models make things up instead of saying they don’t know?
One explanation comes from the Why Language Models Hallucinate paper: common evaluations reward guessing over acknowledging uncertainty, so models learn that a confident answer scores better than an abstention.
Can a model check its own output?
Partly. Chain-of-Verification showed that a model answering its own verification questions decreases hallucinations. For anything that gets published, we still prefer a separate check against a fixed set of source facts that can block the draft.
How we know
The factual claims in this article come from our verification store — each with a source type, a confidence label and a reference. The method is documented on How we know.
– The turning point was when we stopped treating the truth-prompt as the defense and started treating it as the first layer, with the blocking check as the thing we actually trusted. | source: first-hand experience | conf: strong | ref: ebizapple first-hand — decision_rule
– We wrote a strict ‘never invent a fact, unknown beats invented’ instruction and it cut fabrication sharply, but a careful prompt alone never drove invented facts to zero. | source: product testing | conf: strong | ref: ebizapple first-hand — truth-prompt is layer one
– Making the grounding explicit in the prompt, so every claim had to map to a supplied fact, made the blocking check’s job easier and the drafts cleaner on the first try. | source: product testing | conf: observed | ref: ebizapple first-hand — explicit grounding
– What actually stopped the fabrications was a separate check that compared every concrete claim against a fixed set of source facts and blocked anything not backed, regardless of how good the prompt was. | source: product testing | conf: strong | ref: ebizapple first-hand — the blocking check
– The fabrications that survived a strong prompt were the confident-sounding ones: a plausible number, a named source, a tidy statistic the model filled in because the sentence wanted one. | source: product testing | conf: strong | ref: ebizapple first-hand — what survives