AI ENGINEERING | LIVING GLOSSARY | #008
HALLUCINATION
When AI Generates Unsupported Answers
Chapter 2: Engineering AI for Reliability
AIYING — Accomplishing with AI
Imagine an employee asking an enterprise AI assistant:
Question: "What is our company's hotel reimbursement limit?"
AI response: "Employees can claim up to ₹8,000 per night."
The answer sounds clear and authoritative.
But suppose the approved travel policy contains no such limit.
The AI has produced a plausible-looking statement without adequate support.
This is a typical example of what is commonly called an AI hallucination.
1. What Is an AI Hallucination?
In generative AI, a hallucination commonly refers to generated information that is false, fabricated, internally inconsistent, or unsupported by the relevant evidence or task context.
Examples include:
- Inventing facts, dates, or figures
- Producing nonexistent citations
- Misrepresenting what a document says
- Inventing product features or company policies
- Describing nonexistent software functions
- Claiming that an action was completed when it was not
Researchers and practitioners use the term in somewhat different ways. In particular, factual incorrectness and lack of support from supplied sources are related but distinct problems.
A statement can be factually correct but unsupported by the documents provided. Conversely, a model may faithfully repeat an incorrect source.
AI engineering must therefore examine both correctness and grounding.
↑ Back to contents2. Why Hallucinations Occur
Language models generate outputs using learned statistical relationships and the context available during inference.
They do not automatically verify every statement against an authoritative source.
Hallucinations can arise from several interacting factors.
A. Missing information
The information required to answer may not be available to the model.
B. Ambiguous requests
The question may lack important details or permit multiple interpretations.
C. Incomplete or misleading context
Retrieved documents may omit relevant facts or contain outdated information.
D. Learned patterns
A model may generate plausible patterns that do not correspond to actual facts.
E. Complex tasks
Long or multi-step tasks may create additional opportunities for unsupported claims and errors.
F. Pressure to provide an answer
Task instructions and interaction patterns may encourage answering even when the available evidence is insufficient.
3. Common Forms of Hallucination
| Type | Example |
|---|---|
| Factual fabrication | Inventing a company's founding year. |
| Source fabrication | Citing a research paper that does not exist. |
| Source misrepresentation | Claiming a document supports a statement it does not support. |
| Numerical fabrication | Producing an unsupported reimbursement amount. |
| Technical fabrication | Inventing an API method or software configuration. |
| Action fabrication | Claiming that a database was updated without executing the operation. |
These categories are useful for diagnosing failures, although they are not a single universally standardized taxonomy.
↑ Back to contents4. Why Hallucinations Matter
Unsupported AI outputs can cause practical harm when users or connected applications rely on them.
| Application | Potential consequence |
|---|---|
| Customer support | Incorrect promises or misleading policy guidance. |
| Software engineering | Broken code or nonexistent dependencies. |
| Enterprise knowledge | Incorrect interpretation of internal documents. |
| Research | Fabricated references or inaccurate summaries. |
| Business operations | Decisions based on unsupported information. |
| AI agents | Incorrect actions based on fabricated state or tool results. |
The seriousness of a hallucination depends on the application, the decision involved, and the availability of verification or human oversight.
↑ Back to contents5. Detecting Unsupported Outputs
Hallucination detection requires checking generated claims against appropriate evidence or expected behavior.
A. Reference-answer comparison
Compare outputs with verified answers where these are available.
B. Source-grounding checks
Determine whether the cited passage actually supports each claim.
C. Factual verification
Check important statements against authoritative information.
D. Structured validation
Validate generated values, permitted categories, and required formats.
E. Tool-result verification
Confirm whether the claimed operation actually occurred.
F. Human expert review
Use qualified reviewers for complex, ambiguous, or high-impact outputs.
G. Automated evaluation
Use repeatable tests and graders, while recognizing that automated graders can also make mistakes.
6. Reducing Hallucination Risk
No single technique eliminates hallucinations in all circumstances.
A. Provide relevant evidence
Use reliable sources and retrieval when the task requires external knowledge.
B. Improve retrieval quality
Retrieve authoritative, current, and relevant information rather than merely semantically similar text.
C. Specify uncertainty handling
Instruct the system to acknowledge insufficient evidence or request clarification.
D. Validate outputs
Use deterministic checks where possible, especially for calculations, structured fields, and tool operations.
E. Require verifiable citations
Preserve source references and check that claims are supported.
F. Decompose complex tasks
Separate retrieval, reasoning, generation, and verification when doing so improves measurable performance.
G. Evaluate systematically
Test typical cases, difficult cases, missing evidence, and known failure patterns.
H. Apply appropriate human oversight
Escalate consequential or uncertain outputs when automated verification is insufficient.
Fine-tuning and model selection may also influence hallucination behavior, but their effectiveness must be tested for the intended task.
↑ Back to contents7. Industry Applications
Hallucination management is important across many AI engineering applications.
- RAG assistants: Verify whether answers follow retrieved evidence.
- Coding assistants: Run tests and check referenced APIs.
- Research assistants: Verify citations and factual claims.
- Customer support: Restrict policy claims to approved information.
- Document extraction: Check extracted fields against source documents.
- AI agents: Verify tool execution and actual system state.
The appropriate controls depend on the consequences of an incorrect output.
↑ Back to contents8. What Must an AI Engineer Decide?
A. Which claims require verification?
Identify critical facts, numerical values, citations, and action-completion statements.
B. What counts as authoritative evidence?
Define approved information sources and source precedence.
C. When should the system abstain?
Establish rules for missing, conflicting, or insufficient evidence.
D. What can be checked automatically?
Use application logic, databases, calculators, validators, and other trusted tools.
E. What requires human judgment?
Identify high-impact decisions and cases where evidence is ambiguous.
F. How should hallucination rates be measured?
Define clear error categories and test against representative evaluation datasets.
G. How will failures be investigated?
Distinguish retrieval failures, source-quality problems, generation errors, and verification failures.
H. What happens when verification fails?
Design safe fallback behavior, clarification requests, or escalation procedures.
9. Practical Industry Case: The Invented Travel Policy
An employee asks:
The assistant replies:
However, the approved company policy contains no such amount.
Engineering response
- Identify the unsupported reimbursement claim.
- Inspect the retrieved policy passages.
- Check whether the applicable policy version was indexed.
- Verify that the question was routed to the correct documents.
- Test whether the model invented the value despite insufficient evidence.
- Revise the prompt to require evidence for numerical policy claims.
- Implement source validation where feasible.
- Define an appropriate response when the limit cannot be found.
- Add the case to the evaluation dataset.
- Retest the system and monitor related failures.
Better response
Industry insight: A system that acknowledges missing evidence can be more dependable than one that always produces a confident answer.
↑ Back to contents10. Try It Yourself
Mini Project: Detect and Reduce Unsupported Claims
Objective: Compare AI responses with and without explicit evidence requirements.
- Create a fictional one-page company travel policy.
- Include several clearly stated reimbursement rules.
- Deliberately omit one requested reimbursement limit.
- Prepare ten questions, including answerable and unanswerable cases.
- Ask an AI model to answer using the document.
- Record factual claims and cited sources.
- Identify unsupported statements.
- Revise the prompt to require evidence and appropriate uncertainty.
- Repeat the test using the same model and settings.
- Compare unsupported-claim rates.
- Check whether valid answers remain complete and useful.
- Test the revised approach on new questions.
Suggested evaluation measures
- Number of unsupported factual claims
- Percentage of answers fully supported by evidence
- Correct handling of unanswerable questions
- Accuracy of citations
- Completeness of supported answers
Reflection questions
- Did the model invent any facts?
- Did citations actually support the claims?
- Did requiring evidence reduce unsupported outputs?
- Did the revised prompt cause unnecessary refusals?
- Which failures require additional application controls?
Success criterion: Demonstrate a repeatable method for identifying unsupported claims and measuring whether an engineering change reduces them.
11. Connected Glossary Terms
- Token: Influences the available information and output budget.
- Embedding: Supports semantic retrieval but does not guarantee factual relevance.
- Vector Database: Helps retrieve candidate evidence.
- RAG: Makes external evidence available to generation.
- Context Window: Limits how much supporting information can be supplied.
- Prompt Engineering: Specifies evidence use and uncertainty handling.
- Evals: Measures unsupported outputs and other failure patterns.
- Grounding: Connects generated claims to supporting evidence.
Previous lessons:
- #001 — Token
- #002 — Embedding
- #003 — Vector Database
- #004 — RAG
- #005 — Context Window
- #006 — Prompt Engineering
- #007 — Evals
Next lesson: #009 — Grounding: Connecting AI Outputs to Verifiable Evidence.
↑ Back to contents12. Living Update
Editorial baseline: November 2026
Hallucination mitigation remains an active area of AI engineering.
Important approaches to monitor include:
- Improved retrieval and source verification
- Claim-level factuality evaluation
- Automated citation checking
- Uncertainty-aware generation and abstention
- Tool-assisted verification
- Structured output validation
- Model-specific reliability evaluation
- Human review for consequential outputs
New models and techniques may reduce some failure rates, but engineers should verify performance for their own applications.
Living Glossary note: Update this entry as evidence-grounding, factuality evaluation, and verification methods evolve.
↑ Back to contents13. Key Takeaway
The objective is not to make every answer sound certain. It is to make the system dependable when information is available—and appropriately cautious when it is not.
Think like an AI engineer: What should the system do when it cannot establish that an answer is supported by trustworthy evidence?
↑ Back to contents14. References and Further Reading
- Ji et al. — Survey of Hallucination in Natural Language Generation
- Huang et al. — Survey on Hallucination in Large Language Models
- Lewis et al. — Retrieval-Augmented Generation
- OpenAI — Evaluation Guide
- NIST — AI Risk Management Framework
