A hallucination in a chat window is an inconvenience. A hallucination inside a workflow is a business event. The model may invent a refund policy, cite a superseded circular, summarise a contract with the wrong termination date, or send a customer an answer that sounds authoritative and is simply wrong. By the time a person notices, the message has been sent, the ticket has been closed or the payment has been released.
Most teams respond by changing the prompt and lowering the temperature. That occasionally helps and rarely explains the failure. This guide treats hallucination as a system symptom rather than a single model defect. Work through the symptom map, run the diagnostic sequence, then fix the layer that actually failed.
Read the symptom before touching the model
Write down what the user saw, what the workflow did and which evidence the model had at that moment. "It made something up" is not a diagnosis. The useful question is whether the model lacked the source, retrieved the wrong source, ignored the source, combined sources incorrectly, or was asked to do arithmetic or judgement that should have been handled by code.
Symptom, likely cause, fix and prevention
Invented policy details or prices
Likely cause: the model is answering from general training knowledge because no approved source was retrieved, or the prompt does not require a citation.
Fix: restrict the answer to an approved corpus, require a source reference for every factual claim, and add an explicit refusal path when retrieval returns nothing above the relevance threshold.
Prevention: make "no answer" a valid, tested outcome. Review refusals weekly; a workflow that never refuses is probably guessing.
The right document, the wrong version
Likely cause: the index contains obsolete documents, or retrieval ignores effective dates and document status.
Fix: attach version, effective date, owner and status metadata to every chunk, filter retrieval by those fields, and remove or archive superseded copies from the active index.
Prevention: connect ingestion to the document lifecycle so that publishing a new version retires the old one. Test with a question whose answer changed last quarter.
Confident answers from empty retrieval
Likely cause: no minimum relevance threshold, or the fallback prompt invites the model to "use its best judgement".
Fix: set a threshold, return a known fallback message, and route the conversation to a human queue with the original question attached.
Prevention: monitor the retrieval score distribution and the fallback rate. A sudden drop in fallbacks after a model upgrade deserves investigation.
Different answers to the same question
Likely cause: prompt sensitivity, unstable retrieval ranking, or no evaluation set to define acceptable variation.
Fix: use deterministic templates for regulated or repetitive outputs, pin model versions where possible, and add a regression set of paraphrased questions.
Prevention: run the evaluation set on every prompt, model or index change. Record pass rates by intent, not as one overall percentage.
Wrong numbers in summaries
Likely cause: the model is being asked to add, compare or reconcile figures in natural language.
Fix: move calculation into a tool or query, pass the computed result to the model, and require the model to quote the result rather than derive it.
Prevention: classify every numeric field as calculated or extracted at design time. Calculated fields never come from the model.
An action taken on a false premise
Likely cause: the model has direct tool access without a validation step, or the workflow treats a generated value as an approved instruction.
Fix: separate drafting from execution. Require schema validation, a second confirmation for irreversible actions, allow-listed parameters and an idempotency key on every write.
Prevention: log the evidence, the proposed action and the approver for every side effect. Review a sample of approved actions each month.
Failures concentrated in one language
Likely cause: the evaluation set is mostly English, the chunking strategy suits English prose, or the Bangla corpus is thinner than the English one.
Fix: build a bilingual test set, inspect retrieval separately for each language, and adjust chunk size and query expansion where Bangla documents retrieve poorly.
Prevention: report quality by language and by channel. An aggregate score can hide a completely broken Bangla experience.
Diagnostic sequence
- Reproduce with the original inputs. Capture the prompt, retrieved passages, model version, tool calls and final output.
- Inspect retrieval first. Was the correct passage available, ranked highly and inside the context window?
- Test with the passage supplied directly. If the model still invents, the failure is in generation or instruction design, not search.
- Check the date and status of the source. Confirm that the retrieved document was valid at the time of the answer.
- Separate extraction from calculation. Re-run the task with the arithmetic handled by a query or tool.
- Review the action boundary. If the output triggered a write, inspect the validation and approval layer, not only the text.
- Add the case to the regression set. A fixed incident that is not tested will return with the next prompt change.
Prevention controls that survive model changes
- Grounded answers with citations: every factual statement points to an approved source, and the interface lets a reviewer open it.
- Explicit refusal behaviour: the workflow knows what it does not know and hands off cleanly.
- Tool constraints: narrow permissions, schema validation, confirmation for irreversible actions and a full audit trail.
- A regression suite: real questions, paraphrases, outdated documents, empty retrieval and bilingual cases, run on every change.
- Production monitoring: retrieval scores, refusal rates, citation coverage, correction rates and user-reported errors, segmented by workflow and language.
- An incident log: every confirmed hallucination records the cause, the fix and the test that now covers it.
What not to do
Do not rely on "be accurate" as a control. Do not assume a lower temperature removes fabrication; it may make output more repeatable without making it true. Do not evaluate with five friendly questions written by the project team. Do not let a model browse the open web for a policy answer that should come from a controlled document. And do not treat a vendor's benchmark as evidence that your workflow will be safe.
Where this fits in our work
We design and operate AI workflows under Business Automation, including retrieval, tool permissions and human review. The pricing page explains how an agentic deployment is scoped. For the retrieval layer behind many of these failures, read the common RAG knowledge-base problems and fixes.
Sources
- NIST AI Risk Management Framework, accessed 17 November 2026.
- NIST AI 600-1, Generative AI Profile, accessed 17 November 2026.
- OWASP Top 10 for Large Language Model Applications, accessed 17 November 2026.
- ISO/IEC 42001, Artificial Intelligence Management System, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.