A support assistant promises a refund window that the company does not offer. A procurement tool cites clause 14 of a supplier contract that was never signed. A reporting agent adds two figures correctly and labels the result with the wrong month. In each case the model produced text that read as finished work. The business received a defect dressed as an answer.
Hallucination is often described as a model problem. In a workflow it is better understood as an evidence problem. The system asserted something that its available sources, tools or rules did not support. That framing matters because it points to controls a business can actually buy, test and monitor.
What a hallucination looks like in a workflow
The academic definition is broad. For operations, five patterns cover most of the damage.
- Fabricated fact: a policy, fee, delivery time or product attribute that no approved source contains.
- Misattributed evidence: a real document is cited for a claim it does not make, or the citation points to the wrong section.
- Wrong tool action: the model calls a lookup or booking tool with the wrong identifier, date or quantity.
- Plausible arithmetic: a total or variance that looks reasonable but does not reconcile to the underlying rows.
- Stale truth: an answer that was correct before a price change, policy update or staff departure and is now out of date.
All five share a property: the output is fluent. Fluency is why the failure survives a quick review. A blank screen prompts someone to investigate; a confident paragraph often does not.
Where the error enters
It helps to separate four layers, because each has a different owner and a different fix.
- Context: the right document was never retrieved, was chunked badly, or was excluded by a permission filter. The model then fills the gap.
- Generation: the model has the evidence but overgeneralises, merges two records, or invents a connective detail.
- Tool use: the model chooses a valid tool but supplies an invalid argument, or misreads a successful response as a failure.
- Presentation: the interface strips citations, hides uncertainty, or presents a draft with the same visual weight as a verified answer.
Teams frequently buy a larger model when the fault sits in retrieval or permissions. That is an expensive way to leave the cause in place.
A reference flow that contains the risk
- Classify the request and its consequence. A draft marketing sentence and a payment instruction should not travel the same path.
- Retrieve only from approved sources. Apply the asking user's permissions before the content reaches the model, not after.
- Assemble context with citations and timestamps. Record which version of which document was used.
- Generate under a schema. Require named fields, a source list and an explicit "not found" option rather than free prose alone.
- Validate deterministically. Check schema, arithmetic, allowed values, policy rules and cross-source agreement outside the model.
- Route by risk. Low-consequence drafts can proceed; money movement, customer commitments and regulated statements need a reviewer with the evidence in front of them.
- Log and sample. Store the prompt, sources, output, validation result and reviewer decision, then audit a sample every week.
The flow is deliberately unglamorous. Its value is that every step produces evidence a manager can inspect when an answer is challenged.
Worked example: an invoice exception
An accounts-payable workflow reads a supplier invoice, matches it to a purchase order and proposes a journal entry. Hallucination risk appears in the supplier name, tax amount, currency, PO line and payment terms. A defensible design extracts each field with a confidence signal, matches the supplier against the vendor master by exact identifier, applies a tolerance rule to quantity and price, checks for a duplicate invoice number, and stops for human approval when any field is uncertain or the amount exceeds the delegated limit.
Notice what the model is not asked to do. It does not decide whether the supplier is legitimate, and it does not post the entry. Those are rule-based and human decisions. The model accelerates extraction and drafting; the control boundary stays with the finance team.
The mitigation stack
- Grounding: answers must point to approved, current sources, with the retrieved passage available to the reviewer.
- Constrained output: a schema with required fields, enumerations and an abstain option reduces invented structure.
- Deterministic checks: totals, dates, identifiers and policy thresholds are verified by code, not by another model prompt.
- Tool verification: confirm that a tool call changed the intended record, and read back the result before reporting success.
- Abstention: make "I cannot confirm this from the available sources" an acceptable and measurable outcome.
- Risk-based human review: give reviewers the source, the diff and the decision criteria, not a wall of generated prose.
- Evaluation and monitoring: keep a task-specific test set, re-run it after model or prompt changes, and track unsupported-claim reports from users.
Four claims that mislead buyers
- "A bigger model removes hallucination." It may reduce some errors and introduce others. The evidence pipeline still decides whether the answer is supportable.
- "Temperature zero makes it deterministic." Lower randomness does not supply missing context or correct a bad retrieval ranking.
- "Retrieval-augmented generation solves it." RAG improves grounding, but poor chunking, stale indexes and permission gaps remain.
- "Human review catches everything." Reviewers miss errors when they are rushed, given no source, or asked to approve hundreds of outputs an hour.
Limits and residual risk
No control set takes the error rate to zero. The honest position is to measure unsupported claims by task, decide which tasks can tolerate them, and put irreversible actions behind a gate. Cost and latency rise with retrieval, validation and review, so the control level should follow consequence rather than fashion. Long-tail requests and newly changed policies remain the weak spots, which is why source freshness needs an owner.
When it becomes a board issue
Hallucination belongs on the risk register when output can create a customer commitment, move money, influence a regulated decision, affect safety, or damage a reputation at scale. At that point the questions are not about model benchmarks. They are about evidence, approval authority, logging, incident response and who signs off when the system is wrong.
How we build these workflows
We design agentic workflows with retrieval, validation and human checkpoints under Business Automation. The commercial shape of a pilot and rollout is described on the pricing page. If you are writing the surrounding policy, the AI governance policy guide covers approval and accountability.
Sources
- NIST, AI Risk Management Framework, accessed 17 November 2026.
- OWASP, Top 10 for Large Language Model Applications, accessed 17 November 2026.
- Gartner, applying uniform governance across AI agents, accessed 17 November 2026.
- EU Artificial Intelligence Act, implementation timeline, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.