A retrieval-augmented generation system usually works in the demo and disappoints in production. The model is not always the problem. The common failures are in the knowledge base, the retrieval step, the permissions or the evaluation. This guide lists the problems we see most often and the fixes that address the cause rather than the symptom.
RAG combines a search step with a language model. The system retrieves relevant passages from your documents, adds them to the prompt and asks the model to answer with a citation. If the retrieval is wrong, the answer is wrong. If the document is stale, the answer is stale. If the user is not allowed to see the document, the system has a privacy incident.
Problem 1: the answer cites a document that is out of date
Symptom: the assistant quotes an old policy, price list or product manual.
Cause: the corpus has no ownership or review date. Documents are uploaded once and never retired.
Fix: assign an owner and a review date to every source. Add metadata for version, effective date and status. Retrieval should prefer current documents and explicitly label older versions. Build a monthly review of documents that have expired.
Problem 2: retrieval returns the wrong passages
Symptom: the answer is fluent but unrelated, or it misses the obvious document.
Cause: poor chunking, no headings, duplicate content, weak embeddings or a search index that does not match the language of the question.
Fix: chunk by section, not by fixed character count. Keep the heading with the chunk. Remove duplicate and superseded documents. Use hybrid search (keyword plus vector) where the domain has exact terms, product codes or Bangla/English mixing. Add a reranker and test retrieval separately from generation.
Problem 3: the assistant ignores permissions
Symptom: a user gets a summary of a document they cannot open in the source system.
Cause: the index has a separate permission list that has drifted, or the ingestion process copied everything into one collection.
Fix: inherit permissions from the source system at retrieval time. Test with real restricted documents and real users. Log every retrieval and every access decision. If the source system cannot expose permissions, do not index that content for a broad audience.
Problem 4: the model hallucinates despite the context
Symptom: the answer includes a fact, date or number that is not in the retrieved passages.
Cause: the prompt asks for a confident answer, the retrieval is weak, or the model is allowed to answer without evidence.
Fix: require citations for factual claims. Add an "I don't know" path when retrieval confidence is low. Use a grounded prompt that says "answer only from the provided passages". Evaluate with a set of questions where the correct answer is "not in the knowledge base".
Problem 5: multilingual questions retrieve English-only content
Symptom: a Bangla question returns an English document or a poor answer, even though the same content exists in Bangla.
Cause: the embedding model, tokenizer or index treats languages unevenly; chunks mix languages; there is no language metadata.
Fix: tag documents by language. Test retrieval in Bangla, English and mixed-language queries. Use a multilingual embedding model or translate the query as a fallback, but keep the original passage and citation. Do not assume a translated answer is acceptable for legal or policy content.
Problem 6: the system is slow
Symptom: users wait too long, especially on mobile connections.
Cause: too many retrieval calls, a large reranker, long prompts, synchronous document processing or a model that is too large for the task.
Fix: measure retrieval, reranking and generation separately. Cache stable answers. Retrieve fewer, better chunks. Use a smaller model for classification and a larger model only for complex answers. Stream the response so the user sees progress.
Problem 7: cost grows faster than usage
Symptom: the monthly bill rises even though the number of users is flat.
Cause: long contexts, repeated retrieval, retries, agent loops and no per-workflow budget.
Fix: log tokens and tool calls per answer. Set a budget per workflow. Route simple questions to a smaller model. Cache repeated questions. Review the top ten most expensive queries every month.
Problem 8: nobody can prove the system is getting better
Symptom: the team argues about quality based on anecdotes.
Cause: no evaluation set, no labelled answers and no retrieval metrics.
Fix: build a set of 50 to 100 real questions with expected answers and required citations. Measure retrieval recall, answer correctness, citation accuracy and "I don't know" behaviour. Re-run the set after every model, prompt or document change.
Problem 9: prompt injection and data leakage
Symptom: a user asks the assistant to ignore its instructions, reveal another customer's data or call a tool it should not use.
Cause: untrusted content is treated as instructions, the assistant has broad tool access, or the output is not filtered.
Fix: separate instructions from retrieved content. Limit tool permissions. Require approval for actions that change data or money. Test with adversarial prompts. Log and review blocked requests. Follow the OWASP guidance for LLM applications.
Problem 10: the knowledge base becomes a junk drawer
Symptom: the assistant answers from drafts, meeting notes and duplicate files.
Cause: anyone can upload anything, there is no taxonomy and no deletion process.
Fix: create a small content governance process: source of truth, owner, classification, retention and review date. Remove drafts and duplicates. Treat the knowledge base as a product, not a folder.
A diagnostic sequence
- Take one wrong answer and identify the retrieved passages.
- Decide whether the failure is retrieval, permission, document quality or generation.
- Fix the earliest failure in the chain. Do not tune the prompt to compensate for a missing document.
- Add the case to the evaluation set.
- Re-run the set after the change.
Where this fits in our work
If you want help designing or fixing a private knowledge assistant, start with Business Automation. The pricing page explains scope. For the buying decision, read the RAG buying guide, and for the architecture read how RAG works.
Sources
- Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", 2020.
- Microsoft, "Retrieval-augmented generation overview", accessed 16 November 2026.
- Google Cloud, "What is RAG?", accessed 16 November 2026.
- OWASP Top 10 for LLM Applications, accessed 16 November 2026.
- NIST, AI Risk Management Framework, accessed 16 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.