Skip to content
EncryptTechBD
Home Industries Our Work Blog Pricing About Contact Discuss Your Project
বাংলা (BN)
AI & Agentic Workflows 6 min read

Private knowledge assistant problems: symptoms, causes and fixes

Diagnose stale answers, retrieval misses, permission leaks, latency and low trust in a private knowledge assistant, with fixes and prevention.

E
EncryptTechBD · 0 comments
Small humanoid robot sitting on a wooden floor
Photo: Jiuguang Wang / Wikimedia Commons (BY-SA)

A private knowledge assistant can pass a pilot and still fail in daily use. The failures are rarely explained by “the model is not smart enough”. They come from document versions, retrieval design, permission filters, answer policy and the absence of an owner for content. Each of those has a symptom you can observe and a fix you can test.

This troubleshooting guide works through the problems we see most often in company assistants built on internal policies, product documentation, contracts and support notes.

Run this diagnostic sequence first

  1. Reproduce with the exact question and the exact user role. A different role may receive a different answer because the permission filter changed the evidence.
  2. Capture the retrieved passages. Record document IDs, chunk text, scores and timestamps. Without this, every fix is guesswork.
  3. Check the source version. Confirm which file is current, when it was ingested and whether an older copy still exists in the index.
  4. Inspect the answer policy. Read the system prompt and the rules for citing, abstaining and handling conflicting sources.
  5. Test with the evaluation set. Compare the failure with known good questions to see whether the problem is specific or systemic.
  6. Check the user's context. Department, language, device and entry point can all change what the assistant receives.

The assistant cites an outdated policy

Symptom: an employee asks about leave, expenses or a customer commitment and receives the previous version, sometimes with a confident citation.

Likely cause: ingestion appends new content without retiring the old version, document identity is based on filename rather than a stable ID, or the effective date is stored in the body but not used as metadata.

Fix: give every document a stable identity and a version. Store effective-from and effective-to dates, supersede the old version explicitly, remove its chunks from the index and re-run a fixed set of questions. If the source system supports change events, use them so deletion and updates propagate automatically.

Prevention: write an ingestion contract. It should name the content owner, the system of record, the update trigger, the deletion rule and the freshness target. Review the ten most-cited documents every month.

The right document is never retrieved

Symptom: staff know the answer exists in a PDF or wiki page, but the assistant says it cannot find it or answers from a weaker source.

Likely cause: chunking split a table or procedure across boundaries, metadata is missing, the user's wording differs from the document's vocabulary, top-k is too small, or the index has no keyword search to complement embeddings. Bangla and English terminology can also fragment retrieval if the content and queries use different languages.

Fix: inspect the actual chunks, not just the source file. Add headings and metadata to each chunk, test hybrid keyword-plus-vector retrieval, add query expansion for common synonyms and consider a reranking step. Build a retrieval test set with one question per document type.

Prevention: test retrieval quality separately from answer quality.

The answer sounds confident but has no evidence

Symptom: a fluent paragraph with no link, or a citation that does not support the sentence it follows.

Likely cause: the prompt instructs the model to always answer, retrieval returns weak passages but no threshold blocks them, citation is generated from memory rather than passage IDs, or the system summarises across unrelated chunks.

Fix: add a relevance threshold and a clear “not found in the available documents” response. Require citation by passage ID, verify that the cited passage appears in the retrieved set, and route low-confidence answers to a human. Test with questions that have no answer in the corpus.

Prevention: make abstaining a supported outcome. Staff should see the difference between “the policy does not cover this” and “I could not find the policy”.

A user sees a document outside their remit

Symptom: a junior employee receives salary bands, a supplier contract or another department's incident notes.

Likely cause: permissions are enforced in the interface but not in retrieval, the index was built from a shared drive, access-control lists are not synchronised, or answers are cached without the user's role in the cache key.

Fix: apply permission-aware retrieval with deny-by-default. Filter at query time using the user's identity and group membership, synchronise access changes on a schedule, and include the role in every cache key. Log which documents were retrieved for each answer and review the log.

Prevention: run a quarterly access test with real accounts from different departments. Test the leaver process as well, because an assistant can keep answering from an index long after an account is disabled.

Latency and cost climb after ingestion grows

Symptom: answers that took a few seconds now take much longer, and the monthly model bill rises faster than usage.

Likely cause: every update re-embeds the entire corpus, chunks are oversized, the retriever returns too many passages, there is no cache, or a large model is used for simple lookups.

Fix: index incrementally, tune chunk size against retrieval tests, reduce top-k, add a semantic cache for repeated questions, and route simple queries to a smaller model. Measure p95 latency and tokens per answer, not just averages.

Prevention: publish a monthly cost and latency review with thresholds that trigger investigation.

The pilot scores well but staff avoid the tool

Symptom: evaluation results look acceptable, yet people go back to searching the shared drive or asking a colleague.

Likely cause: evaluation questions were written by the builders, answers lack source links, the assistant is not embedded in the workflow, there is no feedback path, or nobody owns incorrect content.

Fix: replace synthetic questions with real tasks from real roles, show sources first, add a one-click “this is wrong” route to the content owner, and review unanswered questions weekly. Measure repeat use and task completion, not only satisfaction.

Prevention: treat the assistant as an operating service with a content owner, a review cadence and a published scope of what it should and should not answer.

Repair or rebuild

Repair is the right path when retrieval can be measured, permissions can be represented in the index and the vendor exposes enough logs to diagnose a bad answer. Rebuild is more honest when content has no owner, access rules cannot be expressed at document level, or the system cannot show which passages produced an answer.

The minimum operating routine

  • Weekly: review unanswered questions, negative feedback and failed retrievals.
  • Monthly: test permissions with real roles, check the most-cited documents and review cost and latency.
  • Quarterly: refresh the evaluation set, test prompt injection and data-exposure scenarios, and confirm retention and deletion rules still match policy.

If you need help diagnosing retrieval and permission problems, or designing an assistant that can show its evidence, start with Business Automation. The pricing page describes how discovery, build and support engagements are structured. For the content side of the problem, read why company knowledge bases fail retrieval.

Sources

Services that deliver this

Want this implemented properly?

Pricing →

Every engagement starts with a written scope. If a requirement is outside it, we quote it separately before starting.

Responses (0)

No comments yet. Add the first one below.

Leave a response

Your email address will not be published. Required fields are marked *.

Keep reading

More field notes