"Private" is often used to mean that a vendor promises not to train on your documents. That is a contractual position, not an architecture. A private knowledge assistant is private because of where the data travels, who can retrieve which passage, and what the system can prove after an answer is given.
This explainer describes that architecture for the manager who approves the build, not the machine-learning specialist. It focuses on components, data flow and the decisions that determine trust six months after launch.
What the assistant is, and is not
The system does not learn your business by training on every file. In the common design, documents are indexed, relevant passages are retrieved at question time, and a language model writes an answer from those passages. The model supplies language and reasoning; your index supplies the facts.
That distinction explains the failure modes. If the index is incomplete or the permission filter is wrong, a better model produces a more confident wrong answer. If sources conflict, the assistant will sound certain about both unless the design forces it to show the conflict.
The reference architecture
- Sources and connectors. Shared drives, intranet pages, policy folders, ticketing and HR systems, each with an owner and a refresh method. Scheduled or event-driven jobs read changes without copying more than necessary, using read-only credentials where possible.
- Ingestion and normalisation. File type detection, text extraction, OCR for scans, table handling, language detection and removal of duplicates.
- Chunking. Documents are split along headings and paragraphs, not fixed character counts, so a retrieved passage makes sense on its own.
- Metadata and access control. Every chunk carries its source, version, department, classification and the identities allowed to see it.
- Index. A vector index for semantic similarity, often paired with keyword search for names, codes and exact phrases.
- Query handling. The question is rewritten for search, expanded with known synonyms, and classified so that out-of-scope requests are refused early.
- Retrieval and ranking. Candidate passages are filtered by permission, scored, reranked and trimmed to a context budget.
- Orchestration. The prompt assembles instructions, retrieved passages and conversation state. Tool calls, if any, are limited and logged.
- Answer, citation and logging. The response returns the answer, the source documents and the exact passages used. Questions, sources, latency, feedback and refusals are recorded, with personal data handled per policy.
Ingestion is where most projects quietly fail
A polished chat interface can sit on a weak corpus. The unglamorous work is deciding which documents are authoritative, removing superseded versions, and extracting text from scans. A policy folder with three revisions and no effective dates will produce three answers.
Build a source register before building the index. For each source, record the owner, the update frequency, the classification and the canonical location. If two systems hold the same policy, choose one as authoritative and mark the other as reference only.
Index and retrieval design
Chunk size is a trade-off. Small chunks retrieve precisely but lose context; large chunks preserve context but dilute relevance and consume the input budget. Heading-aware chunks of a few paragraphs, with a short summary, work well for policy and product documentation.
Pure vector search is not enough. Staff search for invoice numbers, product codes and names that embeddings handle poorly. A hybrid approach — keyword matching plus semantic retrieval, followed by a reranker — covers both. The reranker also gives you a place to enforce recency when two versions of a document compete.
Permissions must be enforced before generation
Filtering after the model writes is too late. The retrieval layer should receive the user's identity and group memberships, then exclude any chunk the user cannot open in the source system. Test with a restricted document and a user who should not see it; one leak ends trust.
Role changes must propagate. If someone moves from finance to sales, the index should reflect the new access at the next sync, not at the next annual review. Where possible, use the source system's permission model as the source of truth rather than maintaining a second one.
Citations, refusals and evaluation
Every answer should link to the passage it used. Citations let the reader verify and give you a debugging trail. A response that cannot cite a source should say so rather than fill the gap with plausible language.
Create a set of real questions with known correct answers before launch. Include questions that should be refused, questions with two valid interpretations, and questions about documents the test user cannot access. Measure answer correctness, citation accuracy, refusal behaviour and latency. Re-run the set after every change to the corpus, the prompt or the model.
Deployment choices
Private cloud or virtual private cloud: the index and application run in your cloud account, with the model called through a private endpoint or hosted in the same environment. This is the usual balance of control and effort.
On-premise: appropriate when data cannot leave a facility or when latency to an external service is unacceptable. It shifts responsibility for hardware, model serving and upgrades to your team.
Managed SaaS: fastest to start, but confirm where data is stored, whether prompts are retained, how deletion works and what the exit path looks like. A no-training clause is necessary but not sufficient.
A worked example
An HR team wants staff to ask about leave, travel and reimbursement rules. The index contains the current handbook, circulars and a benefits summary. A user in sales asks whether unused leave can be carried into the next year.
The query is classified as HR policy. Retrieval returns the current leave section and a circular that amended it; the permission filter excludes a draft policy marked for HR only. The answer states the carry-over rule, quotes the effective date and links to both sources. If they disagree, it says so and shows both rather than choosing silently.
Misconceptions that lead to bad builds
- "Private means secure." Privacy depends on access control, retention, logging and the deployment path, not a label.
- "A bigger model will fix bad retrieval." It produces more fluent answers from the same incomplete passages.
- "Index once and forget." Documents, roles and permissions change. Re-indexing and evaluation are operating tasks.
- "The chat window is the product." The product is the source register, the permission model and the evidence trail.
When it is worth doing
The case is strongest when staff spend real time searching for information that already exists, answers must be consistent across branches, and a wrong answer carries a cost. It is weak when the knowledge fits on one page, the sources are unreliable, or nobody will own the corpus.
This architecture is delivered through Business Automation, alongside document and workflow work. The pricing page explains how a pilot is scoped separately from production. For the operational view, read the year-by-year guide to private knowledge assistants.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, accessed 17 November 2026.
- OWASP, Top 10 for Large Language Model Applications, accessed 17 November 2026.
- NIST, AI Risk Management Framework, accessed 17 November 2026.
- ISO/IEC 42001, AI management system standard, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.