Retrieval-augmented generation, or RAG, is a way to make a language model answer from your documents instead of relying only on what it learned during training. The model does not memorise your files. At question time, the system finds relevant passages, puts them in the prompt and asks the model to answer from that evidence.
That sounds simple, and the basic version is. The engineering is in the retrieval, the permissions and the evaluation. This explainer walks through the pipeline in plain language and shows where projects usually succeed or fail.
The six parts of a RAG system
- Sources: the documents, pages, tickets or database records the assistant is allowed to use.
- Ingestion: the process that extracts text, metadata, headings and permissions from each source.
- Chunking: splitting documents into passages that are small enough to retrieve but large enough to contain meaning.
- Indexing: converting chunks into embeddings and storing them with keyword indexes and metadata.
- Retrieval: finding the most relevant chunks for a question, often with hybrid search and a reranker.
- Generation: asking the model to answer from the retrieved passages, with citations and a fallback when the evidence is insufficient.
Chunking is a product decision
A chunk should be a coherent section, not an arbitrary 500 characters. Keep the heading, the section title and the document date with the text. If a policy has exceptions, keep the exception with the rule. If a table is important, preserve the table structure or store a summary alongside it.
Bad chunking produces answers that are technically grounded but practically wrong: the retrieved passage says "the standard plan includes X", but the exception on the next page says "except for customers in Y". The model cannot cite what it never retrieved.
Embeddings and hybrid search
Embeddings turn text into vectors that capture meaning. A vector search can find a passage that uses different words from the question. Keyword search is better for exact terms: product codes, legal references, names and acronyms. Hybrid search combines both, which is why it is common in production.
A reranker then reorders the candidate passages by relevance to the question. This is often the highest-leverage improvement after clean documents. It is also a cost and latency decision.
Generation with citations
The prompt should tell the model three things: answer only from the provided passages, cite the source for every factual claim, and say "I don't know" when the passages do not contain the answer. The interface should show the citation next to the claim, not in a footnote at the bottom.
If the answer is a recommendation, separate the evidence from the recommendation. "The policy says X; based on that, the recommended action is Y" is easier to audit than a single confident paragraph.
What RAG is good for
- Policy and procedure questions where the source is authoritative.
- Product documentation and support articles.
- Internal onboarding and sales enablement.
- Contract and tender review with citations.
- Customer support where the answer must be traceable.
What RAG is not
- Not a database: it does not replace a system of record or a reporting layer.
- Not a guarantee of accuracy: it improves grounding but does not eliminate bad retrieval or bad documents.
- Not a permissions system: access control must come from the source, not from a prompt.
- Not a one-time upload: documents change, and the index must change with them.
How to evaluate it
Build a set of real questions with expected answers and required citations. Measure retrieval recall (did the right passage appear?), answer correctness, citation accuracy and the share of questions answered with "I don't know" when appropriate. Run the set after every change. If you cannot measure it, you cannot improve it.
A worked example: support policy assistant
Imagine a coaching centre with a 40-page admission and refund policy. The assistant is allowed to answer parent questions about fees, refunds, attendance and transfer rules. The source is the policy PDF plus a structured fee table. The retrieval layer chunks by section, keeps the section heading and effective date, and tags each chunk with the policy version. The answer must cite the section and say "I don't know" when the policy is silent.
In evaluation, the team finds that refund questions retrieve the old policy because both versions are indexed. The fix is not a better prompt; it is retiring the old version and adding a review date. A second issue is that a Bangla question retrieves the English section. The fix is language metadata plus a multilingual embedding model. Both fixes improve the system more than switching to a larger model.
RAG versus fine-tuning
Fine-tuning changes how a model behaves; RAG changes what it can see. Use RAG when the knowledge changes often, the answer must cite a source or the content is too large to embed in a prompt. Use fine-tuning when the task requires a specific style, format or classification behaviour that prompting cannot achieve reliably. Many systems use both: a fine-tuned or prompted behaviour plus retrieval for facts.
Do not fine-tune a model to memorise a policy that changes every quarter. You will rebuild the model instead of updating a document.
Latency and cost
RAG adds work before the model generates: retrieval, reranking and a longer prompt. That affects latency and cost. Measure each stage. A smaller reranker, fewer chunks or a cached answer can improve both. Streaming the answer improves perceived latency even when total time is unchanged.
Where this fits in our work
If you want help designing a private knowledge assistant, start with Business Automation. The pricing page explains scope. For the failure modes, read RAG problems and fixes, and for the buying decision read the RAG buying guide.
Sources
- Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", 2020.
- Google Cloud, "What is RAG?", accessed 16 November 2026.
- Microsoft, "Retrieval-augmented generation overview", accessed 16 November 2026.
- McKinsey, "The State of AI: Global Survey 2026", 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.