Buying a RAG system is not like buying a chatbot subscription. The product is only one part of the system. The rest is your documents, your permissions, your evaluation set and your operating process. A vendor that demos well on a clean sample corpus can still fail on your real content.
This buying guide is for an SME evaluating a RAG platform, an implementation partner or both. It focuses on the questions that separate a production system from a polished demo.
Start with the job, not the technology
Write one page before you talk to vendors:
- Who will use it: staff, customers, partners or a support team.
- What they need to answer: policy questions, product specifications, SOPs, contracts or support cases.
- Which sources: documents, intranet, helpdesk, CRM, shared drive or email.
- Which languages: English, Bangla or both, including mixed-language questions.
- What a wrong answer costs: inconvenience, a bad customer reply, a compliance issue or money.
- What "good" means: answer accuracy, citation rate, time saved, ticket deflection or fewer escalations.
If the job is low-consequence and the documents are public, a simple tool may be enough. If the job involves personal data, money or legal obligations, the requirements are different.
Requirements that matter in production
Permissions and access
Ask how the system inherits permissions from the source. A separate allow-list will drift. Test with a user who is not allowed to see a specific document. If the vendor cannot demonstrate this, stop.
Citations and provenance
Every factual answer should show the source, the document date and the section. A confident paragraph without provenance is not usable for policy or customer support.
Document lifecycle
Ask how documents are added, updated, retired and deleted. What happens when a policy changes? How quickly does the answer change? Can you remove a document from the index and the cache?
Evaluation
The vendor should support an evaluation set that you own. You need to measure retrieval and answer quality before and after changes. If the only quality dashboard is the vendor's, you cannot compare alternatives.
Security and residency
Confirm where documents and prompts are stored, whether they are used for model training, which subprocessors are involved and how deletion works. Bangladesh's Personal Data Protection Act, 2026 makes data handling a business obligation, not just a vendor feature.
Actions and approvals
If the assistant can change a ticket, send an email or update a record, ask for approval rules, idempotency and an audit trail. Start with draft-only actions.
Build, buy or both
Buy when the use case is common, the data is structured and the vendor's permissions and evaluation model fit. Build when the workflow is a differentiator, the integration is unusual or the data cannot leave your environment. Both is common: a managed retrieval layer with your own evaluation, prompts and access controls.
Do not build a vector database from scratch unless you have a engineering reason. Do build the evaluation set and the content governance; those are yours regardless of vendor.
Vendor scorecard
- Use case fit: does it support your page types, languages and sources?
- Permissions: inherited, testable and auditable.
- Citations: source, date, section and confidence.
- Evaluation: your set, repeatable, exportable.
- Integration: APIs, connectors, rate limits and error handling.
- Security: access control, encryption, logging, incident notification and subprocessors.
- Residency: where data is stored and processed.
- Pricing: per seat, per document, per query or per outcome; overage and minimums.
- Exit: export format, index portability, deletion and transition support.
- Support: named contact, response time and escalation path.
Pricing questions
RAG costs are driven by ingestion, storage, retrieval, generation and human review. Ask for a model that includes all five. A low per-seat price can hide a per-query cost that grows with adoption. A "unlimited" plan can hide fair-use limits. Ask what happens at 2x and 5x usage.
The pilot contract
Run a paid, time-boxed pilot on one use case with real documents and real users. Define success before the pilot: retrieval accuracy, answer accuracy, citation rate, time saved and at least one failure mode. Include a exit clause and a data deletion confirmation. Do not sign a multi-year contract before the pilot.
RFP questions to send every vendor
- Which document types and languages do you support, including Bangla and mixed-language queries?
- How do you inherit permissions from the source system, and can we test a restricted document?
- Where are documents, prompts and logs stored? Which subprocessors are involved?
- Is our content used to train models? Can we opt out in writing?
- How do you handle document updates, deletions and retention?
- Can we export the index, prompts and evaluation results at exit?
- What does a typical implementation take, and who owns the work?
- What is included in support, and what is billed separately?
Pilot scorecard
Score the pilot on five dimensions, not on enthusiasm. Retrieval quality: did the right passage appear for 80 percent of test questions? Answer quality: was the answer correct and complete? Citation quality: could a reviewer trace every claim? Permission quality: did restricted content stay restricted? Operational quality: how long did it take to add a document, change a prompt or investigate a bad answer?
Add one commercial dimension: cost per successful answer at the pilot volume and at three times the volume. A pilot that is cheap at 20 users and unaffordable at 100 is not a viable production system.
Common buying mistakes
- Buying a platform before defining the use case and the evaluation set.
- Assuming the vendor's sample corpus represents your documents.
- Skipping the permission test because the demo looked secure.
- Pricing on seats when usage is driven by queries and documents.
- No exit plan for the index, prompts and evaluation data.
- Treating the pilot as a procurement formality instead of a real test.
Where this fits in our work
If you want help scoping a private knowledge assistant, start with Business Automation. The pricing page explains our scope. For the failure modes, read RAG problems and fixes, and for the architecture read how RAG works.
Sources
- Microsoft, "Retrieval-augmented generation overview", accessed 16 November 2026.
- Google Cloud, "What is RAG?", accessed 16 November 2026.
- AWS, "Knowledge bases for Amazon Bedrock", accessed 16 November 2026.
- DataGuidance, "Bangladesh: Personal Data Protection Act 2026", accessed 16 November 2026.
- NIST, AI Risk Management Framework, accessed 16 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.