A support assistant summarises a supplier document and adds a line telling the reader to email account details to an external address. A procurement agent, asked to compare quotations, creates a purchase requisition that nobody requested. In both cases the model did what it was told. The problem was who told it.
Prompt injection is the failure mode where untrusted content becomes an instruction. It is easy to confuse with a hallucination, which is an unsupported assertion, or with ordinary misuse by an authorised user. Triage starts by deciding which of the three you are dealing with, because the fixes are different.
Symptoms that suggest injection rather than a bad answer
| Symptom | What it usually means |
|---|---|
| Output contains instructions, links or contact details nobody asked for | Text inside a retrieved document or web page was treated as an instruction |
| The assistant reveals its system prompt or internal rules | Instruction boundaries are weak or absent |
| An agent calls a tool outside the task, or with unexpected arguments | Excessive agency and missing allow-lists |
| One customer's data appears in another customer's answer | Retrieval filters or tenant isolation failed |
| A knowledge-base answer cites a document that was never approved | Poisoned or unreviewed content entered the index |
| The agent loops, opens tickets or spends far more than usual | Instructions triggered repeated tool use |
| Policy language is bypassed through translation, encoding or role-play | Filters check keywords rather than behaviour |
Diagnose before you patch
- Capture the exact user input, the assembled prompt, the retrieved passages and the tool calls. Without this, you are guessing.
- Identify the trust boundary that was crossed: document, web page, email, chat message or another user's content.
- Identify the capability the instruction reached: a read tool, a write tool, a notification channel or the final answer.
- Scope the incident. Search logs for the same source document, sender or URL across other sessions.
- Preserve evidence before deleting the content or rotating keys.
1. Direct instruction override
Likely cause: the user or a document says to ignore previous instructions, and the system has no clear separation between policy and content.
Fix: keep system rules in a channel the user cannot rewrite, and repeat the critical constraints at the point of action. Validate outputs against the task schema rather than trusting prose. For high-impact actions, require a structured confirmation that names the action, the target and the amount.
Prevention: treat every instruction-shaped string in user content as data. Test with known override phrases, including polite and indirect versions.
2. Indirect injection through retrieved content
Likely cause: a PDF, web page, email or ticket contains hidden or visible text aimed at the model, and the retrieval step passes it through without provenance checks.
Fix: label retrieved content as untrusted, strip active elements, and restrict which domains and repositories can be indexed. Show the source next to the answer so a reviewer can see where a claim came from.
Prevention: maintain a source allow-list, scan new documents before indexing, and re-scan when a document changes. A document that was safe last month is not automatically safe today.
3. Tool misuse and excessive agency
Likely cause: the agent has broad credentials, can chain tools freely, and has no approval step for actions that move money, change records or send messages outside the company.
Fix: give each agent the minimum scope it needs, use per-user identity rather than a shared service account, allow-list tool arguments, and require human approval for irreversible actions. Add rate limits and a hard stop on repeated calls.
Prevention: review tool permissions quarterly and after every new integration. Log the actor, tool, arguments and result for each call.
4. Data exfiltration through answers or links
Likely cause: the model can read sensitive context and render it into a response, an image URL or a link that an attacker can collect.
Fix: redact sensitive fields before they enter the prompt, block outbound links to unapproved domains, and restrict rendering of model-produced HTML. Check the output against the user's entitlements, not only the model's context.
Prevention: use synthetic canary records in the knowledge base and alert when they appear in an output. This catches leakage paths that policy review misses.
5. Poisoned knowledge and cross-tenant leakage
Likely cause: anyone can add to the index, or retrieval searches the whole corpus and filters after the fact.
Fix: enforce tenant and role filters inside the retrieval query, not after it. Require review and an owner for every document in the index. Remove superseded documents rather than relying on ranking to hide them.
Prevention: test with two accounts from different tenants and confirm that neither can retrieve the other's content, including through follow-up questions.
Containment in the first hour
- Disable the affected tool or agent, not the whole business system, if the design allows it.
- Revoke or narrow the credentials the agent used.
- Snapshot logs, prompts and retrieved content.
- Notify the system owner and the person accountable for the affected data.
- List the records read, changed or sent, and decide whether notification obligations apply.
- Patch the specific path, then re-run the attack that worked.
Controls by layer
| Layer | Control | Evidence |
|---|---|---|
| Prompt | Separate policy from user and retrieved content | Prompt template and tests |
| Retrieval | Source allow-list, provenance and tenant filters | Index configuration and test results |
| Tools | Least privilege, argument allow-lists, approvals | Permission matrix and tool-call logs |
| Output | Schema validation, redaction, link blocking | Filter rules and sample outputs |
| Identity | Per-user tokens, scoped sessions | Access review record |
| Monitoring | Anomaly alerts on tool use and data volume | Alert configuration and test |
Prevention tests to keep in the release process
- A regression corpus of injection prompts, including indirect and encoded variants.
- A canary document that instructs the model to reveal a harmless marker; alert if it appears.
- Tool-permission tests that attempt a forbidden action and expect a refusal.
- Cross-tenant retrieval tests with two real test accounts.
- A review of every new data source before it is indexed.
Metrics for the operations dashboard
- Blocked or refused actions by control type.
- Tool calls outside the allow-list, including attempts.
- Human approval overrides and the reasons recorded.
- Time from detection to containment.
- Share of indexed documents with a named owner and review date.
Where our team fits
We design and harden retrieval, tool permissions and approval flows for business agents under Business Automation. Scope and commercial terms are on the pricing page. For the underlying model of how these attacks work, read prompt injection risk in business systems explained.
Sources
- OWASP Top 10 for Large Language Model Applications, accessed 17 November 2026.
- NIST AI 600-1, Generative AI Profile, accessed 17 November 2026.
- MITRE ATLAS, adversarial threat landscape for AI systems, accessed 17 November 2026.
- NIST AI Risk Management Framework, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.