An agent can pass its evaluation and still fail the business. The test set may be too clean, the tools may be mocked, production permissions may differ from the sandbox, or the dashboard may measure model latency while nobody measures whether the task was completed. This guide is for the team already running an agent, or about to, that needs to find out why confidence and outcomes have separated.
Each problem below follows the same shape: what you see, why it happens, the fix and the change that stops it returning. Start with the failures that affect money, customer records or safety, then work outward.
Problem 1: evaluation scores are high but users complain
What you see. The scorecard improves while support tickets repeat the same failures. The team points to the evaluation report; the operations team points to customers.
Why it happens. Test cases were written by the build team, examples leaked into prompts, easy cases dominate the set, there is no holdout, or two reviewers label the same case differently.
Fix. Sample real tasks from the last quarter, include failures and handoffs, keep development and holdout sets separate, and have two reviewers agree on labels before scoring. Measure task success rather than similarity to a reference answer.
Prevention. Refresh the set when the process changes and record why each case exists. An evaluation set without a history becomes a ritual.
Problem 2: the agent passes tests but fails on real tools
What you see. The agent handles the scripted demo, then loops, times out or reports success without changing the record in production.
Why it happens. Mocks return clean data, the sandbox lacks rate limits, pagination and partial failures, permissions differ, or nobody tested what happens when a tool is unavailable.
Fix. Run contract tests against real sandbox APIs, inject errors and timeouts, test retry and idempotency, and verify that the agent stops and escalates when a tool fails.
Prevention. Keep a staging environment that mirrors production permissions and data shape, and review tool changes as part of the release.
Problem 3: a model or prompt change causes a silent regression
What you see. A small prompt edit improves one case and quietly breaks ten others. Nobody notices until a customer does.
Why it happens. Prompts are edited in production, model versions are not pinned, there is no golden set and no canary release.
Fix. Version prompts and model settings, run the evaluation set on every change, canary a small slice and keep a rollback path. Compare the canary against the current version on the same tasks.
Prevention. Treat prompts and tool descriptions as code, with review, change history and an owner.
Problem 4: dashboards are green while tasks fail
What you see. Latency and error rates look healthy, but customers are not getting the outcome they asked for.
Why it happens. Tracing covers model calls only. There is no outcome event, no handoff tracking and no business metric attached to the agent.
Fix. Trace the whole task with a trace ID. Record the tool calls, arguments, result, latency, cost and final outcome. Join that trace to the business system to confirm the action actually happened.
Prevention. Define one primary success metric per agent and review it weekly alongside the technical dashboard.
Problem 5: cost per task climbs without a visible cause
What you see. The monthly bill rises while task volume is flat. The team cannot say which tasks or which tools are responsible.
Why it happens. Retries, context growth, wrong tool loops, duplicated retrieval and verbose output all add cost without adding value.
Fix. Set a budget per task, cap the number of steps, log token and tool cost per trace, alert on outliers and review the ten most expensive tasks each week.
Prevention. Include cost in the evaluation set, not only quality. A correct outcome after eleven retries is still a design problem.
Problem 6: the agent acts outside its authority
What you see. The agent updates a record it should only read, or performs an action that normally needs a manager's approval.
Why it happens. Tool credentials are broad, there is no policy layer, consequential actions have no human approval, and test accounts carry production rights.
Fix. Apply least-privilege credentials, define an explicit allow-list of actions, add approval gates for money, customer records and deletions, and keep an audit log of every action.
Prevention. Test the denial path as carefully as the success path. An agent that cannot be stopped is not ready for production.
Problem 7: handoffs lose context
What you see. A customer is transferred to a person who has to start the conversation again. The agent's reasoning is missing and the queue has no owner.
Why it happens. There is no transcript, no reason code and no structured summary. The agent may close the task before a person sees it.
Fix. Pass a structured summary and the trace link, define what a good handoff contains, assign the queue to a team and measure resolution after handoff.
Prevention. Include handoff cases in the evaluation set and review a sample every week.
Problem 8: an incident cannot be investigated
What you see. A customer complains about an action, and the team cannot reconstruct which prompt, model, tool or data produced it.
Why it happens. Log retention is short, trace IDs are missing, personal data was not redacted and the model version was not recorded.
Fix. Retain traces for the agreed period, redact sensitive fields, record model and prompt versions, and write a short incident runbook with an owner.
Prevention. Run one tabletop exercise per quarter using a real failed task. The gaps will be obvious within an hour.
A diagnostic sequence
Pick ten recent failures and pull the trace for each. Classify the cause as a test gap, tool gap, policy gap, data gap or user-experience gap. Fix the most frequent class first, add each failure to the evaluation set, and re-run after the change. Record the result so the next review starts from evidence rather than anecdote.
Prevention that holds
Keep the evaluation set small enough to run on every change but broad enough to include the ugly cases. Review transcripts weekly, not only dashboards. Give one person authority over the agent's scope and cost. Document what the agent must never do, and test that it cannot.
We design and operate agents with evaluation, tracing and approval controls under Business Automation. The pricing page explains how pilots and ongoing support are scoped. For the concepts behind these checks, read AI agent evaluation and monitoring explained.
Sources
- National Institute of Standards and Technology, AI Risk Management Framework, accessed 17 November 2026.
- OpenTelemetry, Generative AI semantic conventions, accessed 17 November 2026.
- OWASP, Top 10 for Large Language Model Applications, accessed 17 November 2026.
- Gartner, Applying uniform governance across AI agents will lead to enterprise AI agent failure, accessed 17 November 2026.
Responses (0)
No comments yet. Add the first one below.
Leave a response
Your email address will not be published. Required fields are marked *.