Shipping AI agents to production: evaluation, guardrails and a cost ceiling

Building an agent that impresses a room takes an afternoon. Building one that runs unattended, against systems that hold real money and real customer data, is a different exercise — and almost none of the difference is in the prompt.

The demo has properties production does not. One run. A clean input someone chose. A human watching who can stop it. In production there are a thousand runs a day on inputs nobody curated, a retry loop that can spend a month's budget overnight, and a tool call that writes to a live system at 3am with nobody awake.

Start with the evaluation harness, not the feature

This is the single highest-return decision and the one most teams skip. Before building the agent, build the thing that tells you whether it works: a golden set of thirty to a hundred real inputs with known-good outcomes, a scoring function appropriate to the task, and a command that runs the whole set and prints a number.

Without it, every subsequent question is unanswerable. Is the new prompt better? Did switching to a cheaper model hurt? Is quality drifting? You will be answering those from anecdote and the last three examples someone happened to notice.

The evaluation harness is what converts prompt engineering from taste into measurement. It is also what lets you cut costs later without flying blind.

Draw the golden set from real inputs, including the ugly ones — the truncated email, the scanned document at an angle, the request in two languages. A golden set of tidy examples measures nothing you care about.

Make control flow explicit

There is a strong temptation to hand the model a set of tools and let it decide the sequence. It demos beautifully. It is also close to undebuggable, because when a twelve-step run fails at step nine you have no way to resume, no way to replay, and no way to answer an auditor asking what the system did.

We build workflows as explicit state graphs — LangGraph in most cases — where each node is a step with defined inputs and outputs, transitions are declared, and state is checkpointed. That buys four things that matter more than elegance:

• Resumability. A run that dies at step nine restarts at step nine, not step one, and you do not pay for the first eight again.
• Human-in-the-loop as a first-class state. The graph can pause on an approval node for hours or days without holding a process open.
• Replay. Given a recorded run you can step through exactly what happened, which is what an incident review and a compliance review both require.
• Bounded cost. A graph has a finite number of paths. A free-running loop has whatever the model decides, which is how people discover four-figure single runs.

Tools are the security boundary

An agent is only as dangerous as the tools you give it, and this is where the real engineering discipline sits. The mistake is handing an agent a broad credential and a natural-language instruction to be careful.

• One narrow tool per action, never a generic 'run this SQL' or 'call this URL'. If the agent needs to issue a refund, the tool is issue_refund(order_id, amount) with its own permission, not database write access.
• Schema-validated arguments, rejected at the boundary. Never interpolate model output into a query, a shell command or a file path.
• Hard limits in code, not in the prompt. Maximum value per action, maximum actions per run, maximum runs per hour. A prompt is a request; a limit is a guarantee.
• Least-privilege credentials scoped per tool, so a compromised or confused agent cannot reach beyond the one action it was invoked for.
• Mandatory human approval above a threshold you set with the business, not with the engineering team.

Prompt injection is a real attack, not a curiosity

If your agent reads content it did not author — inbound email, uploaded documents, web pages, support tickets — assume that content will eventually contain instructions aimed at the model. Treat everything retrieved as untrusted data. Keep the authority to act in code rather than in whatever the model concluded it should do, and never let retrieved text widen the tool set available in that run.

Earn autonomy in stages

Nothing goes straight to autonomous. Run in shadow mode first: the agent decides, a human decides, and you record both without the agent acting. After a few hundred cases you have an agreement rate, and a list of the specific cases where it was wrong. That is a real basis for a decision.

Then widen the mandate by category rather than globally. Let it act unsupervised on the invoice type where agreement is 99%, while the ambiguous category still routes to a person. Autonomy is a dial with many positions, not a switch.

Instrument cost like a production metric

Token spend behaves like nothing else in your infrastructure: it can multiply overnight from a single logic change, with no error and no alert. Track cost per run from week one, alert on deviation, and enforce a hard per-workflow ceiling in code so the failure mode is a stopped workflow rather than an invoice.

Then reduce it deliberately. Route by difficulty so simple cases hit a small model. Cache aggressively — a large fraction of prompt content is usually identical between runs. Batch where latency allows. It is common to cut spend by more than half with no measurable quality change, and the evaluation harness is what lets you prove the second half of that sentence.

The honest question first

Before any of this: does the problem need an agent? If the steps are known in advance, write the steps. Deterministic code is cheaper, faster, testable and does not need a guardrail budget. Agents earn their complexity when the sequence genuinely cannot be known ahead of time, or when a step requires reading something unstructured that no rule can parse. We have talked clients out of agent projects, and they were right to be talked out of them.