AI agents in production: tool use, memory, guardrails and evaluation

An AI agent is a language model running in a loop: it reads a goal, decides on an action, calls a tool, observes the result and repeats until it can answer or must stop. That loop is powerful because it lets a model look things up, take actions and correct itself. It is also where most of the risk lives, because every iteration is a chance to call the wrong tool, misread a result or drift away from the task. Here is what it takes to run AI agents in production rather than in a demo.

Teams that ship useful agents tend to converge on the same principles. This article covers four of them: tool design, memory, guardrails and evaluation.

Start narrow and make the workflow explicit

The most reliable agents are not general assistants. They own one well-defined job: resolving a class of support tickets, preparing a client onboarding file, reconciling a set of transactions. Before building, write the workflow a skilled human follows, the decisions they make and the systems they touch. Often you will find that large parts are deterministic and should simply be code. Use the model where judgement or language is needed: understanding a request, choosing among options, summarising, drafting.

A useful mental model is a spectrum. At one end sit fixed pipelines with a model call at certain steps; at the other, open-ended loops where the model chooses every step. Move along it only as far as the task requires. More autonomy means more ways to fail.

AI agent tool design is interface design

The model only knows a tool through its name, description and parameter schema. Treat these like a public API for a new colleague.

• Few, well-scoped tools. Prefer get_customer_balance(customer_id) over a generic run_sql(query). Narrow tools are easier to secure, test and choose between.
• Precise descriptions. State when to use the tool, when not to, what each parameter means, units and formats, and what errors look like.
• Strict schemas. Use typed parameters, enumerations and required fields, and validate every call on the server side. Never trust the model's arguments.
• Useful results. Return compact, structured output with clear identifiers. Truncate long results deliberately and say that you did. Return actionable error messages ("customer_id not found; search by phone with find_customer") so the model can recover.
• Idempotency and safety. Read tools should be harmless to repeat. Write tools should support idempotency keys, and destructive or financial actions should require confirmation outside the model, by a human or by a policy engine.

Memory: decide what to remember and why

"Memory" covers several different things, and mixing them causes confusion.

• Working context: the current conversation, tool results and plan. It is limited by the context window, so summarise or drop old tool outputs as the task proceeds, keeping identifiers and decisions.
• Task state: structured progress stored outside the model, such as which steps are complete, the IDs created and the approvals received. Store it in a database, not in prose, so the agent can resume after a failure.
• Long-term knowledge: documents and policies retrieved on demand through RAG.
• User memory: preferences and facts about a person across sessions. Store it explicitly and with consent, let users see and delete it, and attach provenance and dates so stale facts can expire.

The rule of thumb: anything that must be correct (amounts, IDs, statuses) lives in structured state the agent reads through tools, not in what the model happens to remember.

LLM agent guardrails: defence in depth

No single guardrail is sufficient, so layer them.

1. Input checks. Detect prompt injection patterns, out-of-scope requests and personal data that should not be processed. Content arriving from tools (web pages, emails, documents) is untrusted data: it must never be allowed to issue instructions.
2. Least privilege. The agent's credentials should allow only what its job requires, scoped to the current user's permissions. This matters more than any prompt.
3. Policy on actions. A deterministic layer between the model and its tools enforces limits: maximum amounts, allowed recipients, rate limits, business hours, and mandatory human approval for irreversible actions.
4. Output checks. Validate structured outputs against schemas. For customer-facing text, check for unsupported claims, prohibited advice (for example personalised investment recommendations where they are not permitted) and leaked internal data.
5. Budgets and stop conditions. Cap iterations, tool calls, tokens and wall-clock time. An agent that loops is an incident.
6. Audit trail. Log every prompt, tool call, argument, result and decision with a trace ID. You will need it for debugging, for compliance and for building evaluation cases.

Evaluating AI agents: test trajectories, not just answers

Agents fail in ways a final-answer metric cannot see: the right answer reached by calling a forbidden tool, or a wrong refund issued and then politely apologised for. Evaluate at three levels.

• Component tests. For each tool-selection decision, does the model pick the right tool with valid arguments? These are fast and deterministic enough to run in CI.
• Trajectory tests. Scripted scenarios with simulated tools and users, covering happy paths, ambiguous requests, tool errors, adversarial inputs and policy edge cases. Assert on the sequence: required steps happened, forbidden actions did not, and confirmation was requested before writes.
• Outcome metrics in production. Task success rate, escalation rate, average steps and cost per task, time to resolution, and human override rate. Sample conversations for review every week.

Build the scenario set from real failures. Every production incident should become a test that runs before the next release. Version prompts, tools and models together, and compare new versions against the current one on the same suite before rollout. Because model outputs vary, run important scenarios several times and track pass rates rather than single results.

Rollout pattern

A safe path to production looks like this: run in shadow mode first (the agent proposes, humans act); then assisted mode (the agent acts after one-click approval); then autonomy for low-risk actions with sampling and monitoring, while high-risk actions keep human approval. Expand scope only when the metrics support it.

Closing thought

The model is the most visible part of an agent, but reliability comes from the engineering around it: narrow tools with strict contracts, state that lives outside the model, guardrails that do not depend on the model behaving well, and evaluation that tests behaviour. Get those right and the agent becomes a dependable colleague rather than a clever demo.