Building AI Agents for Production — What Nobody Tells You
Everyone talks about building AI agents. The demos look clean, the GitHub stars pile up, and the tutorials make it seem straightforward. Then you try to run it in production — and quickly realize that nobody told you about the hard parts.
Table of contents:
- LLM Response Inconsistency Is Not a Bug You Can Fix
- Tool Call Failures Will Happen
- Context Windows Fill Up Faster Than You Expect
- Cost Will Surprise You at the Worst Time
- Latency Is a User Experience Problem
- Error Handling Is Not Optional
- Security Cannot Be an Afterthought
- If You Cannot Observe It, You Cannot Improve It
- The Real Lesson
1. LLM Response Inconsistency Is Not a Bug You Can Fix
The most unsettling thing about building on top of LLMs is that the same prompt does not always produce the same output. You test your agent thoroughly in development, everything looks great — and then in production, it starts responding differently. Not wrong, just... different. Enough to break your downstream logic.
Most developers treat this as a bug to fix. It is not. It is a fundamental property of the system you are building on.
The goal is not to make the LLM deterministic. The goal is to make your system resilient to its non-determinism.
Foysal Zihak
What actually helps:
- Set
temperatureas low as your use case allows. For tool-calling agents,0or0.1is almost always the right choice. - Use structured outputs wherever possible. JSON mode or strict schema enforcement removes a huge class of inconsistency problems.
- Never parse LLM output with regex. Define the shape of the response upfront and validate against it.
- Build your agent logic to be tolerant of variation — do not write code that assumes a specific phrasing in the response.
2. Tool Call Failures Will Happen — Plan for Them Before Launch
Agent frameworks make tool calling look easy. In practice, two things go wrong constantly.
First: the agent calls the wrong tool. It misinterprets the user's intent, picks a tool that is semantically close but functionally wrong, and confidently executes it.
Second: the agent enters an infinite loop. A tool returns an unexpected result, the agent does not know how to proceed, and it keeps retrying — burning tokens and time until something external stops it.
// Always enforce a max iteration limit
const MAX_ITERATIONS = 10;
let iterations = 0;
while (agentIsRunning && iterations < MAX_ITERATIONS) {
const result = await agent.step();
if (result.done) break;
iterations++;
}
if (iterations >= MAX_ITERATIONS) {
// Handle gracefully — do not silently fail
await agent.abort("Max iterations reached");
}What actually helps:
- Write tool descriptions with extreme precision. The LLM decides which tool to call based on your descriptions — treat them like API contracts, not comments.
- Enforce a hard iteration limit at the runtime level, not the application level.
- Every tool should return a structured result that includes a success/failure status the agent can reason about.
- Log every tool call with its input, output, and duration. You cannot debug what you cannot see.
3. Context Windows Fill Up Faster Than You Expect
In development, your conversations are short. In production, users are thorough. They ask follow-up questions, backtrack, and provide context across many turns. Before you know it, you are hitting context window limits — and the agent starts forgetting earlier parts of the conversation.
The naive solution is to just pass the full conversation history every time. This works until it does not — and when it fails, it fails expensively.
What actually helps:
- Implement a summarization strategy. After a certain number of turns, summarize older messages into a compressed context block and replace them.
- Separate long-term memory (facts about the user, persistent state) from short-term memory (current conversation). Store long-term memory externally and inject only what is relevant.
- Track token counts on every request. Do not wait for the API to throw an error — monitor and manage proactively.
- Design your prompts to be as concise as possible. System prompts that run to 2,000 tokens are a problem you created for yourself.
4. Cost Will Surprise You at the Worst Time
Token costs seem trivial per request. Then your agent starts being used at scale — or worse, an agent loop runs unchecked overnight — and you wake up to a bill that makes no sense.
Cost management is not something you add later. It needs to be designed in from the beginning.
What actually helps:
- Set hard spending limits at the API level and enforce them in your application.
- Choose models deliberately. Use a smaller, faster model for simple classification or routing tasks. Reserve the expensive model for the steps that genuinely require it.
- Cache aggressively. If the same prompt with the same context appears more than once, you should not be paying for it twice.
- Log token usage per request, per user, and per session. You cannot optimize what you are not measuring.
5. Latency Is a User Experience Problem, Not Just a Technical One
AI agents are slow compared to traditional software. A single LLM call might take 2–5 seconds. An agent that makes three tool calls, with an LLM step in between each, can easily take 15–20 seconds to respond. Most users will not wait that long.
What actually helps:
- Stream responses wherever possible. Showing the user that something is happening — even partial output — dramatically improves perceived performance.
- Run independent tool calls in parallel. If your agent needs to fetch data from two sources before reasoning, fetch both simultaneously.
- Set response time expectations in the UI. A spinner with no context is worse than a spinner that says "Analyzing your request..."
- Use sub-50ms cold starts if you are running self-hosted agents. This is one of the reasons I built AetherCore in Go — by the time Python finishes initializing, a Go binary has already processed the first request.
6. Error Handling Is Not Optional
Traditional software fails loudly. An unhandled exception crashes the process and you know exactly where it failed. AI agents fail quietly. The agent produces a confident-sounding response that is completely wrong, and nobody notices until a user reports it — if they report it at all.
What actually helps:
- Define what "failure" looks like for every step in your agent pipeline. Do not assume the LLM will always return something usable.
- Implement fallback behaviors. If the primary tool fails, what happens? If the LLM returns malformed output, what happens?
- Never surface raw error messages to users. Handle them gracefully — log the full error internally, show the user something meaningful.
- Test failure modes explicitly. Intentionally break tools, return malformed data, and verify your agent handles it correctly.
7. Security Cannot Be an Afterthought
When an AI agent can execute tools — make API calls, write files, send messages — the attack surface grows significantly. Prompt injection, where malicious content in a tool's output tries to hijack the agent's behavior, is a real and underappreciated threat.
// Example: Sanitize tool outputs before feeding back to the LLM
function sanitizeToolResult(result: string): string {
// Strip any content that looks like system instructions
return result
.replace(/<system>/gi, "")
.replace(/ignore previous instructions/gi, "")
.trim();
}What actually helps:
- Treat every tool result as untrusted input. Sanitize it before passing it back to the LLM.
- Apply the principle of least privilege to every tool. If a tool only needs read access, do not give it write access.
- Run tool execution in a sandbox. The agent's reasoning and the tool's execution should be isolated from each other.
- Implement Zero Trust for every tool call — authenticate and authorize explicitly, every time, with no ambient permissions assumed.
8. If You Cannot Observe It, You Cannot Improve It
This is the one that bites developers most often in production. The agent works in development. You deploy it. Something goes wrong. You have no idea what happened because you have no visibility into what the agent was doing.
Observability for AI agents is fundamentally different from observability for traditional systems. You are not just tracking request latency and error rates — you need to understand the agent's reasoning, its tool usage patterns, where it hesitates, and where it fails.
What actually helps:
- Log every LLM call with its full prompt, response, token count, latency, and model version.
- Log every tool invocation with its input, output, success status, and duration.
- Use structured logging from day one. Free-form log strings are useless at scale.
- Implement distributed tracing for multi-step agent workflows.
- Build a simple internal dashboard. Even a basic view of agent activity will reveal patterns you would never notice from raw logs.
The Real Lesson
Building an AI agent that demos well takes an afternoon. Building one that runs reliably in production, under real load, with real users, over time — that is a fundamentally different engineering challenge.
The good news is that none of these problems are unsolvable. They are just problems that nobody mentions in the tutorials, because tutorials optimize for getting something working quickly, not for keeping it working reliably.
Production is not a harder version of development. It is a different discipline entirely.
Build accordingly.
Foysal Zihak
