Building a proof-of-concept LLM agent is deceptively easy. You make an API call, the model returns something clever, and it feels like magic. Then you try to run it in production and you quickly discover that "clever" and "reliable" are very different things.
Here's what I've learned building agents that actually hold up.
The core problem: non-determinism
Regular software is deterministic. You call a function with the same inputs and you get the same output. LLMs are not. The same prompt can return different results, and sometimes those results are subtly wrong — not obviously broken, just slightly off in a way that corrupts downstream logic.
This changes how you have to engineer around them.
Tool use (function calling)
Most agents need to interact with the world — query a database, call an API, run some code. The modern way to do this is structured tool use, where you define a set of functions the model can call and let it decide when to use them.
const tools = [
{
name: "search_documents",
description: "Search the knowledge base for relevant documents",
input_schema: {
type: "object",
properties: {
query: { type: "string" },
limit: { type: "number" }
},
required: ["query"]
}
}
];The model returns a structured tool call rather than free text, which you then execute and feed back into the conversation. This is far more reliable than asking the model to describe what it wants to do in prose and parsing the response yourself.
One thing I've learned: write your tool descriptions as carefully as you'd write a function's documentation. The model uses the description field to decide when to call a tool — vague descriptions lead to wrong tool selections.
Structured outputs
When you need the model to return data your code will consume, don't ask for free text and parse it. Use JSON schema to enforce the shape of the response.
const response = await client.messages.create({
model: "claude-sonnet-4-6",
max_tokens: 1024,
tools: [{
name: "extract_result",
input_schema: {
type: "object",
properties: {
confidence: { type: "number", minimum: 0, maximum: 1 },
summary: { type: "string" },
action: { type: "string", enum: ["approve", "reject", "escalate"] }
},
required: ["confidence", "summary", "action"]
}
}],
tool_choice: { type: "tool", name: "extract_result" },
messages: [...]
});Setting tool_choice to a specific tool forces the model to respond in that schema. No more regex parsing, no more "the model said 'Yes' instead of 'true'".
Retries and fallbacks
LLM calls fail. The API times out, the model returns a malformed response, or a tool execution throws an error mid-chain. You need retry logic, and it needs to be smarter than just sleeping and trying again.
A few patterns that work:
- Retry with context — if a tool call fails, tell the model what failed and let it try a different approach rather than blindly retrying the same call.
- Fallback models — have a cheaper, faster model handle simple steps and reserve the more capable (slower, more expensive) model for steps that actually need it.
- Circuit breakers — if the same agent step fails repeatedly, stop and surface the error rather than looping indefinitely. Infinite retry loops burn money and never recover gracefully.
Observability
This is the one I'd do first if I were starting over. Agents are multi-step processes and when something goes wrong, you need to see the full chain of what happened — every prompt, every model response, every tool call and its result.
OpenTelemetry works well here (I've written about it before). Trace each agent run as a root span and add child spans for each step. Log the full prompt and response at each span. It makes debugging a 5-minute job instead of a 2-hour one.
Conclusion
LLM agents aren't hard to build, but they require the same engineering rigour as any other distributed system. Enforce structure at the model boundary, design for failure at every step, and invest in observability early. The non-determinism doesn't go away — you just build systems that handle it gracefully.