Building Reliable AI Agents in Production Without State Loss

Why Basic LLM Tool Calling Fails in Production

Building reliable AI agents in production is a completely different challenge than writing a simple single-turn tool-call script in a Jupyter notebook. When I build prototypes, everything works beautifully because the happy path is the only path I test. But when real users interact with these systems, things break in unexpected ways.

Flaky APIs, network timeouts, rate limits, and non-deterministic model outputs are common issues in production. If your code is not designed to handle these failures, your agent will crash, lose its state, or enter infinite loops. To prevent this, we need to move past basic tutorials and adopt engineering practices that ensure reliability and predictability.

In this guide, I will share the architectural patterns and strategies I use to build production-grade agentic workflows. We will explore durable execution, advanced error recovery, deterministic routing, human oversight, and observability. These patterns will help you transition from fragile demos to systems that run reliably at scale.

Durable Execution and State Management

If you store your agent's memory in-memory, you are running a fragile system. A simple server restart or a process crash halfway through a multi-step task means your entire workflow and state vanish instantly. This is why building reliable AI agents in production requires durable execution.

Durable execution guarantees that your code runs to completion, even if the underlying server crashes. When an agent is executing a multi-step workflow, each step is saved or checkpointed. If the process restarts, the execution resumes from the exact last saved checkpoint instead of starting over.

Why is durable execution important for AI agents?

Durable execution keeps long-running workflows running. It lets multi-step agents recover from server restarts and API timeouts automatically. They never lose state or restart from scratch. This is crucial because agentic workflows can take hours or days to complete.

To implement this, you can use specialized tools. Platforms like Temporal or LangGraph checkpointers provide the infrastructure to save state at every step. For instance, LangGraph offers built-in persistence layers that save the state of your agent's graph after every node execution.

According to the official Temporal documentation, combining durable execution with a reasoning framework gives you both reliability and flexibility. You do not have to write custom database code to save variables, retry failed steps, or handle timeouts. The platform does it for you.

What is the difference between LangGraph and Temporal in agent architectures?

LangGraph handles the decision-making and reasoning. Temporal handles the durability. It guarantees that those decisions execute reliably with built-in retries and state persistence. They work together to make your agents resilient.

When I design agent architectures, I use LangGraph to define the decision-making graph. The LLM decides which path to take based on the user's input. Then, I wrap that execution in a Temporal workflow. If a third-party API fails or the server goes down, Temporal handles the retries and state recovery.

Here are the key aspects of state management you must consider:

• State persistence: Save the agent state to a database like PostgreSQL after every step.
• Thread isolation: Keep user sessions separate so one user's agent run does not interfere with another.
• Replayability: Ensure you can replay previous agent steps for debugging and auditing.

Using these practices helps you avoid losing progress. It also reduces the cost of running agents, as you do not have to pay for repeated LLM calls when a workflow restarts.

Advanced Error Handling and Graceful Recovery

When you build agentic systems, failures are guaranteed to happen. Third-party APIs will go down, rate limits will trigger, and language models will return malformed JSON. To build reliable systems, we must handle these errors gracefully.

First, categorize your errors. Not all failures are the same, and your system should treat them differently. I categorize failures into three main groups:

• Transient errors: These are temporary issues like network timeouts or HTTP 503 errors.
• Rate limits: These occur when you exceed API limits, returning HTTP 429.
• Validation errors: These happen when the LLM returns data that does not match your expected schema.

For transient errors, you should use exponential backoff with jitter. This means your system waits a bit longer after each failed attempt before retrying. It prevents your system from overwhelming the external API.

How do you handle LLM tool-calling errors in production?

You handle tool-calling errors by using strict input/output schemas with Pydantic, implementing exponential backoff for rate limits, and feeding error messages back into the LLM for self-correction. This prevents malformed outputs from breaking your entire system.

When an LLM returns a malformed JSON tool call, do not crash the system. Instead, catch the validation error and send it back to the LLM as a new message. Tell the model exactly what went wrong and ask it to correct its output.

This self-correction pattern usually fixes malformed tool calls within two retries. For example, if a tool expects an integer but the LLM sends a string, your validation layer catches the error. The agent receives a prompt like: 'The tool failed because the input was a string instead of an integer. Please try again.'

The engineering team at Baseten emphasizes that handling these errors at the application level is critical. If you rely solely on the LLM to get it right the first time, your production system will fail.

Here is a simple retry strategy you can implement:

• Wrap your tool calls in a try-catch block.
• If a validation error occurs, log the error and format it for the LLM.
• Send the error back to the LLM with a maximum retry limit of two attempts.
• If it still fails, escalate the error or trigger a fallback action.

By implementing these recovery mechanisms, you make your agents much more resilient. They can correct their own mistakes without requiring human intervention.

Deterministic Offloading and Routing

A common mistake when building AI agents is using the language model for everything. If your agent needs to calculate a date, format a string, or perform a database query, you should not ask the LLM to do it. LLMs are non-deterministic and can easily make simple arithmetic or logical mistakes.

Every LLM call adds latency, cost, and a potential point of failure. The most reliable agents are those that use language models only when necessary. For everything else, you should offload the work to standard, deterministic code.

When should you use deterministic code instead of an LLM?

You should use deterministic code for precision calculations, date comparisons, strict database lookups, and routing decisions with clear rules. This reduces latency, cost, and non-deterministic errors in your system. It makes your agentic workflows highly predictable.

For example, if your agent needs to find a customer's order history, do not ask the LLM to write a SQL query. Instead, write a deterministic Python function that fetches the order history using a customer ID. The LLM should only be responsible for extracting that customer ID from the user's prompt.

This approach is often called deterministic routing. You use standard code to guide the workflow based on specific conditions. If the user wants to check an order status, route them directly to the order tool without running an LLM planning step.

According to the team at LangChain, defining clear paths in your agent's graph improves performance. Instead of letting the LLM wander freely, you constrain its choices.

Here are some tasks you should always offload to deterministic code:

• Math calculations: Never let an LLM do arithmetic.
• Date and time math: Use standard libraries to calculate relative dates.
• Strict input validation: Use regular expressions or schema validators.
• Known API endpoints: Call specific APIs directly when the user's intent is clear.

By keeping the LLM's role narrow, you make the system faster and cheaper. You also make it much easier to test, because the deterministic parts of your code will always behave the same way.

Human-in-the-Loop Integration

Even the most advanced AI agents make mistakes. In a production environment, some mistakes are too costly to allow. If your agent is responsible for sending emails to clients, processing payments, or deleting data, you must have human oversight.

We call this Human-in-the-Loop (HITL) integration. It means the agent pauses its execution and waits for a human to review and approve its proposed action before proceeding.

When should you use human-in-the-loop oversight for AI agents?

You should use human-in-the-loop oversight whenever an agent's confidence score drops, when handling ambiguous inputs, or prior to executing irreversible, high-risk actions like payments or data deletion. This ensures that humans remain in control of critical decisions.

Implementing HITL requires a system that can pause and resume execution. This is where durable execution platforms shine. If your agent needs approval to send an invoice, the workflow state is saved, and the process pauses.

A notification is sent to a human reviewer via Slack, email, or a custom dashboard. Once the reviewer clicks 'Approve' or 'Reject', the workflow resumes. It continues from the exact point where it was paused, using the reviewer's input.

You can compare how different frameworks handle this. As discussed in the LangGraph vs Temporal comparison, both platforms offer ways to pause workflows. LangGraph uses state interrupts, while Temporal uses signals to resume paused executions.

Here are the best practices for setting up human checkpoints:

• Define clear triggers: Only pause for actions that are irreversible or high-risk.
• Provide context to reviewers: Show the reviewer the exact reasoning the agent used.
• Allow edits: Let the human reviewer edit the agent's proposed action before approving it.
• Set timeouts: Define what happens if a human does not respond within a certain timeframe.

Adding human oversight does not make your agent less useful. Instead, it builds trust. It allows you to deploy agents for complex tasks that would otherwise be too risky to automate.

Observability and Tracing

Debugging a traditional web application is straightforward because the execution paths are deterministic. Debugging an AI agent is much harder. An agent might make five different LLM calls and invoke three different tools to answer a single user query.

If the final answer is wrong, how do you know which step failed? Was it a poor prompt, a bad tool response, or a reasoning error in the third LLM call? Without proper observability, you are left guessing.

Comprehensive tracing is essential for production agents. You need to record every single interaction, prompt, completion, and tool execution in a structured way.

How do you debug multi-step agent failures?

You debug multi-step agent failures by using LLMOps tracing tools to monitor exact inputs, outputs, token usage, and latency for every step. This allows you to pinpoint exactly where a prompt regression or tool failure occurred.

Dedicated LLMOps tools like LangSmith, Phoenix, or Arize provide detailed trace trees. These tools show you the hierarchy of calls. You can see the main agent loop, the sub-agents it invoked, and the exact tools that were called.

For each step, you can inspect:

• The exact system prompt and user message sent to the LLM.
• The raw JSON response returned by the model.
• The latency and execution time of each tool call.
• The number of tokens consumed, which helps you monitor costs.

In a technical presentation on OpenAI and Temporal patterns, developers shared how combining tracing with durable execution makes debugging much easier. If a workflow fails, you can inspect the exact state at the moment of failure.

Here is what you should implement in your observability stack:

• Structured logging: Log all inputs and outputs as JSON.
• Correlation IDs: Track all steps in a single user request with a unique ID.
• Performance alerts: Set up alerts for high latency or sudden increases in token usage.
• Prompt versioning: Track which version of a prompt was used for each execution.

With these tools in place, debugging becomes systematic. You can quickly identify prompt regressions and optimize your agent's performance over time.