A SaaS company built an onboarding agent the conventional way: an LLM, a system prompt, a list of tools, and a while-loop that kept calling the model until it said it was done. The demo worked. The agent could read a new customer’s signup form, create their workspace, invite their teammates, configure their integrations, and send a welcome email - all from a single conversational prompt.
In production it failed in ways the team did not have words for. Sometimes the agent skipped sending the welcome email. Sometimes it invited teammates twice. Sometimes it created the workspace, started configuring integrations, then circled back to “let me first create the workspace” and tried to make a duplicate. One out of every twenty runs ended in a state the team could not explain by reading the trace.
The bug was not the prompt. The bug was that there was no defined workflow at all. The agent was a stateless loop being asked to remember what it had done by re-reading its own conversation history. Of course it lost track. Conversation history is not state. It is a transcript.
The fix was to delete the loop and model the workflow as an explicit state machine - “intake → workspace_created → teammates_invited → integrations_configured → email_sent → done,” with each state having defined entry conditions, defined exit conditions, and defined error transitions. The LLM still did the actual work, but it no longer chose the workflow. Three weeks of incidents stopped overnight.
This is the central architectural question for any production AI agent: explicit state machine or prompt loop. This guide is for engineers and CTOs choosing where on that spectrum their agent should live. It is part of the larger question of why your AI experiments are failing - the gap between an impressive demo and a system that does the same thing the same way every time.
The Two Schools
Strip away the framework branding and there are two fundamentally different ways to structure an agent.
The prompt-loop school. A single LLM call inside a while-loop. The prompt describes the goal, the tools, and the rules. The LLM decides what to do next on each iteration. State is whatever the LLM remembers from its conversation history. Examples: classic ReAct agents, single-prompt assistants, most “give the LLM tools and let it figure it out” patterns.
The state-machine school. An explicit graph of states and transitions. Each state defines what should happen there, what conditions move to the next state, and how errors are handled. The LLM is invoked inside states to do specific work, but it does not choose what state comes next - the transition logic does. Examples: LangGraph’s StateGraph, Temporal workflows with conditional branches, any agent built around an explicit FSM library.
This is not a vendor question. It is an architectural choice that exists in every framework. LangGraph makes it cleaner because it ships state machines as a first-class primitive, but you can build state-machine agents on Temporal, Inngest, raw Python, or a vendor-neutral runtime. You can also build prompt-loop agents inside LangGraph if you want to. The framework is downstream of the choice.
The Honest Tradeoff
Prompt loops are faster to build, more flexible at runtime, and degrade gracefully when the input does not match the expected shape. State machines are more reliable, more debuggable, and dramatically easier to reason about - at the cost of upfront design work and less flexibility when the workload changes. The wrong choice for your workload is the more expensive one.
Why Prompts Are Not State
The deepest reason prompt loops fail in production is that prompts are stateless. An LLM call runs, produces output, and disappears with no memory of what happened before and no way to inspect what went wrong, causing agents to crash mid-execution with no idea why they failed (Medium, 2026).
When you build “state” by stuffing conversation history into the next prompt, you are asking the model to recompute its own state every turn from a noisy transcript. This works for short conversations. It collapses for long ones.
Three specific failure modes consistently appear:
Re-doing completed work. The model reads its history, gets confused about what it actually completed, and tries to do it again. This is the duplicate-workspace bug. The model “knows” it should create a workspace because the prompt says to, and the conversation history is ambiguous about whether the previous turn actually succeeded.
Skipping required work. The model sees a successful tool call buried 20 turns back and treats the whole conversation as “we did the thing” - even when subsequent steps are required. This is the skipped-email bug.
Losing the plan. The model started with a 7-step plan, executed steps 1-3, then on turn 4 decided to “first” go do step 7 because the prompt mentioned it. The plan is whatever the model decided most recently, not what was decided originally.
These are not prompt-engineering problems. You cannot prompt your way around statelessness. The model is doing exactly what you asked - running an LLM call against a context window. The fix is to make state a first-class object, not an emergent property of a transcript.
What Explicit State Looks Like
A state-machine agent has three explicit primitives.
1. A State Object
A typed, structured representation of where the workflow is right now. Not a chat transcript. A schema: { workspace_id: str | null, teammates_invited: list[str], integrations_configured: dict, email_sent: bool, current_step: Literal["intake", "workspace", "invites", "integrations", "email", "done"], errors: list[Error] }.
Frameworks like LangGraph treat state as a first-class object shared across nodes, where each node returns a partial state update, and reducers can merge values coming back from parallel branches (Medium, 2026). The implementation detail matters less than the concept: state is data, not text.
2. Nodes
Functions that run inside a state and produce a state update. A node knows what it consumes from state, what it writes to state, and what side effects it produces. A node is typically a small, focused piece of work - one LLM call, one tool invocation, one database write, one validation check. Long nodes are a smell. Break them up.
3. Edges
Transitions between nodes. Edges can be linear (“after node A, always go to node B”), conditional (“after node A, if state.confidence > 0.8 go to B, else go to C”), or cyclical (“loop back to node A if state.retry_count < 3”). Edges are where the workflow control logic lives, and they are explicit, readable, and testable.
Edges define transitions between nodes that may be linear, conditional, or cyclical - for example, if confidence is below a certain threshold, execution can loop back for refinement (Medium, 2026). The LLM may inform a decision (by returning a structured score), but the decision itself is a deterministic edge function.
stateDiagram-v2
[*] --> intake
intake --> validate_signup
validate_signup --> create_workspace: valid
validate_signup --> human_review: invalid
create_workspace --> invite_teammates: success
create_workspace --> retry_workspace: transient error
retry_workspace --> create_workspace: retry < 3
retry_workspace --> human_review: retry >= 3
invite_teammates --> configure_integrations
configure_integrations --> send_welcome_email: all integrations ok
configure_integrations --> partial_done: some integrations failed
partial_done --> send_welcome_email: send anyway with note
send_welcome_email --> done
human_review --> done: human resolved
done --> [*] Read that diagram and you can predict every behavior the agent will exhibit. Read a prompt-loop ReAct agent’s system prompt and you cannot. That is the difference.
When to Use Each Approach
The question is not “is one better than the other?” The question is “which fits this workload?”
Use a Prompt Loop When
- The workflow is genuinely open-ended (the agent might do A, then B, then C, then loop back to A, depending on what it finds)
- The number of distinct paths is too large to enumerate (a research assistant exploring an unknown topic)
- Speed-to-prototype matters more than reliability (early discovery work, internal tools, demos)
- You have strong evals catching regressions and you are okay with iteration
Use a State Machine When
- The workflow has identifiable phases with clear entry and exit conditions
- The same input must produce the same execution path (compliance, finance, healthcare)
- You need to resume mid-workflow after a failure
- You need to insert human approval at specific points
- You need to debug production incidents by reading state history
- You need to test each phase independently
The honest summary from production practitioners: LangGraph is the right fit when your AI application needs a clear process, memory across steps, and the ability to adapt as it runs, helping you design the full path the AI should follow, including decisions, retries, and checks (Sider, 2025). Apply the same logic to whatever framework you use - if your workload looks like a defined process, model it as one.
Our own deeper-dive on the LangGraph implementation pattern is here: A Developer’s Guide to LangGraph: Building Stateful, Controllable LLM Applications. LangGraph is one good implementation of the state-machine school, but the principles apply regardless of the library you pick.
State Shape: The Decisions That Matter
Once you commit to a state machine, the biggest design decision is what goes in the state object. Get this wrong and the state machine inherits all the pain of the prompt loop.
What Belongs in State
- Workflow position: which node/state you are in
- Decisions already made: results of prior LLM calls and tool calls that downstream nodes depend on
- Idempotency markers: IDs of side effects you have already performed, so retries do not duplicate them
- Pending work: queued items, work that is in flight, work waiting on humans
- Errors and retry counts: to make retry/escalation decisions
What Does Not Belong in State
- Full LLM conversation history. Keep it in a separate log; reference it by ID. State should be compact and inspectable.
- Large blobs. Documents, images, embeddings - reference by ID, store externally.
- Anything you can recompute. State should be the minimum set of facts the workflow needs to make decisions.
The test for state shape: if you printed the state object during a production incident, would you be able to explain what the agent should do next? If yes, the shape is right. If you would need to read 40 turns of conversation to figure it out, the shape is wrong.
Mutation Discipline
In multi-node and parallel-branch workflows, state mutation gets tricky. Two nodes that run in parallel may both want to write to the same key. The solution is reducers - functions that merge state updates predictably. Most modern frameworks support this. The discipline is to think of state updates as mathematical operations on a shared object, not as imperative assignments.
What Breaks in Production
State-machine agents fail differently from prompt-loop agents. The failure modes are real and worth naming.
| Failure Mode | What It Looks Like | Mitigation |
|---|---|---|
| Workload outgrew the state shape | New requirement appears that no state field can express | Versioned state schema; migration strategy on entry |
| Too granular | 47 nodes, each doing trivial work, hard to follow | Collapse adjacent nodes; aim for nodes that map to meaningful business steps |
| Too coarse | One mega-node doing 80% of the work | Split based on natural retry boundaries |
| Edges hardcoded to a model’s quirk | ”If LLM said ‘I think’, go to node X” | Use structured outputs and edge conditions over typed scores, not text matching |
| Implicit retry loops | Cyclical edges with no exit condition | Every cycle needs a counter and a max-retry escape edge |
| State explosion | State object grows to megabytes | Move large data out of state, reference by ID |
The deeper risk: forcing every workflow into a state machine when the workload is genuinely open-ended. State machines do not free you from thinking. They make you think upfront. Workflows that genuinely cannot be specified in advance should not be state machines. Honesty about which is which matters.
State Machines Sit Inside Orchestration and Runtime
A layering point we keep returning to because teams keep flattening it.
- The orchestration pattern - supervisor, orchestrator-worker, hierarchical, swarm - decides who calls whom across multiple agents.
- The state-machine question - the topic of this guide - decides how each agent (or the whole system) structures its workflow internally.
- The durable execution question - Temporal, Inngest, Restate, DBOS, or a checkpointed framework - decides whether the workflow survives failure.
These are independent. You can run a supervisor pattern (orchestration) where the supervisor is a state machine (workflow) running on Temporal (durable execution). You can run an orchestrator-worker pattern where workers are prompt loops and the orchestrator is a state machine. The combinations matter, and getting one right does not save the others.
The teams that ship reliable agents pick deliberately at each layer. The teams that struggle conflate the layers, then can’t tell which one is failing.
The State Machine vs. Prompt Loop Decision Determines Your Reliability Ceiling
If your agents skip steps, repeat work, or behave differently across identical inputs, the problem is almost always the workflow structure - not the prompt, not the model. metacto helps mid-market and enterprise teams choose where on the state-machine spectrum each agent should live, design state shape, and ship agents that do the same thing the same way every time. Talk to us about your agent architecture.
A Closing Reframe
The conversation around AI agents focuses too much on the model and not enough on the structure around it. Whether you use Claude or GPT-5 matters less than whether your agent has a defined workflow at all. The reliability ceiling of any agent is set by its workflow architecture, not by the underlying LLM.
State machines are not a religion. They are a tool for one specific job: making workflows do the same thing the same way every time. When the workflow needs to do the same thing the same way every time, use one. When the workflow genuinely cannot be specified, do not. The honesty about which is which is the engineering work.
This is one layer of the system underneath the chat box - the gap between an impressive AI pilot and a production-ready agent that runs unattended for years. The state machine you do or do not draw is the design document for the system you will eventually have to debug at 2 a.m. Draw it deliberately - or bring in our Operational AI practice to design it with you.
Frequently Asked Questions
What is an AI agent state machine?
An AI agent state machine is an explicit graph of states and transitions that defines how an agent's workflow progresses. Each state has defined entry conditions, exit conditions, and error transitions. The LLM is invoked inside states to do specific work (read a form, draft an email, classify an intent), but the LLM does not choose what state comes next - explicit transition logic does. This contrasts with a prompt-loop agent, where a single LLM call inside a while-loop decides what to do next on each iteration.
When should I use a state machine vs. a prompt loop for an AI agent?
Use a state machine when the workflow has identifiable phases with clear entry and exit conditions, when the same input must produce the same execution path (compliance, finance, healthcare), when you need to resume mid-workflow after failure, or when you need to insert human approval at specific points. Use a prompt loop when the workflow is genuinely open-ended, when the number of distinct paths is too large to enumerate, or when speed-to-prototype matters more than reliability.
Why do prompt-loop agents fail in production?
Prompts are stateless. When you build 'state' by stuffing conversation history into the next prompt, you ask the model to recompute its own state every turn from a noisy transcript. This causes three specific failure modes: re-doing completed work (the model gets confused about whether the previous turn succeeded), skipping required work (the model treats a buried successful step as 'we did the thing'), and losing the plan (the model abandons the original plan for whatever was mentioned most recently). These are not prompt-engineering problems - they are structural.
What goes into the state object of an AI agent state machine?
The state object should contain workflow position (which state you are in), decisions already made (results of prior LLM and tool calls that downstream nodes depend on), idempotency markers (IDs of side effects already performed so retries do not duplicate them), pending work (queued items, in-flight work, work waiting on humans), and errors and retry counts. Do not put full LLM conversation history, large blobs, or anything you can recompute in state. Keep state compact and inspectable.
Is LangGraph the only way to build state-machine AI agents?
No. LangGraph is one well-designed implementation of the state-machine school and ships state graphs as a first-class primitive, but the architecture is framework-agnostic. You can build state-machine agents on Temporal workflows with conditional branches, on Inngest with step functions, on raw Python with an FSM library, or on any durable runtime. The framework is downstream of the choice. Pick the framework after you decide whether your workload wants a state machine at all.
How does state machine design relate to orchestration patterns and durable execution?
These are three independent layers. Orchestration patterns (supervisor, orchestrator-worker, hierarchical, swarm) decide who calls whom across multiple agents. State machine vs. prompt loop decides how each agent structures its workflow internally. Durable execution decides whether workflows survive infrastructure failure. You can mix them freely - a supervisor pattern where the supervisor is a state machine running on Temporal is a common production architecture. Conflating the layers is a common source of brittle systems.
What are the failure modes of state-machine AI agents?
State machines fail differently from prompt loops. Common failures include: workload outgrew the state shape (new requirements no state field can express - fix with versioned schemas), too granular (dozens of trivial nodes - collapse adjacent ones), too coarse (one mega-node doing 80% of the work - split on retry boundaries), edges hardcoded to a model's quirk (fix with structured outputs over text matching), implicit retry loops with no exit (every cycle needs a max-retry counter), and state explosion (state object grows to megabytes - move large data out, reference by ID). The state machine does not free you from thinking; it forces the thinking upfront.