When to Reach for an Agent, and When to Ship the Workflow

For the last eighteen months, every architecture review I have sat in has ended the same way. Someone proposes a workflow. Someone else asks whether it should be an agent. The discussion goes in circles, because we are using the word "agent" as if it were a category, when in production it is a commitment. The commitment to absorb non-determinism, to pay for it in latency and cost, and to operate it with discipline that the rest of the system does not need.

This article is a decision framework. Not a comparison of frameworks, not a list of agent types. The single question: given this workflow, is an LLM-driven agent the right primitive, or should this be code. The answer is almost always "code". The interesting question is when it isn't.

A note on scope. Everything that follows assumes a bounded, tool-calling, single-or-few-turn agent: a small model call embedded in a workflow, with a verifier, a budget, and a clear input-output contract. Long-horizon autonomous agents — coding agents, computer-use agents, multi-day research loops — have a different reliability shape, a different cost model, and a different decision framework. I am not covering that ground here. If you are evaluating whether to ship an autonomous coding agent into your engineering workflow, this is not the article for that decision.

I am going to assume you have shipped both. If you have not, the framing may feel academic. Skip to the section called The decision matrix, work through it on a real workflow, and come back to the rest.

What an agent actually is

An LLM agent is a system where the next step is decided by a language model at runtime, given a goal, a set of tools, and a history of what has already happened. That is the entire definition. Everything else — ReAct loops, planner-executor splits, multi-agent debate, tool-calling, MCP, memory hierarchies — is implementation detail.

Compare this to a workflow. A workflow is a system where the next step is decided by code, given the current state. The state machine is explicit. The transitions are deterministic given the input. You can draw the whole thing on a whiteboard and a junior engineer can implement it without asking a single question about the domain.

The cost difference is not subtle. In the systems I have operated, an agent run is an order of magnitude more expensive than a deterministic pipeline that does the same visible thing, often more. The latency difference follows the same shape — seconds for the agent, milliseconds for the code. The failure profile is the larger gap. A pipeline either works or it does not. An agent succeeds in some sense and fails in a way you did not anticipate a meaningful fraction of the time — low single digits for well-scoped tasks, much higher for open-ended ones, and the failure rate is sensitive to the exact distribution of inputs you serve.

None of this makes agents bad. It makes them expensive and unreliable primitives. You reach for them when the alternative is worse.

The cost of non-determinism

Here is the part most architecture reviews skip. Every time you let a model decide the next step, you are not just paying for the tokens. You are paying for the operational surface that comes with non-deterministic behavior.

You are committing to evaluation infrastructure. When the next step is decided by code, you have unit tests, integration tests, and a clear pass/fail at each boundary. When the next step is decided by a model, you need an evaluation harness that runs against sampled production traffic, an LLM-as-judge or human review pipeline, and a way to detect regressions when the model provider ships a quiet update at 3am. None of that is optional. Every team I have seen skip it has eventually rebuilt it under pressure.

You are committing to observability at a different granularity. A 200 OK on an agent endpoint tells you almost nothing. The agent may have succeeded. The agent may have hallucinated a tool call that happened to be valid JSON. The agent may have called the right tool with the wrong arguments. The agent may have looped twenty times before producing something the user accepts. You need traces that include the prompt, the tool calls, the retrieval results, the validator outputs, and the final response. That is a real engineering investment, not a library you install.

You are committing to a different incident response playbook. When a pipeline breaks, you read the stack trace, find the bad input, and patch the code. When an agent breaks, you read the failed eval, sample a hundred production traces, find a pattern in the model output, write a prompt patch, run the eval again, and ship. The mean time to recovery is hours, not minutes. You can compress it with discipline, but you cannot reduce it to the level of a normal service.

You are committing to cost governance. An agent without token budgets will find a way to make your bill look like a typo. This is not a joke. I have seen a single user interaction cost $47 because a prompt template was slightly less specific than its predecessor and the agent decided to make forty tool calls. The same workflow as code would have cost fractions of a cent.

You are committing to an adversarial threat model. This is the one most architecture reviews skip, and it is the one that bites teams in production. The moment your agent reads user-supplied content, fetches a URL, or ingests a document from an external system, that content becomes part of the prompt. A well-crafted string in a customer email, a hidden instruction in a web page, a payload in a PDF — any of these can attempt to override the system prompt, exfiltrate data through tool calls, or steer the agent toward actions the user did not request. The shape of attack is exactly the shape you would recommend for a high-value agent: read a document, extract fields, write to a system of record. The injection surface lives at the boundary between unstructured input and structured action. You need to think about it the same way you think about input validation in a web framework, except the input is untrusted natural language, the validator is the model itself, and the model is not a reliable validator of its own inputs. Treat every retrieved or tool-returned string as untrusted. Isolate the data channel from the instruction channel — by sandboxing, by structural markers, by never quoting retrieved content into the instruction stream — and run a second model pass or, better, a deterministic check on the action before any tool call touches state. Detecting instructions in arbitrary natural language is still an open problem in the security literature, so do not rely on prompt-level scrubbing as a defensive layer; treat it at best as a reduction in attack surface, never as an elimination.

If you cannot commit to all five of these, do not ship an agent. Ship the workflow. The user will not notice the difference, and your on-call rotation will thank you.

Where agents actually win

Agents are the right primitive in a narrow but real set of situations. The pattern across all of them is the same: the input space is open, the solution space is open, and the cost of building an explicit state machine exceeds the cost of operating a probabilistic one.

The input is unstructured and varies in ways you cannot enumerate. A user types a question into a search bar. A support inbox receives emails in twelve languages with thirty different intents. A document is uploaded in an unknown format with unknown structure. If you try to build the explicit state machine, you end up with hundreds of branches, each one covering a case the model would have handled natively. The agent is cheaper than the rules engine.

The task requires synthesis across heterogeneous sources. A research workflow that pulls from a dozen APIs, a documentation corpus, and a few spreadsheets, and produces a one-page summary. The combinatorial space of possible source combinations is too large to enumerate. The output is not strictly verifiable but it is groundable — you can check that the claims are supported by the sources.

The task is low-stakes and high-volume, where a small failure rate is acceptable. Internal tooling, draft generation, content summarization for human review, exploratory data analysis. The user is in the loop and can correct the agent. A 5 percent failure rate is fine because the user would have spent ten minutes on the task anyway.

The task definition itself is moving faster than you can ship code. A sales team that wants a new lead-scoring rubric every quarter. A documentation team that wants a new FAQ entry every week. A security team that wants new detection rules every time a new CVE drops. If the rate of change exceeds your deployment cadence, the agent absorbs the change at runtime. The cost is the non-determinism tax; the benefit is that the workflow does not bottleneck on engineering.

Notice what is not on this list. None of these are agent use cases, and treating them as such is where most agent projects lose money:

• Customer-facing workflows where the user has no recourse if the agent is wrong.
• Mission-critical automations where a small fraction of silent failure is unacceptable.
• Anything where the cost of a wrong answer exceeds the cost of a human doing the task.
• Anything where the input is structured enough that a rules engine would work.

The decision matrix

When you are evaluating a specific workflow, walk through these questions in order. Some are hard gates — a "no" disqualifies the agent and you should ship the workflow instead. Others are soft gates — a "no" raises the bar on what you have to commit to operationally, but does not by itself rule the agent out. Treat the hard gates as a stop sign and the soft gates as a checklist of things you have to be honest about before you ship.

1. Can you write down the input space explicitly? (hard) If yes, you do not need an agent. You need a parser and a switch statement. If the input is "an email from a customer" and the space of possible emails is unbounded, you do not have an explicit input space. The agent earns its keep here.

2. Is the success criterion verifiable by code? (soft) If yes, use the verifier. The agent produces a candidate output, the verifier checks it, and the loop terminates when the verifier approves. This is the verifier-driven agent pattern, and in production it has held up as one of the more reliable ways to constrain non-determinism. If no code verifier exists, the agent is operating without a programmatic floor — but this is not, on its own, a reason to abandon the agent. It is a reason to substitute a different floor. A human in the loop reviewing the output is a floor, and it is the legitimate floor for low-stakes tasks like draft generation and summarization-for-review that the previous section explicitly recommends as agent use cases. Whether that floor holds is a question of cost and operational discipline, which the remaining questions help you reason about.

3. What is the cost of a silent wrong answer? (hard) If the cost is high — a wrong medical summary, a wrong financial report, a wrong security classification — the answer is no agent. A human in the loop, or no automation. If the cost is a confused user who can ask a follow-up, the agent is acceptable.

4. Can you commit to the operational surface? (soft) Evaluation harness, observability, runbook, cost governance, and an adversarial threat model. If you cannot commit to all five, the agent is going to make your on-call rotation miserable. Pick a smaller workflow and ship the agent there, where the blast radius of an incident is contained.

5. Is the workflow rate-of-change higher than your deployment cadence? (soft) If yes, the agent is justified by the deployment velocity gain. If no, ship the code. You will spend less over a year.

6. Is the workflow at the boundary of structured and unstructured data? (soft) The strongest agent use cases are at the boundary. "Read this PDF, extract the invoice number, look it up in the system, update the record, send a confirmation." Each step is a tool call, the glue is the model. The verifier has to match the decision: a successful database write proves output validity — the call landed, the schema matched, the record updated — but it does not prove that the extracted number was the right number. A wrong invoice number, in the right format, will still write successfully. For that you need a semantic cross-check, against another field on the same document or a checksum from the issuing system, before the write is allowed. If the workflow is 100 percent structured, use code. If the workflow is 100 percent unstructured, you are probably better off with a human.

Walk through the matrix. A hard-gate "no" at any point means ship the workflow instead. A soft-gate "no" means the agent is still in the running, but you have to commit to whatever compensating control the question called out. If you walked through all six and the answer is still "agent", build the agent. If at any hard gate the answer was "no", stop.

Measuring reliability correctly

Before we get into how to build the agent that the matrix says is justified, we need a shared vocabulary for what "reliable" even means here. The reliability target for an agent is not one number. It is at least three, stacked, and they have to be tracked separately because they fail in different ways and have different remediations.

Service-level reliability. Did the request hit the agent and come back with a non-error response in a reasonable time. This is the layer your existing tooling already covers. The HTTP success rate, the p95 latency, the deploy success rate. Necessary but not sufficient. A 99.5 percent target here is conventional and reasonable.

Output validity. Did the agent return something that conforms to the contract it was supposed to return. JSON that parses. Tool calls with the right schema. Outputs that pass the type check before they get rendered. A 200 with malformed JSON is not a success. The target here should usually be tighter than the service-level reliability, because the failures here often surface to the user as broken UI. 99.9 percent is a reasonable starting point.

Task success. Did the agent actually do the thing the user wanted. This is the layer that takes real eval work to measure, because "did the user get value" is fuzzier than "did the JSON parse." The tools for this in 2026 are evals run on a sample of production traffic, with grading by either a human, a verifier program, or another LLM. The target here is product-dependent. In the systems I have worked on, the floor was set by the cost of a user-visible failure and the time it would take to detect a regression. For low-stakes internal tooling, somewhere in the high 90s is realistic. For high-stakes customer-facing workflows, the bar has to be higher — and is often unreachable without a human in the loop, which is why question three of the decision matrix tells you not to automate these at all.

The reason all three are needed is that they fail independently. A model regression can collapse task success while service-level reliability stays at 100 percent. A bad deploy can collapse service-level reliability while task success is unaffected. A schema change can collapse output validity while the other two are fine. If you only track one, you get a partial view of reality, and partial views are how customers churn while your dashboard is green.

The other adjustment that the SRE playbook does not anticipate is that error budgets for agent workflows burn in spikes, not in smooth curves. A model regression after a provider-side update is not independent. It hits every request in the affected class until you switch models. Alert on the rate of burn, not on the total. The total tells you the story after the fact. The rate tells you the story while there is still time to act.

These three layers come up again and again in what follows. Output validity is the contract a schema validator enforces. Task success is the contract an eval harness enforces. A system can be at 100 percent on one and broken on the other. The hybrid architecture in the next section is essentially an argument for keeping these two contracts on different sides of a clean boundary.

The hybrid architecture

The most reliable production systems I have seen are not pure agents. They are deterministic workflows with a small, well-defined agent step embedded at the critical decision points. The pattern looks like this.

Two agent steps, surrounded by deterministic code. The agent steps are the parts where the input is genuinely open and the cost of being wrong is recoverable. The deterministic steps are the parts where the cost of being wrong is not.

Notice that the two agent steps use different verifiers, and that distinction matters. format_response produces structured output — a JSON payload, a template, a record. The failure mode there is malformed: the model produced a string that does not parse, or a JSON object that does not match the schema. A strict schema_validator catches that class of failure cleanly. classify_intent produces a label. The failure mode there is different: the model returned perfectly valid JSON with a perfectly wrong intent, with high confidence. A schema_validator happily passes it through. What you need is a semantic check — a confidence threshold below which you fall back to a rule-based classifier or escalate to a human, and an evaluation harness that measures task success on the actual classification distribution, not just output validity.

This is the same distinction the previous section drew between output validity and task success. The verifier type has to match the failure mode. Schema validation is necessary-and-sufficient for format. Schema validation is necessary-but-not-sufficient for intent.

When this pattern is done well, the system has the reliability profile of a workflow and the flexibility profile of an agent. When it is done poorly, the agent steps leak into the deterministic steps and the failure modes become impossible to reason about. The discipline is to keep the boundary explicit and to verify that the agent output conforms to the right kind of contract for the kind of decision being made.

The failure modes, named

Treating "the agent is broken" as a single failure mode is what produces incident reviews that go nowhere. The reality is a small number of distinct failure modes, each with a different signal and a different remediation. Naming them is what lets the team build a runbook.

Model regression. The provider shipped a model update and the behavior on a class of inputs changed. The output validity rate or the task success rate drops on the affected bucket. The fix is to pin to a specific model version, switch providers, or reprompt against the changed behavior. The detection is your eval running on production traffic and noticing the drop.

Tool failure. A tool the agent calls is returning errors, the wrong shape, or stale data. The output validity may stay high if the agent recovers gracefully. The task success will drop because the agent is operating on bad inputs. Detection is per-tool error rates and per-tool semantic checks. The runbook is to verify the tool independently of the agent, isolating whether the issue is the tool or the agent's use of it.

Retrieval drift. The retrieval pipeline is returning documents that are stale, irrelevant, or duplicated. The agent's outputs feel slightly off. The user does not always notice individual failures, but downstream satisfaction and conversion metrics slip in ways that are hard to attribute. Detection requires sampling retrieval results and grading them. The runbook is to verify the index freshness, the embedding pipeline, and the similarity thresholds.

Prompt regression. A change to the prompt template, often well-intentioned, has broken a class of requests. The window between deploy and detection is the danger zone. Detection is an eval that runs on every prompt change and an alert on task success rate after deploys. The runbook is to revert the prompt change and triage in a non-production environment, which is much faster if the prompt is behind a runtime flag you can flip without redeploying.

Cost spike. Something has changed in the cost shape of the work. A new prompt is longer than the old one. A bucket is escalating to the expensive model more often than expected. A user has discovered a way to drive up token usage. Detection is per-bucket and per-user cost dashboards with alerts on derivative changes. The runbook is to identify the cost source and either contain it, optimize it, or surface it as a billing issue.

Hallucination under confidence. The agent is producing outputs that look right and are wrong. This is the hardest failure mode to detect because the surface signals are clean. Detection requires either a verifier that catches the specific class of hallucination or a sampled review by a human. The runbook is to harden the verifier and to retrain or reprompt against the failure mode.

Prompt injection. Untrusted content that reached the model — a customer email, a fetched page, a retrieved document — contains instructions that override the system prompt and steer the agent toward actions the user did not request. The output looks like a normal agent response, but the goal was set by the attacker, not the user. Detection requires either a separate model pass that judges whether the agent is following the system prompt or the original user goal, or a deterministic allowlist on which tool calls are reachable from which inputs. The runbook is to trace the injected instruction back to its source, harden the data-instruction boundary, and add the new pattern to the eval suite. This is the failure mode that has the highest blast radius when it slips through, and it is the one most teams do not have a runbook for when they first ship an agent.

Each of these has a different signature, a different signal, and a different remediation. The runbook should have a section for each. Pattern matching the symptom to the failure mode is the first step. Without a named failure mode, the team is in "the agent is broken" mode, and that mode does not converge on a fix.

The tells

After a while, you start to recognize the projects that are going to be expensive. The first tell is an architecture diagram with more boxes than the system can support. Three agents, a planner, an executor, a verifier, a memory layer, a knowledge layer, an orchestrator. If you cannot fit the system on a single whiteboard, you are going to spend most of your engineering budget on coordination rather than the actual problem.

The second tell is "we'll evaluate it later". There is no version of shipping an agent without evaluation infrastructure that ends well. The eval is not optional. It is the only signal you have that the system is doing what you think it is doing. If the team is planning to add it after the demo, the demo is going to ship and the eval is going to be retrofitted under pressure, badly, six months later.

The third tell is no clear verifier. If the team cannot articulate a programmatic check that would catch a wrong output, the system is operating without a floor. It might work. It might not. The team will not know which until a user complains, and at that point the debugging will take weeks.

The fourth tell is the agent is the easy part of the project. The hard part is the data, the integrations, the eval harness, the observability, the cost governance, the runbook. If the team is spending most of its time on the prompt and not enough on the surrounding engineering, the project is going to look great in a demo and fall apart in production.

The fifth tell is the workflow is being modeled as an agent because it is more fun to build. This is the most common tell. The right primitive is a state machine with twelve states and four transitions. The team is going to build a planner, an executor, a memory layer, and three months of debugging because the state machine was less interesting than the agent.

When not to use an agent

The list is longer than the list of when to use one. Do not use an agent when the input is structured enough that a parser would work. Do not use an agent when the success criterion cannot be verified by code and the cost of being wrong is high. Do not use an agent when the workflow is rate-of-change-bound to engineering anyway. Do not use an agent when you cannot commit to the operational surface. Do not use an agent because the demo was impressive.

Do not use an agent when a small piece of code would solve the problem. The number of workflows that genuinely require an LLM is much smaller than the number of workflows that someone has built an LLM for. Most of the time, the right answer is a state machine, a few rules, a parser, and a clear contract. The state machine will be faster, cheaper, more reliable, and easier to operate. That is not a concession. It is the right answer for most problems.

When in doubt, ship the code. You can always replace it with an agent later. The reverse is much harder.

Summary

An agent is a commitment. It is a commitment to evaluation infrastructure, observability, incident response, cost governance, and an adversarial threat model. It is a commitment to a reliability model that is fundamentally different from the rest of the system. The decision to use an agent should be made with the same rigor as the decision to use a database, a queue, or any other expensive primitive.

Use an agent when the input space is open, the task is verifiable or low-stakes, and the cost of building the explicit state machine exceeds the cost of operating the probabilistic one. Use a workflow when any of those conditions are not met. The hybrid pattern — deterministic workflow with a small, well-defined agent step at the critical decision points — is the most reliable architecture for the cases where the answer is genuinely "both".

The number of workflows that genuinely require an agent is smaller than the number that someone has built an agent for. That gap is where most of the engineering waste in this space is happening. Closing it starts with the decision matrix, and ends with a system that does the thing the user actually needs, at a cost and reliability profile the business can sustain.