Engineering
The demo works on the first try. Then the agent meets a real API — one that rate-limits, times out, and returns cryptic errors — and every sad path the demo never hit becomes a Tuesday. Here are the five failure modes that turn an agent into an incident, and what actually fixes each.
klanex engineering · August 2026 · ~7 min read
An agent demo calls one reliable API, once, on the happy path, while you watch. Production is the opposite of all four of those. The agent calls flaky third-party APIs, thousands of times, on paths no one watched, while you sleep. The reasoning that looked magical in the demo is still fine — the model is good at deciding what to do. What breaks is the part in between the decision and the API: the execution.
Almost every "our agent is flaky in prod" story is one of the same five failures. None of them are exotic. They're the default behavior of an LLM wired straight into a real API.
LLMs invent fields that don't exist, drop required ones, and fat-finger enum
values — confidently. Wired directly to the API, that produces a 400
three steps into a five-step workflow, with the target's cryptic error message
stranded far from the model that caused it. The agent gets back "invalid request"
and has to reverse-engineer which of its own tokens was wrong.
The fix is to validate the intent before it leaves — against
a JSON Schema you control — and to return the failure in a form the model can act
on, not a form a human has to decode. A raw 400 is a dead end; a
structured hint pointing at the exact missing field is a self-correction loop.
# calling the API directly — the model has to decode this:
400 {"error": {"type": "invalid_request_error", "param": null,
"message": "You cannot pass both amount and amount_off."}}
# through a schema gate — written to be pasted back into the model:
422 {"error": {"code": "SCHEMA_INVALID",
"llm_hint": "The payload is missing required property 'charge_id'.
Add it and resubmit; do not change amount."}}
429, 503, and connection timeouts are the normal weather
of third-party APIs, not edge cases. A naive agent hits one and either stalls,
crashes the run, or — worse — retries immediately and hammers an already-struggling
endpoint until it opens the circuit for everyone. So teams hand-roll retry logic
into every single tool, slightly differently each time, and none of it survives the
process being recycled.
Retries belong in infrastructure, not in the agent loop. Exponential backoff spreads the load instead of amplifying it; a per-host circuit breaker stops throwing requests at a target that's already down; and a dead-letter path keeps a permanently broken call from silently wedging forever. The agent shouldn't know or care that attempt three succeeded after two backoffs.
Agents reason for 30 to 60 seconds mid-task. If the tool call is a synchronous, held-open HTTP request, that connection is ticking toward a timeout the whole time — and when it fires, it takes the agent's operational context with it. The deeper version of this problem: any retry state that lives in the agent process dies when the pod is recycled or the request is cancelled. The in-flight call simply vanishes, and nothing knows it was supposed to happen.
The fix is to make execution asynchronous and durable. Submit the intent, get an ID back in milliseconds, and let the actual call outlive the agent that requested it — its state persisted, its retries owned by a queue. The agent moves on; the result arrives later by webhook or a poll.
This is the one that costs real money. A network blip drops the response to a call that actually succeeded; the agent (or your retry wrapper) tries again; and now you've issued two refunds, sent two emails, or created two tickets. Retries and side effects are fundamentally in tension, and "just don't retry" isn't an option when the whole point was resilience.
The fix is an idempotency key. Tag the intent with a key derived from the thing you're acting on, and a second attempt with the same key returns the original result instead of executing again. Exactly-once, without the agent having to reason about whether its last call went through.
The most expensive failures are the quiet ones. Something breaks overnight, there's no audit trail of what the agent tried, and the only way to recover is to re-prompt the LLM to rebuild the call from scratch — expensive, non-deterministic, and often impossible because the original context is long gone. And the whole time, the raw production credentials the agent used were sitting in the model's environment, one prompt-injection away from leaking.
The fix is three things you should never build per-agent: a queryable audit trail of every intent and attempt; byte-exact replay so recovery is one call, not a fresh trip through the model; and credentials sealed out of the model's reach entirely. (We wrote up the credential half of this separately — how klanex handles agent credentials.)
Notice what none of these are: none are failures of reasoning. The model picked the right tool and the right arguments. Everything that broke happened in the gap between the decision and the effect — validation, retries, durability, idempotency, audit. That gap is a distinct layer of the stack, and it's the same layer every team rebuilds, badly, one tool at a time.
That layer is exactly what klanex is. You submit the tool call as an intent; klanex schema-gates it, executes it asynchronously with backoff and per-host circuit breakers, enforces idempotency, seals the credentials, keeps the audit trail, and reports back with a signed webhook — the full list of things your agent can't reliably provide for itself. Your agent's job is to reason. Something else's job is to make the call survive.
Start free — 1,000 executions a month, the full reliability engine, no credit card to explore.