Back to AI/ML Overview
♻️ Knowledge Site · Reliability for Agentic Software · Faults, Retries, Recovery

Agent Reliability EngineeringEvery step can fail. The outcome still has to be acceptable.

An agent runs for minutes, calls a model that answers differently each time, and writes to systems that remember. A timeout, a retry or a restart in the middle can leave a duplicate write, a lost run or a wrong answer that looks fine. This site climbs, one mechanism at a time, from what reliable means to timeouts, idempotency, queues, checkpoints, circuit breakers, service level objectives and fault injection, asking one question at each step: what is still true after this fails?

🎯The problem

A retry is not a recovery. When the call to a roof-assessment provider times out, the agent cannot know whether the first attempt was applied. Try again carelessly and the underwriter sees two recommendations; give up and the run is lost. Distributed systems solved these problems long ago, but an agent adds a model that answers differently each time and runs that last for minutes.

🔑The question every page asks
What is still true after this fails? Each chapter breaks the same synthetic insurance quote in one new way, then repairs it, and a fault-injection suite proves the repair holds.

🗺️What is inside

Timeouts, retries and idempotency

How long to wait, when to give up, how to try again without a second write: deadlines, backoff with jitter, retry budgets and idempotency keys.

Queues, checkpoints and sagas

Messages that arrive twice, runs that die in the middle and resume from the last saved step, and compensation for the steps that cannot be undone.

Breakers, limits and SLOs

Stop calling a dependency that is down, share one quota fairly, and set objectives with an error budget and alerts worth waking for.

Fault injection and incidents

A recovery path that never ran is a guess. Injected faults with invariants, a replayable game day, runbooks, and failures that become permanent tests.

🖱️How to read it

  • New to reliability: read in order. Each chapter starts in plain English with one tiny example.
  • Experienced: jump to idempotency, checkpoints, fault injection or the insurance lab. The rigorous part sits in folded blocks.
  • Fourteen interactive labs, each with a plain table next to it. Every diagram is clickable.
  • The example is synthetic and its agent is a scripted stand-in. Results show how the mechanism works, not how good any model is.