An agent runs for minutes, calls a model that answers differently each time, and writes to systems that remember. A timeout, a retry or a restart in the middle can leave a duplicate write, a lost run or a wrong answer that looks fine. This site climbs, one mechanism at a time, from what reliable means to timeouts, idempotency, queues, checkpoints, circuit breakers, service level objectives and fault injection, asking one question at each step: what is still true after this fails?
A retry is not a recovery. When the call to a roof-assessment provider times out, the agent cannot know whether the first attempt was applied. Try again carelessly and the underwriter sees two recommendations; give up and the run is lost. Distributed systems solved these problems long ago, but an agent adds a model that answers differently each time and runs that last for minutes.
How long to wait, when to give up, how to try again without a second write: deadlines, backoff with jitter, retry budgets and idempotency keys.
Messages that arrive twice, runs that die in the middle and resume from the last saved step, and compensation for the steps that cannot be undone.
Stop calling a dependency that is down, share one quota fairly, and set objectives with an error budget and alerts worth waking for.
A recovery path that never ran is a guess. Injected faults with invariants, a replayable game day, runbooks, and failures that become permanent tests.