"Retries, timeouts and idempotency: the plumbing that keeps agents alive at 3am"
Your agent demo worked. Production is where it waits out a ten-minute default timeout, double-sends an email after a retry, and hammers a provider that is already down. The distributed-systems basics — timeout budgets, backoff with jitter, idempotency keys, circuit breakers, dead-letter queues — apply doubly to LLM systems.
AI engineering
Ahmed
- #AIengineering
- #Reliability
- #Agents
- #Production
17 July 2026
Your agent worked in the demo. It will die in production, at 3am, and the failure will not be the model being stupid — it will be a hung worker, a double-sent email, or a fleet of retries hammering a provider that is already down. If you lead a team shipping LLM systems, the risks that should keep you up at night are not prompt quality. They are the distributed-systems basics that have bitten every integration since the first flaky HTTP call — and they bite harder here, because the model is a slow, expensive, flaky dependency that can also trigger side effects in the real world.