How to handle failures in distributed systems: timeouts, retries and breakers

At noon the national T20 final sale opens: 1 crore people try for 1.2 lakh seats. The payment gateway never goes down. It only gets slow. Yet SeatJaldi stops answering everything, even the seat map and login.

It stays that way for 22 minutes, and about ₹45 lakh of bookings are lost. Last week, to be safe, Arjun Nair set every layer of the booking path to “try up to 3 times”. A tap on Pay passes through five layers, from the phone app to the payment client that calls the payment gateway.

“Why did everything stop if only payments were slow?” Meera Kulkarni asks. “And what do I tell users?” That afternoon Sandeep Patil walks in. He is SeatJaldi’s new SRE, a site reliability engineer, who keeps the site running.

Sandeep reads Zoya Mirza’s dashboards for ten minutes. “The payment gateway did not fail,” he says. “It took its time, and we waited for it with everything we had. Twenty-two minutes of everything. Next time, call me. Preferably not at 3 am.”

Arjun stares at the whiteboard. SeatJaldi has four servers and a load balancer. How did one slow part take all of them down?

Assume every call can fail or turn slow. The one line to keep: retries and overload feed each other, so break the loop on purpose.

• A part turns slow, so calls to it wait and pile up.
• Callers try again, adding load, so it slows further.
• A valve on the loop breaks it.

• You tap Pay, the spinner turns, you tap again, then “something went wrong”. The call had no time limit, and each tap added one.
• On a match night a food app freezes, and even the menu will not open. Every worker was waiting on one slow part.

A dead service usually fails at once, so callers move on. A slow one makes every caller wait, each holding a worker. A closed toll plaza sends cars elsewhere; a crawling one jams the roads behind it.

Little’s law: calls in flight (started, not yet finished) = calls arriving each second x seconds per call. Say each server on the booking path gets 50 payment calls a second and has 200 workers (assumptions).

• Normal, 0.2 seconds: 50 x 0.2 = 10 calls in flight.
• Slow, 10 seconds: 50 x 10 = 500 calls in flight.
• 200 workers are all busy after 200 / 50 = 4 seconds, and the seat map and login need them too.

The first guard is a time limit, like the driver who gives up and takes the old road.

A timeout decides when to stop waiting; a retry decides whether to try again. Every call needs the first, and only one layer the second. Step 3 showed why: when every layer retries, one tap becomes 243 calls.

Retry next to the failing call, as in step 6, and not the instant a try fails, like a driver honking.

Step 6 also waited a random time. If every caller waits exactly 200 milliseconds, they all return together, like drivers rejoining at once.

If the part stays sick, even careful retries keep it busy. Step 7’s valve, the traffic police, is a circuit breaker. Engineers say it “opens” when it stops calls, like a switch breaking a circuit.

Meanwhile the fan still needs an answer, like the plaza’s board: “Closed, pay later by app.” That was step 8.

Set each timeout a little above the slowest normal reply, retry next to the failing call, and open a circuit breaker after repeated failures.

• Timeout: choose how many false timeouts you accept, say 1 in 1,000, and read that percentile, p99.9. If it is 1.5 seconds (assumed), 2 seconds fits.
• Nested calls: send one “finish by” deadline with the call, so each layer knows the time left and stops in time.
• Retries: 3 tries at 5 layers is 3^5 = 243 times the load. Add a retry budget, a cap on retries: 60 a minute per server, about 2 in 100 calls. In flight stays near step 5’s 100.
• Circuit breaker: open after 5 failures in 10 seconds and wait 30 (assumed). Then let one test call through, like the police letting in a few cars. Alert when it opens, and test that state on purpose, since it rarely runs.
• Cascading failures often start with a deploy or settings change hitting every server at once, so roll changes out a few servers at a time.

What just happened? Up to 3 tries, each with a 2-second timeout, and random waits of up to 0.1, then 0.2 seconds. A 503 reply (service unavailable) is returned, not retried, so check it. The app makes the key once, when the Pay screen opens, and reuses it on every tap and retry.

Here timeout=2 limits connecting and each gap while the reply arrives, not the whole call. Rarely used paths tend to break, so fake a payment gateway that always times out, and expect 3 calls, then the error.

Try it yourself

1. 1

   Pick the numbers

   Your server gets 40 payment calls a second and has 150 workers. One layer makes 3 tries of up to 2 seconds, waiting up to 0.1, then 0.2 seconds.

2. 2

   Work it out

   If every try times out, how long at most is a worker busy, and how many calls are in flight? Now all 5 layers get “retry 3 times”: a try plus 3 retries. How many reach the payment gateway?

3. 3

   Check

   You are right with 6.3 seconds, 252 in flight (over 150 workers) and 4 x 4 x 4 x 4 x 4 = 1,024 calls.

At noon, with no timeout, each payment call held a worker for 10 seconds: 500 in flight for 200 workers. Four servers and a load balancer did not help; all waited on the same slow part.

Then 3 tries at 5 layers turned one tap into 243 calls, like honking at a jammed plaza. Sandeep’s rebuild breaks the loop on purpose: time limits, one retry layer, a circuit breaker and a fallback. “Next time the payment gateway takes its time,” he says, “we stop waiting with everything we have.”

Meera tells users: “Your seat is safe for 10 minutes. Payment will finish in a moment.” But with circuit breakers in place, bots still hit the booking API at 50x normal. How do we say “slow down” politely?