Module 6 · Doing Work Later, Safely · Lesson 25 of 35
What does idempotent mean? Making retries safe
12 minFree System design course · Beginner to intermediateLesson 6.3By Saket Bhatnagar
Researched and drafted with AI, reviewed and edited by Saket Bhatnagar.
In short
- What does idempotent mean? An operation is idempotent if doing it twice has the same effect as doing it once, like a lift button pressed five times.
- Replies get lost and messages arrive twice, so repeats will happen.
- For a payment, the app sends one unique key per payment. The server remembers the key and its first result, and answers any repeat from that.
Video lesson · coming soon
What does idempotent mean? Making retries safe
Lala, your AI mentor, will walk you through this lesson in a short animated video. Until then, everything is here to read and try.
- 01 214 people, charged twice
- 02 Five presses, one lift
- 03 The key check before the charge
- 04 Lost reply, retry, one charge
- 05 The SMS worker’s already-sent table
- 06 Practise the retry question
Monday morning, Meera Kulkarni’s phone will not stop buzzing. “₹299 debited twice for one ticket.” That is lesson 24’s complaint: the SMS group got one event twice and sent two “Confirmed” messages, and payment retries charged 214 users ₹299 twice.
Each of the 214 tapped Pay once. The second charge came from SeatJaldi’s own app, sent to the outside service that moves the money.
In a crowded stadium, the payment gateway’s “paid” reply never reached the phone. So the app politely sent the payment again, and it was charged again. Meera refunds 214 × ₹299 = ₹63,986.
“Let us turn off retries,” says Arjun Nair. Zoya Mirza asks how a fan whose reply is lost would ever learn they paid. So Arjun tries again: a queue that delivers each message only once.
Without a retry, the user cannot tell if they paid. With one, they may pay twice. How can the server know the second request is the same payment as the first?
What does idempotent mean?
Idempotent, from lesson 4, means doing it twice leaves things as doing it once would. POST is not idempotent, and a payment is a POST.
Keep one line in your head: a repeat will come, so doing it twice must equal doing it once.
- You tap Pay, the screen freezes, you tap again, and your bank sends two debit messages. Nothing told the two requests apart.
- A page says “Do not press back or refresh”. The app is trying not to send the same request twice.
What is at least once delivery?
At-least-once delivery means a message always arrives, but it may arrive more than once.
A sender cannot tell a lost message from a lost “got it”, so it sends again. A message queue does the same when a worker is slow to say “done”.
Picture a loose wire between the lift button and the lift’s controller. One press may arrive twice. A good controller still sends one lift.
Can a network promise exactly once? Not by itself. A system can still make the effect happen once: one charge, one SMS.
| Promise | What it means | What the receiver must do |
|---|---|---|
| At most once | Zero or one time; may be lost | Accept that some are lost |
| At least once | One or more times; never lost | Be safe to run twice |
| Exactly once | Each message has its effect once | Usually at least once, plus spotting repeats |
How do you avoid duplicate payments with an idempotency key?
Send a unique key with each payment, and let the server remember the result for that key. The server, not the network, stops the double charge.
The app makes the key once, when the Pay screen opens for a booking, and sends it with every tap and retry. It is the lift’s “already called” light: a repeat finds it on and gets the lift already on its way.
The key check before the charge
Drag to exploreThe server first tries to insert the payment’s key. A new row means charge, then save the reply. No new row means send back the saved first reply, with no second charge.
Diagram connections:
- Pay app to Insert key: new row? (Pay with key)
- Insert key: new row? to Charge ₹299 (1 row: new key)
- Insert key: new row? to Read saved reply (0 rows: seen)
- Charge ₹299 to Save the reply (Paid)
- Save the reply to Reply to the app (Send reply)
- Read saved reply to Reply to the app (Send reply)
- Reply to the app to Pay app (Result)
- Pay app: Sends the payment and its key.
- Insert key: new row?: Payment API → Key store: insert, or nothing if the key exis…
- Charge ₹299: Payment API → Payment gateway, only when the key is new.
- Save the reply: Payment API → Key store: keep the gateway’s answer.
- Read saved reply: Payment API → Key store: the first answer, no second charge.
- Reply to the app: Payment API → Pay app: the same answer for every try.
Workers need the same trick, called deduplication. The SMS worker checks an “already sent” table first, like a controller ignoring a doubled signal.
How do you build the key check and the deduplication table?
Claim the key in its own quick write before any work, and store a short summary of the request beside it. Save the result at the end.
POST /payments HTTP/1.1
Host: api.seatjaldi.in
Idempotency-Key: "7f3a9c1e-5b2d-4e60-a1c4-0d8e3b7f2a91"
Content-Type: application/json
{ "booking_id": "BK-10457", "amount_rupees": 299 }What just happened? The key, made once when the Pay screen opened, travels in an HTTP header, unchanged on every tap and retry.
The server claims the key with one insert that the database refuses to repeat. In PostgreSQL:
CREATE TABLE idempotency_keys (
idem_key text PRIMARY KEY,
request_hash text NOT NULL,
response_body jsonb,
created_at timestamptz NOT NULL DEFAULT now()
);
INSERT INTO idempotency_keys (idem_key, request_hash)
VALUES ('7f3a9c1e-5b2d-4e60-a1c4-0d8e3b7f2a91', 'h-BK-10457-299')
ON CONFLICT (idem_key) DO NOTHING;
-- INSERT 0 1: new key, go and charge
-- INSERT 0 0: seen; if request_hash differs, refuse;
-- if response_body is empty, still running (409)What just happened? The table refuses a second row for one key, so two retries arriving together cannot both get 1. INSERT 0 1: charge, then save the reply in response_body. INSERT 0 0: check the hash, then return the saved reply.
- Lesson 4’s methods: GET only reads, and PUT and DELETE change things but are idempotent. POST is neither.
- The HTTP standard says a client should not retry a non-idempotent request on its own. The exception: it knows a repeat is harmless, or that the first try never landed. The key gives it that.
- Same key, different request: refuse it, never send the old reply. An expired IETF draft, not a standard, suggests 422 (cannot process) for this, and 409 for a repeat while the first still runs.
- Guard every hop: app to server, server to payment gateway, worker to SMS.
How long should the light stay on? Far longer than any retry. Say the app retries 3 times, 5 seconds apart: a 15-second window.
Keys are cheap. Assume 1 lakh payments a day, and about 1 KB a row with the saved reply and the index: 100 MB a day. Kept 30 days (an assumption), that is 3 GB.
To test: one payment twice, then twice at once: one charge. Same key, new amount: refused. One event twice: one SMS.
Try it yourself
Design the table
The SMS worker gets ticket.booked events at least once. Design a table that stops a second SMS for the same booking and kind, “confirmed” or “cancelled”.
Write the insert
Write the INSERT it runs before sending, and say what it does on 1 row and on 0.
Check your answer
PRIMARY KEY (booking_id, kind), then INSERT with ON CONFLICT DO NOTHING. One row: send. Zero: skip. You are right if the unique key is booking id plus kind, so a later “cancelled” SMS still goes. Insert first means a crash before the send loses that SMS, which beats sending two.
Back to the stadium. The payment gateway charges ₹299, the reply is lost again, and the app retries with the same key.
The server finds the key, like the lift’s light already on, and returns the first result. The fan pays ₹299 once, and a double tap like Kabir’s in lesson 23 charges once too: both taps carry the Pay screen’s one key. The SMS worker, handed the event twice, finds BK-10457 in its table and sends one SMS. The server knew the second request was the same payment because it carried the same key.
Meera tells users the ₹63,986 is on its way back. Retries are safe now. But during the national final, a slow payment gateway made every layer retry at once, and the whole site fell over.
Check yourself
Check yourself
The app retried a payment after a lost reply, and the user was charged twice. What was missing?
Pick an answer to see why it is right or wrong.
Check yourself
A queue promises at-least-once delivery. What must the SMS worker do?
Pick an answer to see why it is right or wrong.
Check yourself
Arjun’s app makes a fresh idempotency key on every retry. What happens after a lost reply?
Pick an answer to see why it is right or wrong.
Remember these three
- Messages arrive at least once, so doing it twice must equal doing it once.
- Make one idempotency key per payment, reuse it on every tap and retry, and let the server return the first result.
- Workers deduplicate by inserting the id first and skipping if the row already exists.
Questions people ask
- What does idempotent mean in simple words?
- An operation is idempotent if doing it twice has the same effect as doing it once. Pressing a lift button five times still brings one lift, but pressing Pay five times must not take five payments.
- What is at least once delivery?
- It means a message is delivered one time or more, and never zero. If the receiver does not confirm in time, the message is sent again, so the receiver must be safe to run twice.
- How do you avoid duplicate payments?
- Send a unique idempotency key with each payment, made once, when the Pay screen opens. The server stores the first result under that key, so a retry gets that result back instead of a second charge.