Module 7 · Keeping It Running and Safe · Lesson 27 of 35
What is rate limiting? Saying "slow down" before the system falls
12 minFree System design course · Beginner to intermediateLesson 7.2By Saket Bhatnagar
Researched and drafted with AI, reviewed and edited by Saket Bhatnagar.
In short
- What is rate limiting? It caps how many requests one user may send in a set time, and turns extras away early with HTTP 429, “too many requests”.
- Think of darshan slips at a temple: a family may use a few at once, but nobody gets a hundred. Bursts pass; floods do not.
- When the whole system is full, the least important work is dropped first, so bookings keep working.
Video lesson · coming soon
What is rate limiting? Saying slow down politely
Lala, your AI mentor, will walk you through this lesson in a short animated video. Until then, everything is here to read and try.
- 01 Fifty times normal, mostly bots
- 02 Darshan slips at a temple
- 03 The bucket in front of bookings
- 04 A burst, a 429 and a bot
- 05 Five tokens for seat holds
- 06 Practise it, then ask how reliable
With circuit breakers in place, SeatJaldi survives its next sale. But at noon on Sunday, Sandeep Patil’s dashboard shows the booking API at 50 times normal traffic. Most of it is not fans.
It is bots: programs that tap Book faster than any person, grab seats and resell them. The circuit breakers keep the site up, but only by refusing everyone, fans included.
Arjun Nair has a fix. “Let us block the bad IP addresses.” By evening, a college hostel in Nagpur is blocked: hundreds of real students behind one address. The bots have moved on.
“How do I stop the bots without turning away my own fans?” asks Meera Kulkarni. “And what do I tell users?” Zoya Mirza writes the puzzle on the board. A fan taps Book 3 times in a second; a bot taps 500 times. Both are too fast. How can the system tell them apart without reading minds?
What is rate limiting?
Rate limiting caps how many requests one user or client can send in a set time, such as 10 a minute. It turns the extras away early, before they reach slow, costly parts like the database.
The part that does the counting is the rate limiter. A common way to count is with tokens kept in a bucket: each request must spend one token to get through.
Keep one line in your head: a bucket of tokens, where bursts are fine and floods are turned away early.
- Each user has their own bucket, holding up to a set number of tokens.
- Tokens drip in at a steady pace, and every request spends one.
- A short burst spends saved-up tokens. A flood empties the bucket.
A token bucket in front of the booking service
Drag to exploreEach request asks the rate limiter for a token, and a refill clock adds tokens at a steady pace. With a token, the request goes on to the booking service. With none, a 429 reply goes straight back to the client.
Diagram connections:
- Client to Rate limiter (Request)
- Rate limiter to Token bucket (Take token)
- Refill clock to Token bucket (Add tokens)
- Rate limiter to 429 reply (Bucket empty)
- Rate limiter to Booking service (Token found)
- 429 reply to Client (Sent back)
- Client: A fan’s app or a bot, each with its own bucket.
- Rate limiter: Takes one token per request, or answers 429.
- Token bucket: Holds up to 100 tokens for one client.
- Refill clock: Adds 10 tokens a second, up to 100.
- 429 reply: Too Many Requests, often with a Retry-After header.
- Booking service: Holds seats and takes bookings.
- A login screen says “Too many attempts. Try again in 30 seconds.” The server counted your tries, often answering 429 with a Retry-After of 30.
- A messaging app sends your first ten photos at once, then one at a time: a burst, then the refill’s pace.
- In a sale, an offer banner says “We are very busy”, yet checkout works. Low-priority work was dropped first.
How does a token bucket allow bursts but stop floods?
A token bucket saves tokens up to a limit and adds new ones at a steady pace. Like Selvi’s devotees, a quiet user saves up for a burst. A bot that grabs each slip as it appears only gets the new ones.
What just happened? The bucket’s size, 100, set the biggest burst. The refill, 10 a second, set the long-run speed.
What does HTTP 429 mean, and what is load shedding?
HTTP 429, from lesson 4, means “Too Many Requests”: this one client is too fast, and Retry-After may say when to return. Load shedding is different: it drops low-priority work when the whole system is overloaded, whoever sent it.
At the temple, the board is the 429: it tells one devotee to come later. Sending the elderly and the sick in first is load shedding: it is about the whole hall. In a big sale, SeatJaldi drops analytics first (taps the app reports for the team’s charts), then seat-map views; bookings and payments go last. A shedder often answers 503, “cannot take this right now”.
| Answer | What it says | Caused by | What the client does |
|---|---|---|---|
| 429 Too Many Requests | You are too fast | This one client | Waits for Retry-After, if sent |
| 503 Service Unavailable | Cannot take this now | Total load, or a fault | Waits, then retries with jitter |
| Load shedding | Low-priority work dropped | Total load | Gets a 503 for low-value calls; bookings still work |
Where do engineers put a rate limiter, and what trips them up?
Put the limiter at the edge, where saying no is cheapest, and again before costly services. Key it on the user, with one shared count.
- Where: at the API gateway or edge servers, and in the booking service for seat holds.
- Key: the user id, or an API key (a code each partner app sends to say which app it is). An IP is weak: one hostel, many people.
- Counts: in one shared Redis store that every app server uses, so all see the same count.
Why not count per minute instead? Say the limit is 100 a minute, reset each minute. A client sends 100 at 11:59:59 and 100 at 12:00:00: 200 in 2 seconds. A token bucket has no window edge. Its worst minute is its size plus one minute of refill.
Stripe’s engineering team describes four kinds: one limits requests a second, one limits requests in progress at once, and two shed load while keeping room for critical calls.
Limits also protect you from your own apps, such as a buggy app caught in a retry storm. If the shared counter breaks, Stripe advises failing open: let requests through rather than block everyone.
import time
class TokenBucket:
def __init__(self, capacity, refill_per_second):
self.capacity = capacity
self.tokens = capacity
self.rate = refill_per_second
self.last = time.monotonic()
def allow(self):
now = time.monotonic()
self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.rate)
self.last = now
if self.tokens >= 1:
self.tokens -= 1
return True
return FalseWhat just happened? allow() adds tokens for the time that passed, caps them at the bucket’s size, and spends one if it can. Seat holds use TokenBucket(5, 1 / 6). It lives in one server’s memory: that is the trap.
Try it yourself
Pick the rule
Each user should make about 10 seat holds a minute, and a fan may tap a few times at once.
Choose the bucket
Pick a bucket size and a refill. Hint: 10 a minute is one token every 6 seconds.
Test it
A bot sends 500 requests a second for a minute. How many get in?
Check your answer
You pass if your refill is one token every 6 seconds and your bucket holds 3 to 10, so a fan’s 3 quick taps pass. The bot gets at most your size + 10 in the first minute, then 10 a minute. With 5 tokens, that is 15 of 500 x 60 = 30,000.
Back to Sunday’s sale. Sandeep and Zoya give every logged-in user a seat-hold bucket: 5 tokens, one more every 6 seconds. The Nagpur hostel is unblocked: the limit follows the user, not the address.
So how does the system tell them apart? The fan’s 3 taps fit in 5 tokens. The bot gets 5 at once, then 10 a minute, and 429 for the rest. Bot holds fall by about 80%. The rest come from bots using many accounts, like one person sending a hundred cousins for slips.
If the system fills anyway, analytics is shed first. Meera tells users: tap once, and if the app says wait, it says how long. Then investors ask: “How reliable are you? 99.9%? 99.99%?” Meera does not know what to promise.
Check yourself
Check yourself
A fan taps Book 3 times in a second; a bot taps 500 times. Which design tells them apart without reading minds?
Pick an answer to see why it is right or wrong.
Check yourself
4 app servers each keep their own bucket in memory. The limit is 10 holds a minute per user. What can one user really do?
Pick an answer to see why it is right or wrong.
Remember these three
- A bucket of tokens: bursts are fine, floods are turned away early.
- The bucket’s size sets the burst; the refill sets the long-run speed.
- Key limits on the user, share one count, and shed low-priority work first when the system is full.
Questions people ask
- What is rate limiting?
- Rate limiting caps how many requests a user or client may send in a set time, such as 10 a minute. Extras are turned away early, so the slow, costly parts stay free for everyone.
- What does HTTP 429 mean?
- It means “Too Many Requests”: the client is sending faster than the limit allows. The server may add a Retry-After header saying how many seconds to wait.
- What is load shedding?
- Load shedding is dropping the least important requests on purpose when the whole system is overloaded. Analytics goes first, then browsing; bookings and payments go last.