Google Cloud Storage's October 2026 Outage Explained: Bypassed Capacity Checks

Your uploads start failing with 503 errors, the ones that succeed get slow, and naive retry code can make things worse. That is what the Google Cloud Storage outage of October 2026 looked like to apps writing data in one region.

Picture a common setup. A billing app in Pune runs in us-central1 and saves each day's payments report, a small CSV file, to a Cloud Storage bucket in the same region. The setup is made up, but the symptoms below come straight from the official report.

Late in the afternoon of 8 October, India time, those saves begin to fail. Some return HTTP 503, and others hang until the app gives up. Reading old reports mostly still works, because reads saw only a minor impact.

Now three bad things can happen inside the app. A retry loop with no pause hammers a service that is already overloaded. A retry after a lost reply may overwrite a newer file. And a job that never gives up holds a worker for hours.

This guide uses that app as its running example. First it explains what happened inside Google Cloud Storage, using only what the report says. Then it shows what the app should do, with code we ran and checked.

You need five terms to follow the story. Two more, backoff and preconditions, are explained later, right where the code uses them.

• Bucket and object: a bucket is a named container in Cloud Storage, and an object is one file inside it, such as a daily report file.
• Region and zone: a region is a group of data centres in one area, like us-central1 in Iowa. A zone, like us-central1-b, is one isolated part of a region.
• Load balancer: the layer that receives requests and decides which server handles each one.
• Capacity-aware routing: routing that checks how much spare room each server has before sending it more work.
• HTTP 503 Service Unavailable: a status code that says the server cannot handle the request right now, so a later retry may work.

Writes to Cloud Storage in us-central1 saw elevated 503 errors, timeouts and latency for 12 hours and 30 minutes on 8 October 2026. The problem returned on 9 October in zone us-central1-b for 5 hours and 41 minutes.

The report says customers "primarily observed HTTP 503 UNAVAILABLE errors on WriteObject operations and client-side timeouts". WriteObject is the storage write call, and a client-side timeout means the app stopped waiting. Writes that did succeed had "significantly higher latencies than normal", and the report gives no figure for it.

The report says other regions and other Google Cloud services were not affected. It gives no error-rate percentages and does not mention lost data, so this guide does not guess at either.

The timeline in the report's own times

The incident page gives all times in US/Pacific, which was PDT (UTC minus 7 hours) in October. The table adds India Standard Time (UTC plus 5:30), which is 12 hours 30 minutes ahead of PDT.

Notice that two start times appear: 04:00 and 05:19. The report does not explain the gap. Use 04:00 as the official start; a section near the end looks at it more closely.

A configuration change to the storage load-balancing layer skipped the normal capacity check. It piled writes onto a few servers that were already short of capacity. Google calls it "a traffic sorting optimization configuration update".

According to the report, storage backend nodes in us-central1 had "a localized capacity shortfall due to increased traffic". A node is one storage server. The traffic-sorting update was meant to ease that local shortage.

Instead, the update ended up "inadvertently bypassing the standard capacity-aware routing logic, resulting in a severe load imbalance". The report says a few nodes "received an overwhelming proportion of incoming write traffic despite having low available serving capacity". Those nodes were overloaded, and their ingestion services began timing out.

The report also names a second, separate problem. An issue in the underlying database service caused extra metadata processing errors. Metadata is the information about each object, such as its name and size. The report gives no more detail, so neither does this guide.

Capacity-aware routing sends each new request to a server that still has room, and moves traffic elsewhere when a server reaches its limit. The load balancer needs a number for each server's capacity, plus a live measure of its load.

The report does not describe how Google Cloud Storage measures capacity inside its own load-balancing layer. So the clearest public example is a different product: Google's Cloud Load Balancing, which customers configure themselves.

Its documentation says "the balancing mode defines how the load balancer measures capacity". In plain words, you choose what counts as full: open connections, requests per second or CPU use. When the servers in one zone reach their target, new requests go to another zone, if you have servers in two or more zones. The target is a soft limit, not a hard wall: the page says it "isn't a circuit breaker", and when every zone is full, traffic is "distributed by overfilling proportionally".

The same page notes that a capacity scaler of 0 means "the backend is completely drained". Draining is the manual version of the same idea: stop sending new work to a server. Google's engineers drained the most overloaded storage nodes during the outage.

An everyday picture: ticket counters at a busy station

Think of a large railway station with ten ticket counters. A supervisor at the entrance looks at every queue and sends each person to a short one. That supervisor is capacity-aware routing.

Now someone adds a shortcut to speed things up: people with a certain kind of ticket go straight to counters 3 and 4. Nobody checks those queues any more. Counters 3 and 4 jam, while the supervisor's careful checks never run for those people.

Map it back. The counters are storage nodes, the queues are their load, the supervisor is the capacity check, and the shortcut is the traffic-sorting update. Where the picture breaks: a real balancer follows coded rules, not a person's judgement, and the report never says how the sorting chose its targets.

Because a new routing path can send traffic directly to its targets without passing through the code that checks capacity. The report says the update was "bypassing" capacity-aware routing; it does not publish the code or the config.

So the points below are general lessons that fit the report's wording. They are not claims about how Google's system is built.

• Safety checks often live in one code path. A new path that picks servers by another rule, such as sorting, can skip that check unless the check is enforced underneath every path.
• Optimisations are often tested when there is spare room everywhere. A shortcut that ignores capacity looks fine in testing and only fails when some nodes are already short.
• A change made to fix a capacity problem runs exactly when capacity is tight. That is the worst moment for a mistake in capacity handling.

Google's preventive actions match these lessons. One is ensuring "that traffic distribution logic cannot bypass pre-spilling safeguards". The report does not define pre-spilling safeguards, but the point is clear: the safety check must sit where no routing path can avoid it.

Another action is to validate load-balancing changes "in alternative, smaller regions before being rolled out to larger, highly utilized regions". A third is to speed up internal data movement within highly used storage cells, to absorb load spikes.

Because a rollback was incomplete. The report says the recurrence "was caused by an incomplete rollback of an upload planner feature", which broke load balancing again and overloaded a few smaller storage nodes in us-central1-b.

The report does not explain what the upload planner does, so this guide does not guess. The lesson is general and painful: a rollback is a change too, and a partial one can leave the system in a state nobody tested.

The final fix was complete. The report says engineering "fully reverted the problematic load balancing configuration change". After traffic and use settled, the drained nodes were put back into service.

Why Google could not just add more servers

The first idea was extra capacity. Engineers tried to allocate emergency capacity, "but this was blocked by systemic regional capacity limits". The region had no spare room to give.

So they took three other steps. They drained traffic from the most overloaded nodes, deployed a config change to turn off the traffic-sorting optimisation, and rolled back a recent update to the metadata database.

Retry it, but with exponential backoff, jitter and a total deadline. The Python client uses full jitter. That means after each failure, double the longest allowed wait, then pick a random wait between zero and that limit. Google's retry strategy page lists HTTP 408, 429 and 5xx responses as transient problems worth retrying, and it recommends backoff with jitter.

A retry only helps if the request is safe to repeat and the failure is temporary. The table shows how the common answers split, using the Python client's own list of retried codes.

The fifth row is the tricky one. During this outage many clients saw timeouts, and a timeout tells you nothing about whether the write landed. That is why the next section adds preconditions.

Why jitter matters when thousands of clients fail together

Without jitter, every client that failed at the same moment retries at the same moment. Those retries arrive as a wave, and the wave lands on servers that are already short of capacity.

This pure-Python simulation needs no cloud account. It sends 1,000 clients through 4 retries and counts the busiest 100-millisecond slot.

import random
from collections import Counter

random.seed(42)
CLIENTS = 1000   # clients whose write failed at the same moment
RETRIES = 4      # each client retries 4 times
CAP = 32.0       # longest allowed wait, in seconds

def retry_times(jitter):
    t, times = 0.0, []
    for attempt in range(RETRIES):
        limit = min(CAP, 1.0 * 2 ** attempt)        # 1, 2, 4, 8 seconds
        wait = random.uniform(0, limit) if jitter else limit
        t += wait
        times.append(t)
    return times

for jitter in (False, True):
    buckets = Counter()
    for _ in range(CLIENTS):
        for t in retry_times(jitter):
            buckets[int(t * 10)] += 1               # 100 ms slots
    peak = max(buckets.values())
    print(f"jitter={jitter}: busiest 100 ms slot gets {peak} retries")

Run it with python3 and you should see exactly these two lines, because the random seed is fixed:

jitter=False: busiest 100 ms slot gets 1000 retries
jitter=True: busiest 100 ms slot gets 172 retries

Without jitter, all 1,000 clients hit the server in the same slot, four times over. With full jitter, where each wait is random between zero and the limit, the busiest slot drops to 172. The total work is the same; it is just spread out.

What the Python client already does for you

The official Python client retries for you. We read and ran google-cloud-storage 3.17.1 with google-api-core 2.43.0 on Python 3.13, as of 24 October 2026.

• DEFAULT_RETRY retries 408, 429, 500, 502, 503 and 504, plus connection errors and timeouts.
• The first wait limit is 1 second, it doubles each time, it never exceeds 60 seconds, and retrying stops after 120 seconds in total.
• Each wait is drawn at random between 0 and the current limit. That is full jitter, and our test of 20,000 runs stayed inside those bounds.
• Uploads such as upload_from_string use DEFAULT_RETRY by default, even with no precondition. Google's retry page also says Objects insert is retried by default in Python.

Send a condition the server checks before writing. For a new object, if_generation_match=0 means write only if no live object with this name exists yet.

A generation is the version number Cloud Storage gives each object's data. Google's request preconditions page says that with this precondition, "retries of the upload don't accidentally write the object twice". If the object already exists, the server answers 412 Precondition Failed.

Here is the billing app's upload with a custom retry policy and the precondition. It uses only documented parts of google-cloud-storage 3.17.1.

from google.cloud import storage
from google.cloud.storage.retry import DEFAULT_RETRY
from google.api_core.exceptions import PreconditionFailed

# Exponential backoff with full jitter: each wait is random, up to 1 s,
# then up to 2 s, 4 s ... never more than 32 s. Stop retrying after 300 s.
write_retry = DEFAULT_RETRY.with_delay(
    initial=1.0, multiplier=2.0, maximum=32.0
).with_timeout(300.0)

client = storage.Client()
blob = client.bucket("demo-bucket").blob("reports/2026-10-08.csv")

try:
    # if_generation_match=0 means "create only if no live object exists yet",
    # so a retried write can never overwrite a newer copy.
    # timeout=30: give up on an attempt if no reply bytes arrive for 30 s.
    blob.upload_from_string(
        "a,b\n", if_generation_match=0, retry=write_retry, timeout=30
    )
    print("Uploaded, generation", blob.generation)
except PreconditionFailed:
    print("Object already exists: an earlier attempt may have succeeded")

The retry line copies DEFAULT_RETRY and changes two things: a 32-second cap on each wait, and a 300-second retry deadline. The deadline is checked between attempts, so a request already in flight can keep running past it. The upload call's own timeout= argument is separate from the retry deadline. It defaults to 60 seconds in 3.17.1, and the sample lowers it to 30 so a silent attempt is abandoned sooner. It is not a total time limit: the client passes it to the Requests HTTP library, where it limits the wait for the next bytes.

How to check it works without a cloud account

We ran this exact code against a small fake storage server on our own machine. The client sends requests there when you set the STORAGE_EMULATOR_HOST environment variable to the server's address.

1. 1

   Make the first two writes fail

   The fake server answered the first two upload requests with 503. In our latest run the client waited about 0.90 seconds and then about 0.92 seconds, inside the 1-second and 2-second limits. Your waits will differ, because they are random.

2. 2

   See the third attempt succeed

   The third request returned 200 OK, and the script printed: Uploaded, generation 1001. Every attempt carried ifGenerationMatch=0.

3. 3

   Run the same upload again

   The object now existed, so the server returned 412. The script printed: Object already exists: an earlier attempt may have succeeded.

4. 4

   Turn retries off to compare

   With retry=None, the very first 503 raised a ServiceUnavailable exception and nothing was retried.

We tested both cases with a 5-second retry deadline. Against a server that never answered, a 4-second timeout gave up after about 8.5 seconds. Against a server that sent its reply slowly in small pieces, a 2-second timeout let one attempt run for about 12 seconds. That was the full time the server took to send it. If a hard time limit matters, enforce it outside the client, for example with a job-level timeout in your worker or queue.

A 412 on a create-only upload is often good news after an outage. It may mean an earlier attempt landed even though its reply was lost. But a 412 only says some live object has that name, perhaps from an earlier run or another writer. Read the object back and compare it with what you sent before you mark the job done.

Plan for failures that outlast your retry deadline. The first episode lasted 12 hours and 30 minutes, far longer than any sensible retry window.

• Queue failed writes. Save the job in your own queue or database and process it later, instead of keeping a worker busy for hours.
• Alert on your own 503 rate. The earliest status update on the incident page came hours after the official start, so do not wait for a status page.
• Know your escape route. The report says other regions were unaffected, so a design that can write to a bucket in another region had options. Weigh that against cost and data-residency rules.
• Treat your own rollbacks as changes. Use a checklist that confirms every part of a feature is reverted, which is the exact lesson of 9 October.

The table puts each of Google's preventive actions next to the lesson for a team of any size.

Why the start time is 04:00 in one place and 05:19 in another

The 8 October status updates say the issue began at 05:19 PDT. The page header and both reports say 04:00 PDT, and only the full report adds a detection time, 04:50 PDT. The two start times are 1 hour and 19 minutes apart.

The report does not explain the difference, so this guide does not either. Treat 04:00 PDT as the official start, because the final report uses it. If you compare against your own logs, check both times.

Notice one more gap in the timeline table. The earliest update shown on the incident page is timed 11:24 PDT, more than seven hours after the official start. Your own monitoring can see a problem well before any status page does.

Use it as a short case study about routing, capacity and client behaviour. Interviewers listen for whether you separate the provider's failure from what your own system controls.

1. 1

   State the facts in one line

   In October 2026, writes in us-central1 saw elevated 503 errors, timeouts and latency for 12 hours and 30 minutes, then again in one zone for 5 hours and 41 minutes.

2. 2

   Name the mechanism

   A traffic-sorting change skipped capacity-aware routing, so a few low-capacity storage nodes received too many writes.

3. 3

   Explain why the fix took time

   Emergency capacity was blocked by regional limits, so engineers drained nodes, disabled the optimisation and later fully reverted the change.

4. 4

   Turn to the client side

   Retry 408, 429 and 5xx with exponential backoff, full jitter and a total deadline. Never retry forever.

5. 5

   Make retries safe

   Use preconditions such as ifGenerationMatch=0 for creates, so a retried write cannot duplicate or overwrite data.

6. 6

   Close with resilience

   Queue writes that outlast the deadline, alert on your own error rate, and decide in advance whether you can fail over to another region.