Railway's October 2026 Security Incident Explained: Rotating Leaked Secrets

Every API key, database password and token you pasted into that platform may now sit with an attacker. Encryption on the platform's side does not change that if the attacker reached credentials that can decrypt it. That is the core lesson of the Railway security incident, disclosed on 21 October 2026.

Picture a common setup. A small team runs a web app, a PostgreSQL database and a payments integration on a deploy platform. The setup is made up, but every step it follows below comes from Railway's own report and guides.

The team keeps six environment variables on the platform. They are a database URL, a payments key, a cloud storage key, an AI model provider key, a registry password and a GitHub token. An environment variable is a named value, like DATABASE_URL, that the platform hands to your app when it starts. A secret is any value that grants access, such as an API key or a password. Teams store secrets this way so they never sit in the code.

On the morning of 22 October, India time, this team reads that some of those values may have been decrypted by someone else. The rest of this guide answers three questions in order. What happened? Why did encryption not help? What exactly should the team do now?

An attacker got into Railway's build infrastructure, the servers that turn your code into a running app. There they widened their access, which is called privilege escalation. They pulled encrypted customer secrets, then obtained credentials that let them decrypt a subset.

Railway's report puts the attack timeline at 20 October, 13:58 UTC to 22 October, 03:32 UTC. In IST, India Standard Time (UTC plus 5 hours 30 minutes), that is 19:28 on 20 October to 09:02 on 22 October. At 03:32 UTC Railway confirmed the attacker no longer had access.

Running customer apps stayed online, but new builds and deploys were paused. The dashboard, API and CLI were also unavailable at times. Railway's disclosure says it believes the incident is part of a series of attacks on platform companies. Their aim, it says, was AI inference providers, the companies that run AI models for other apps.

The timeline in UTC, from the preliminary report and status page

Two terms appear in the table. CI means continuous integration: the automated system that builds and tests code on every change. An artifact is a file that a build produces, such as a package or a bundle of settings.

Notice two moments as you read. First the attacker held encrypted secrets they could not read. A few hours later, after the attacker obtained CI credentials, a subset was decrypted using a combination of stolen credentials.

The report does not say which CI system these credentials came from, or which stolen credential did what.

A common way to store them is envelope encryption: each secret gets its own key, and that key is locked by a master key. Railway's report says the keys it holds were envelope encrypted. That is why, it says, the unauthorised query alone cannot show which keys were affected.

Think of a bank's safe deposit room. Each box has its own small key, sealed in a pouch tied to that box. Only the manager's master key opens a pouch. A thief who carries away boxes and pouches still cannot open them. A thief who tricks the manager into opening pouches can open every box.

In real systems, the box key is the data encryption key, or DEK. Each DEK encrypts one value.

The manager's master key is the key encryption key, or KEK. Encrypting the DEK with the KEK is called wrapping it. Google Cloud's key management docs describe this pattern.

The encrypted value and the wrapped DEK sit side by side in the database. The KEK stays inside a separate key service. To read a secret, a program asks the key service to unwrap the DEK, then uses the DEK to decrypt the value.

Where the picture breaks: a real KEK never leaves the key service. Attackers borrow its power through credentials that are allowed to call it, rather than stealing the key itself.

Google's docs say access to keys in the hierarchy is governed by identity and access management. In plain words, the key service decides who may unwrap. Cloud KMS can also record each Decrypt call in its Data Access audit logs, if those logs are enabled. So the question that matters is not whether data is encrypted. It is which machines and credentials are allowed to ask for decryption.

A common belief is that encrypted means safe even if stolen. Here it did not, because the attacker went on to obtain credentials that could decrypt. Encryption at rest, meaning data scrambled while stored, only protects against someone who holds the data without the keys. The timeline above shows both stages: unreadable secrets first, then decryption after credentials were stolen.

The demo uses AES-GCM, a standard encryption mode that also detects tampering. It was run on Python 3.13.17 with the cryptography package pinned at 50.0.1. Beginners can skip the code and read the output and the box after it. The hex on the second output line is random and changes on every run.

"""Envelope encryption demo for the SevyDevy Railway incident guide.
Tested with Python 3.13.17 and cryptography 50.0.1 (pip install cryptography==50.0.1).
The hex on the second output line is random and differs on every run."""
import os
import sys
import cryptography
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.exceptions import InvalidTag

print("python", sys.version.split()[0], "| cryptography", cryptography.__version__)

# The KEK lives in a key service, never in the database.
kek = AESGCM.generate_key(bit_length=256)

def encrypt_variable(plaintext, kek):
    dek = AESGCM.generate_key(bit_length=256)        # one data key per variable
    n1, n2 = os.urandom(12), os.urandom(12)
    ciphertext = AESGCM(dek).encrypt(n1, plaintext, None)
    wrapped_dek = AESGCM(kek).encrypt(n2, dek, None)  # wrap the DEK with the KEK
    return {"ct": ciphertext, "n1": n1, "wrapped_dek": wrapped_dek, "n2": n2}

def decrypt_variable(row, kek):
    dek = AESGCM(kek).decrypt(row["n2"], row["wrapped_dek"], None)  # unwrap
    return AESGCM(dek).decrypt(row["n1"], row["ct"], None)

row = encrypt_variable(b"PAYMENTS_API_KEY=pay_live_demo_123", kek)
print("Stored row:", row["ct"].hex()[:32], "...")
stolen = dict(row)                 # the attacker copies the row now
read_before = decrypt_variable(row, kek)

# 1. The row alone, with a guessed key.
try:
    decrypt_variable(stolen, AESGCM.generate_key(bit_length=256))
except InvalidTag:
    print("1. Stolen row with a guessed key: InvalidTag")

# 2. The row plus use of the real KEK.
# holding kek here stands for permission to call unwrap in the key service
print("2. Stolen row with the real KEK:", decrypt_variable(stolen, kek).decode())

# 3. The platform re-wraps the same DEK under a new KEK.
new_kek = AESGCM.generate_key(bit_length=256)
dek = AESGCM(kek).decrypt(row["n2"], row["wrapped_dek"], None)
row["n2"] = os.urandom(12)
row["wrapped_dek"] = AESGCM(new_kek).encrypt(row["n2"], dek, None)
try:
    decrypt_variable(row, kek)
except InvalidTag:
    print("3a. Database copy with the old KEK: InvalidTag")
print("3b. Stolen copy with the old KEK:", decrypt_variable(stolen, kek).decode())
read_after = decrypt_variable(row, new_kek)
print("3c. Secret value unchanged by re-wrap:", read_before == read_after)

python 3.13.17 | cryptography 50.0.1
Stored row: fab5bbfec3ce3fb877b5ac7544534e59 ...
1. Stolen row with a guessed key: InvalidTag
2. Stolen row with the real KEK: PAYMENTS_API_KEY=pay_live_demo_123
3a. Database copy with the old KEK: InvalidTag
3b. Stolen copy with the old KEK: PAYMENTS_API_KEY=pay_live_demo_123
3c. Secret value unchanged by re-wrap: True

So the answer to the prediction is no. Re-wrapping with a new KEK changes the lock on the copy in the database. A stolen copy stays readable until the credentials that could use the old KEK are revoked or that key is disabled. Neither step helps once a value has been read in plain text. To kill that value, revoke it at the service that issued it, such as your payments, cloud or database provider.

Railway rebuilt its internal tooling, rolled every key twice and moved builds onto machines that serve one customer each. Its preliminary report lists these actions.

• Shut down potentially compromised internal tooling and rebuilt infrastructure tooling from validated source.
• Rolled all keys, then rolled them again after confirming the intruder was out.
• Moved builders off multi-tenant machines, where many customers' builds share hardware, to single-tenant environments that serve one customer. This sped up a rollout already planned.
• Rolled credentials for all storage buckets on Railway, rolled all GitHub App installations and revoked all Railway access tokens.
• Added a way to mass revoke Railway application permissions whenever activity looks suspicious.
• Engaged an external forensics firm and contacted authorities.

The status page update posted at 17:00 UTC on 22 October adds that API and project tokens created before the incident stay invalid. If you use the Public API, or the CLI with a token, create a new token.

If you used Railway before 21 October 2026, treat every secret stored there as exposed and rotate it, whether or not you got an email. Railway recommends this for all customers who signed up before 21 October. Its earlier disclosure told every user to roll all variables, so also rotate anything you added during the attack timeline. For the team in our example, if its platform were Railway, that means all six variables.

Railway's preliminary report, last modified on 23 October 2026, says at most 1.85% of all workspaces had some variables potentially compromised. That is about 100,000 workspaces, or 1 lakh. The report says Railway is contacting directly affected customers and those it sees as high risk: workspaces with very many variables, or with many AI keys from vendors. Status updates on 23 October said that outreach was still underway.

Some users got key-reset emails from upstream vendors. Railway says such a notice does not imply your credentials were impacted. The report's FAQ says Railway first sent providers a list of potentially decrypted keys, asking them to revoke those keys. Some vendors then asked for all keys to be revoked. Elsewhere it says some vendors received all API keys on the platform, regardless of whether those were known to be impacted. Either way, a revoked key fails until you add a new one.

The report states user accounts were not accessed. It also says Railway is not ruling out access to user source code, and will confirm in the final report. As of 24 October 2026 that question is open.

Build servers run code that customers upload, and CI systems hold credentials that can reach many other systems. That combination makes them a natural path to secrets. OWASP's Secrets Management Cheat Sheet notes that CI/CD tooling regularly consumes high-privilege credentials.

On a multi-tenant platform, the build step runs code the platform did not write, on machines it shares. In general, anyone who escalates there next looks for credentials those machines can use. In Railway's case, the report links the CI credentials to decrypted artifacts holding further API access.

Multi-tenant vs single-tenant builders

Railway says it moved builders from its third generation isolated multi-tenant setup to its fourth generation single-tenant environments. Multi-tenant means many customers' builds share the same machines, separated by software. Single-tenant means a build machine serves one customer.

Single-tenant builders shrink the blast radius of a build escape, meaning how much one break-in can reach. They do not remove the need to guard CI credentials and key access, which this incident also used.

Save the evidence, revoke the most dangerous credentials first, prove the old ones fail, then hunt for misuse. OWASP puts it plainly: exposed keys should be revoked immediately, and a new secret must be quick to create and deploy.

When a leak is suspected, accept a short outage to kill the old secret fast. Railway's PostgreSQL guide says to regenerate first in that case. It offers a zero-downtime method only when compromise is not suspected.

1. 1

   1. List every secret and every consumer

   Write down each variable, which services read it and anything outside the platform that uses it, such as laptops or CI jobs. Missed consumers break later.

2. 2

   2. Close public paths first

   Remove public network access to databases that nothing outside the project needs. Railway warns that with public access on, a leaked password stays usable from anywhere through its TCP proxy until rotated.

3. 3

   3. Export audit logs before revoking

   Save logs as JSON or CSV first, as Railway's GitHub guide advises, so the evidence survives when credentials are deleted.

4. 4

   4. Rotate the highest-value keys first

   A sensible order is to start with cloud keys that can create resources, payment keys and AI model provider keys. Then do database passwords, registry passwords and Git tokens.

5. 5

   5. Create, switch, redeploy

   Create the new credential, update the variable and redeploy every service that reads it. On Railway, containers read variables only at startup, so an update does nothing until you redeploy.

6. 6

   6. Kill the old credential

   Deactivate or delete the old key at the provider. On AWS, deactivate first so you can briefly re-enable it if a forgotten service breaks, then delete.

7. 7

   7. Prove the old key is dead

   Call the provider once with the old key and expect an authentication error. Railway's AWS guide, for example, says requests signed with a deactivated key fail. Where a last-used date exists, as in AWS IAM, check it stops moving.

8. 8

   8. Hunt for misuse in the exposure window

   Search provider logs from 20 October onwards for actions, regions or IP addresses you do not recognise. Check bills too; Railway's AWS guide suggests comparing costs in the exposure window with the month before.

Where to check for misuse, by secret type

Two details trip people up. CloudTrail event history is per region, and Railway's AWS guide notes attackers often use regions you do not watch. As of 24 October 2026, Railway's GitHub guide says GitHub keeps the personal security log for 90 days and the organisation audit log for 180 days.

Limit who can decrypt, shorten how long a stolen credential works and make every decrypt visible. These are design choices, and the sources below support each one.

• Narrow key access. Google's KMS docs place the KEK behind identity and access management. Grant unwrap rights only to the services that must decrypt.
• Prefer short-lived credentials. OWASP advises dynamic secrets, created on demand and expiring soon, to reduce credential reuse.
• Keep tamper-resistant audit logs. OWASP says auditing must resist attempts to tamper with or delete logs. It separately says responders need information about the lifecycle of a secret.
• Isolate builds per customer. Railway's move to single-tenant builders is an example of cutting what one escape can reach.
• Build a mass revoke button before you need it. Railway added one after the incident; having one ready would likely shorten a response.

Explain it as a trust-boundary story: what was stored, which boundary failed, how to shrink the blast radius and how customers recover. Interviewers want that reasoning, not the news itself. Keep each step below to two or three sentences.

1. 1

   State the asset

   Customer secrets stored as environment variables, envelope encrypted, and typically decrypted when an app is built or deployed.

2. 2

   Name the real boundary

   Encryption protects stored data. The boundary that failed was access to credentials able to decrypt, reached through build infrastructure.

3. 3

   Shrink the blast radius

   Single-tenant builders, narrow unwrap permissions and separate credentials for CI so one escape reaches less.

4. 4

   Detect and respond

   Log every decrypt, alert on bulk reads of the secrets table and keep a tested mass revoke path.

5. 5

   Give the customer runbook

   Rotate at the issuing provider, highest value first, prove the old key fails and review audit logs for the exposure window.

6. 6

   Admit what is unknown

   Say the entry point is undisclosed as of 24 October 2026. Interviewers trust candidates who separate facts from guesses.