Secrets Management Explained: Vaults, Short-Lived Credentials and Rotation

The password ends up in many places, nobody can say who has seen it, and changing it means touching every copy at once. Secrets management is the practice that fixes this.

Picture a common setup. A food delivery app in Bengaluru has an orders service, a payments service and a nightly report job. All three read one .env file with the database password and a payment gateway key. The setup is made up, but every problem below is real.

At first this feels tidy. Then the team grows to twelve developers. The file gets pasted into chat for a new joiner, copied to three servers, saved as a CI setting and committed once by mistake.

• Copies spread. Each copy is another place the secret can leak from, and you cannot list them all.
• Git remembers. Deleting a committed .env file in a new commit leaves the old contents in history. You will run this below.
• Nobody can answer who read it. A file on disk keeps no record of which person or process opened it.
• Changing it hurts. Rotating the database password means editing every copy and restarting every service at the same moment.
• Everyone gets everything. The report job only needs read access, yet it holds the payment key too.
• Environment variables travel. Every child process inherits them, and crash reports or debug pages can print them.

None of this means the .env file is evil. It means a file was asked to do a job that needs access control, logging and rotation.

You need about eight words. Each one is explained again where it first matters, so you can skim this table now and come back.

Secrets management is a system, not one tool. It decides where secrets live, who may read each one, how long each stays valid, and how a leak gets caught and fixed.

In practice the app no longer carries the secret around. At start-up or on demand, it proves its identity to a secrets manager. The manager checks a policy, returns only the secrets that app may read, and records the request.

The OWASP Secrets Management Cheat Sheet, a free guide from the OWASP security community, asks teams to standardise and centralise this. It also asks for least privilege, meaning each person or service gets only the access its job needs.

A bank locker room is a fair picture. The room is the secrets manager. The ID check at the door is the identity check. The register at the desk is the audit log. The bank's own master key, which you never hold, is the KMS key.

Where the picture breaks: a good vault can also mint a brand-new key for every visit and make it stop working an hour later. No bank does that with lockers. That trick, dynamic secrets, is covered in the engineer half.

A .env file is fine because it follows a good rule: keep settings out of code. It becomes painful because a file has no access control, no audit trail and no safe way to change a value everywhere.

The Twelve-Factor App, a well-known set of guidelines for web apps, defines config as everything that varies between deploys. It asks for strict separation of config from code and suggests environment variables. Its test is useful: could you open-source the codebase today without leaking a credential?

Environment variables passed that test better than hard-coded passwords did. The OWASP cheat sheet adds a warning, though. It says environment variables are generally accessible to all processes and may end up in logs or system dumps.

We checked that on Linux. A child process inherited the variable, and the file /proc/PID/environ showed it in plain text to the same user. That file holds the environment the program started with, so a running process also misses later changes.

Pause and predict: does deleting the file remove the secret?

Suppose someone commits the .env file, notices, deletes it and commits again. The file is gone from the folder. Is the password gone from the repository? Run this in an empty folder to find out. The values are fake.

git init -q demo && cd demo
printf 'DATABASE_URL=postgres://app:[email protected]:5432/shop\nAWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE\n' | tee .env
git add .env && git commit -qm "add config"
git rm -q .env && echo ".env" | tee .gitignore
git add .gitignore && git commit -qm "remove .env"
ls -A
git show HEAD~1:.env

Output on git 2.43.0, as of 25 October 2026. The tee command saves its input to the file and also prints it, so the first three lines are just the files being written:

DATABASE_URL=postgres://app:[email protected]:5432/shop
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
.env
.git
.gitignore
DATABASE_URL=postgres://app:[email protected]:5432/shop
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE

What just happened? The folder listing shows no .env file. Yet git show printed the full file from the previous commit. Anyone who clones the repository can run the same command, and git log -p shows every version.

So the fix for a committed secret is never only a cleanup commit. You treat the value as exposed and replace it. The scanning section shows how to find such leaks.

No. A Kubernetes Secret stores each value as base64 text, and base64 is an encoding anyone can reverse. The Kubernetes docs say Secrets are stored unencrypted in etcd by default.

Kubernetes is a system that runs containers across many machines. etcd is the database where it keeps cluster objects. A Secret is the object meant for small sensitive values, up to 1 MiB each.

Base64 turns bytes into safe text characters. It has no key, so decoding needs nothing but the text. Try it with the fake password from a Secret's data field:

printf 'FAKE-db-pass-123' | base64
# RkFLRS1kYi1wYXNzLTEyMw==

echo 'RkFLRS1kYi1wYXNzLTEyMw==' | base64 -d
# FAKE-db-pass-123

That ran with GNU coreutils base64 9.4. If a Secret manifest with real values sits in a git repository, the secret is effectively public to everyone who can read that repository.

Who can read a Secret inside the cluster?

More people than most teams expect. The docs say anyone with API access can read or change a Secret, and so can anyone with access to etcd. Anyone allowed to create a Pod in a namespace can also read every Secret in that namespace.

The docs list four safety steps. Turn on encryption at rest. Use RBAC, role-based access control, with least privilege. Limit which containers mount each Secret. Consider an external secret store.

What does encryption at rest change?

Encryption at rest makes the API server, the front door for every Kubernetes request, encrypt Secrets before writing them to etcd. You pass a configuration file to the API server with the flag --encryption-provider-config. The first provider in its list encrypts every new write.

The provider called identity means no encryption at all. The aescbc and aesgcm providers keep their key in that configuration file on the control plane host, the machine that runs the cluster's management software. The kms v2 provider, stable since Kubernetes v1.29, keeps the master key in an outside KMS.

Turning it on does not touch Secrets already stored. The docs give a command that rewrites every Secret so the new provider encrypts it: kubectl get secrets --all-namespaces -o json piped into kubectl replace -f -.

Envelope encryption uses two layers of keys. A fresh data key encrypts the secret. A master key, kept inside a KMS, encrypts that data key. Only the encrypted data key is stored next to the secret.

This answers an old puzzle: where do you keep the key that encrypts your secrets? The OWASP cheat sheet says keys should not sit next to the secrets they encrypt unless those keys are encrypted themselves. Envelope encryption is exactly that arrangement.

Here is the whole idea in under forty lines of Python. It uses AES-GCM, a common cipher mode that also detects tampering. It ran on cryptography 45.0.7, pinned with pip install cryptography==45.0.7.

"""Envelope encryption in miniature with cryptography 45.0.7 (AES-256-GCM).
A data key (DEK) encrypts the secret; a key-encryption key (KEK) encrypts the DEK.
In real systems the KEK stays inside a KMS or HSM and never reaches your app."""
import os
from cryptography.hazmat.primitives.ciphers.aead import AESGCM

kek = AESGCM.generate_key(bit_length=256)          # lives in the KMS in real life
secret = b"DATABASE_PASSWORD=FAKE-db-pass-123"

dek = AESGCM.generate_key(bit_length=256)          # fresh data key per secret
n1 = os.urandom(12)
ciphertext = AESGCM(dek).encrypt(n1, secret, None)
n2 = os.urandom(12)
wrapped_dek = AESGCM(kek).encrypt(n2, dek, None)   # "wrap" the DEK with the KEK
del dek                                            # only the wrapped DEK is stored

print("stored ciphertext bytes:", len(ciphertext))
print("stored wrapped DEK bytes:", len(wrapped_dek))

# Read path: unwrap the DEK with the KEK, then decrypt the secret.
dek2 = AESGCM(kek).decrypt(n2, wrapped_dek, None)
print("decrypted:", AESGCM(dek2).decrypt(n1, ciphertext, None).decode())

# Rotating the KEK only re-wraps the small DEK; the big ciphertext is untouched.
new_kek = AESGCM.generate_key(bit_length=256)
n3 = os.urandom(12)
rewrapped = AESGCM(new_kek).encrypt(n3, dek2, None)
dek3 = AESGCM(new_kek).decrypt(n3, rewrapped, None)
print("after KEK rotation, same ciphertext decrypts:", AESGCM(dek3).decrypt(n1, ciphertext, None).decode())

# Tampering is detected by GCM's authentication tag.
bad = bytearray(ciphertext); bad[0] ^= 1
try:
    AESGCM(dek3).decrypt(n1, bytes(bad), None)
except Exception as e:
    print("tampered ciphertext rejected:", type(e).__name__)

Output, as of 25 October 2026:

stored ciphertext bytes: 50
stored wrapped DEK bytes: 48
decrypted: DATABASE_PASSWORD=FAKE-db-pass-123
after KEK rotation, same ciphertext decrypts: DATABASE_PASSWORD=FAKE-db-pass-123
tampered ciphertext rejected: InvalidTag

What just happened? Rotating the master key only re-wrapped a 32-byte data key. The 50-byte ciphertext of the secret was never touched. With millions of stored values, re-wrapping small data keys is far less work than re-encrypting every secret.

Real services follow the same shape. AWS documents that Secrets Manager asks AWS KMS for a 256-bit AES data key, encrypts the secret outside KMS, and stores only the encrypted data key in the secret's metadata. It requests a new data key whenever the value changes.

The Kubernetes kms provider does the same for etcd: data keys encrypt the objects, and a key-encryption key in a remote KMS encrypts the data keys. One detail AWS calls out: the secret's name, description and tags are not encrypted, so never put a secret in them.

A dynamic secret is created on request and destroyed on a timer. Instead of one shared database password, each service gets its own login that expires after a set time, such as one hour.

OpenBao is a community fork of HashiCorp Vault, managed under the Linux Foundation's OpenSSF. Both have a database secrets engine that generates database credentials dynamically based on configured roles. A role holds the SQL that creates the user.

In the OpenBao docs example, a role has default_ttl 1h and max_ttl 24h. Reading database/creds/my-role returns a generated username, a password, a lease_id and lease_duration 1h. The lease is the time limit attached to that login.

Three things improve at once. A leaked login stops working on its own. Revocation is a single command on the lease. Because every service has its own SQL username, the docs note that auditing becomes much easier.

The OWASP cheat sheet recommends dynamic secrets where possible. It also makes a point many teams miss: restarting the app does not revoke a stolen credential. Only expiry or revocation at the source does.

Dynamic secrets have a cost. Your app must renew leases and reconnect with new credentials, and your database sees many short-lived users. Use them first for the highest-risk access, such as production databases and cloud accounts.

You rotate without downtime by letting two valid secrets overlap. Create the new one, make the backend accept it, switch clients over, and only then retire the old one.

Downtime comes from the gap between changing the password in the database and every client learning about it. AWS Secrets Manager names two strategies for that gap. Single user rotation changes one user's password. AWS says this leaves a short window with a low risk of denied calls, and advises a retry strategy.

Alternating users rotation keeps two database users, user and user_clone, and changes the password of the one not in use. After rotation both are valid, so an app that fetches the secret mid-rotation still gets working credentials. AWS suggests it for high-availability apps.

When rotation runs as an AWS Lambda function, a small piece of code AWS runs for you, the work splits into four named steps. Labels mark each version of the secret:

1. 1

   createSecret

   Generate a new value and store it as a new version labelled AWSPENDING. Nothing uses it yet.

2. 2

   setSecret

   Change the credential in the database or service so it accepts the AWSPENDING value.

3. 3

   testSecret

   Log in to the database or service with the AWSPENDING value to prove it works.

4. 4

   finishSecret

   Move the AWSCURRENT label to the new version. The old version gets AWSPREVIOUS, so the last known good value is kept.

The same pattern works without AWS. In the delivery app, if the payment gateway allows two active keys, the team can add the new key, move every service to it, then delete the old one.

Your app has to cooperate

Rotation fails quietly when apps cache secrets forever. Read secrets through the secrets manager's client or a mounted file, refresh them on a timer or on login failure, and retry once with the fresh value.

Kubernetes shows why delivery matters. The docs say Secrets mounted as volumes are updated in place, eventually. A Secret mounted through subPath does not receive updates. An environment variable is fixed when the process starts, as our /proc test showed, so it needs a restart.

You remove it by letting the platform vouch for the program. The workload gets a short-lived signed token that proves who it is, and trades it for access, so no long-lived password is stored.

Every vault has a bootstrap problem: the app needs some credential to log in to the vault itself. The OWASP cheat sheet notes you often have to secure the primary secret of the secrets manager in another system.

CI pipelines show the fix clearly. GitHub documents that with OpenID Connect, or OIDC, each job gets a fresh signed token from GitHub. The cloud provider checks it and returns an access token that works only for that single job, then expires.

GitHub's docs say this means you no longer duplicate cloud credentials as long-lived GitHub secrets. The cloud side's trust settings decide which repositories and branches may use which role. Set them narrowly, so no other workflow can borrow production access.

Cloud platforms and Kubernetes offer the same idea for running services, through attached roles and service account tokens. The exact setup differs per platform, so follow its official guide rather than copying settings from a blog.

Treat log masking as a seatbelt, not a guarantee. GitHub Actions redacts registered secrets from logs, but its docs say automatic redaction is not guaranteed, because a value can be transformed in many ways.

• Do not wrap several secrets in one JSON, XML or YAML blob. GitHub's docs say structured data can make redaction fail, and advise one secret per value.
• Register anything derived from a secret, such as a signed token or a base64 copy, with the add-mask workflow command so it is redacted too.
• Never echo a secret to check that it is set. Print its length or a yes or no instead.
• Remember forks. GitHub does not pass secrets, other than GITHUB_TOKEN, to workflows triggered from a forked repository.
• Give CI the narrowest access. OWASP suggests letting the deployed app fetch its own secrets, so the pipeline never sees them.

You run a scanner that reads every commit, not just the files on disk, and you add a check that runs before each commit. Then, if anything is found, you rotate first and clean up second.

We scanned the demo repository from earlier with two open-source scanners. We used gitleaks 8.16.0 from the Ubuntu 24.04 package archive and detect-secrets 1.5.0 from PyPI. Both are older releases, so newer versions may find more.

# inside the demo repo from earlier
gitleaks detect --source . --redact      # reads every commit
# WRN leaks found: 1   (exit code 1)

gitleaks detect --source . --no-git      # reads only the files on disk
# INF no leaks found   (exit code 0)

gitleaks protect --staged                # what a pre-commit hook runs
# exit code 1 if the staged change holds a secret

Two lessons came out of this. A scanner that only reads current files misses exactly the leak git keeps forever. And tools differ: gitleaks 8.16.0 did not flag the password inside the database URL, while detect-secrets did. This is our observed behaviour; pin your versions and test them.

Scanners match patterns and high-randomness strings. A plain password like the fake one above can slip past one tool and not another. Treat a clean scan as a good sign, not proof.

What GitHub does on its side

As of 25 October 2026, GitHub's docs say secret scanning reads the entire git history on all branches, and runs free and automatically for public repositories. Private organisation repositories need GitHub Secret Protection.

When GitHub finds a partner's secret type, it notifies that provider so they can act, for example by revoking the key. Push protection goes further and blocks a push that contains a secret. For your own pushes to public repositories on GitHub.com, it is on by default.

What to do when a secret leaks

1. 1

   Revoke or rotate first

   Replace the exposed value at its source right away. OWASP's incident guidance calls for immediate revocation of exposed keys.

2. 2

   Check the audit logs

   Look at what the key was used for between the leak and the revocation. This is where a secrets manager's audit log pays for itself.

3. 3

   Remove it from where it leaked

   Delete it from code, logs and tickets. Rewriting git history is optional clean-up, and it does not help anyone who already cloned the repository.

4. 4

   Add the missing guard

   Add a pre-commit scan, turn on push protection, or move that secret into the vault so the same mistake cannot repeat.

Start with the simplest option that gives you access control and an audit log for production. Move to dynamic secrets when the cost of a leaked long-lived password becomes too high.

Whichever you choose, the same three habits matter most. Each workload gets its own identity and its own narrow secrets. Every secret has an owner and a rotation plan. Every repository has a scanner before commit and after push.

Interviewers want to hear that you treat secrets as a lifecycle, not a storage location. Use this frame, and tie each step to the system you are designing.

1. 1

   Name the secrets

   List them: database logins, third-party API keys, signing keys, TLS keys. Say which service needs each one.

2. 2

   Pick the store

   Central secrets manager, encrypted with envelope encryption, master keys in a KMS. Never in code or images.

3. 3

   Solve identity

   Workloads authenticate with platform identity or OIDC, so there is no long-lived bootstrap password.

4. 4

   Limit and log

   Least-privilege policies per service, and an audit log of every read that cannot be tampered with.

5. 5

   Plan rotation

   Short TTLs or dynamic credentials where possible, and two-version overlap for keys that must be rotated.

6. 6

   Plan for leaks

   Pre-commit and repository scanning, masked CI logs, and a tested runbook: revoke, check audit logs, clean up.