Feature Flag System Design: Sticky Rollouts, Safe Defaults and Kill Switches

Without flags, every change reaches every user at once, and the only undo is another deploy. Good feature flag system design fixes that, but only if the flags themselves cannot break.

Picture a common setup. A food delivery app is replacing its checkout page and wants to try it on 10% of users in Pune first. The setup is made up, but the problems are the ones real teams hit.

The team also has a recommendations widget that calls a slow service. During the dinner rush, that service sometimes stalls and drags the whole checkout down. They want a switch that hides the widget within seconds.

A feature flag system design gives them both. The new checkout ships dark, which means the code is deployed but switched off. Then the team turns it on for small groups and watches the numbers.

Flags bring their own failure modes, and this guide is about those. A user can flip between old and new pages on every refresh. The flag service can go down and take checkout with it. Old flags can pile up until nobody knows what is safe to delete.

Six words carry most of this guide. Each one is explained here once and used the same way after.

• Flag: a named switch, such as new-checkout, whose answer your code reads at runtime. Martin Fowler's site calls the place in code where you read it a toggle point.
• Variant: one possible answer of a flag. A simple flag has two variants, on and off. An experiment can have three or more.
• Evaluation context: the facts about this request that rules can use, such as user ID, email or city. The OpenFeature standard calls the main ID the targeting key.
• Targeting rule: a condition like "city is Pune" that decides which variant a request gets.
• Bucket: a fixed number, from 1 to 100 here, that a user gets for one flag. Percentage rollouts compare it with the rollout percent.
• SDK: the software development kit, a library your app imports. It downloads the flag rules and answers flag checks inside your process.

Feature flag system design splits the job into two halves. A control plane is where people create and change flags. A data plane is where your servers read those flags on every request.

The control plane is a normal web app: a dashboard, an API and a database with an audit log. It can afford to be slower, because people change flags a few times a day, not thousands of times a second.

The data plane must be fast and must keep working when the control plane is down. So a common design copies the whole rule set into every app server. The flagd project calls this in-process evaluation and notes it has no I/O overhead, meaning no network or disk wait per check.

Some systems use remote evaluation instead, where the app asks the flag service for each answer. flagd supports this as RPC mode, short for remote procedure call, and its docs name the cost: the latency of a remote request. Remote evaluation suits browsers and phones, because rules shipped to a user's device can be read by that user.

The SDK walks a short, fixed order. If the flag is switched off, it returns the default. Otherwise it tries each targeting rule from top to bottom, and the first rule that matches wins.

Here is the checkout flag written in flagd's JSON format. Staff with an example.com email always get the new page. Users in Pune or Jaipur go into a 20% rollout. Everyone else stays on the old page.

{
  "flags": {
    "new-checkout": {
      "state": "ENABLED",
      "variants": { "on": true, "off": false },
      "defaultVariant": "off",
      "targeting": {
        "if": [
          { "ends_with": [{ "var": "email" }, "@example.com"] }, "on",
          { "in": [{ "var": "city" }, ["Pune", "Jaipur"]] },
          { "fractional": [["on", 20], ["off", 80]] },
          "off"
        ]
      }
    },
    "kill-recommendations": {
      "state": "ENABLED",
      "variants": { "on": true, "off": false },
      "defaultVariant": "off"
    }
  }
}

We loaded this file into flagd-core 5.0.0, the JavaScript engine behind flagd's in-process providers, and checked five requests. A tester with an example.com email got on. In Pune, user-7 got off and user-42 got on, because the 20% split put them in different buckets.

A Delhi user got off from the last line of the rule. A Pune request with no targeting key got off with the reason DEFAULT, because the split had no ID to hash. Out of 100,000 Jaipur users, 20,205 got the new page.

It gives every user a fixed bucket number for each flag and turns the flag on for buckets up to the rollout percent. The number comes from a hash, so nothing is stored per user.

A hash function turns any text into a random-looking number. The same text always gives the same number, on every server and every day. The open-source projects we tested all bucket this way, each with its own hash function.

The naive way: a coin toss per request

The first idea many people have is to call a random number function on each request. It gets the share right, but users flip. In our run, 1,788 of 10,000 users got different answers on two requests at 10%.

That is about 18%, which matches the maths: 2 times 0.1 times 0.9. A user who sees the new checkout, refreshes and sees the old one will report a bug, and an experiment can no longer tell which page caused what.

The sticky way: hash the flag name and the user ID

We tested the Node SDK of an open-source flag platform, version 6.12.2, linked in the sources. It joins the flag's group ID and the user ID with a colon, runs murmur3, takes the remainder after dividing by 100 and adds 1. Murmur3 is a fast non-cryptographic hash: it spreads values evenly but is not built for security.

The code below rebuilds that scheme in Python. It returned the same buckets as that Node SDK for every plain ASCII ID we tried. ASCII here means basic English letters, digits and symbols.

import mmh3  # pip install mmh3==5.2.0

def bucket(flag_key, user_id):
    # Same input shape as the tested Node SDK: "flagKey:userId", seed 0
    h = mmh3.hash(f"{flag_key}:{user_id}", seed=0, signed=False)
    return h % 100 + 1  # a number from 1 to 100

def is_enabled(flag_key, user_id, rollout_percent):
    # On for buckets 1 up to the rollout percent; 0 percent means nobody
    return bucket(flag_key, user_id) in range(1, rollout_percent + 1)

print(bucket("new-checkout", "user-42"))           # 75
print(is_enabled("new-checkout", "user-42", 50))   # False
print(is_enabled("new-checkout", "user-42", 80))   # True

counts = [0] * 100
for i in range(1, 100001):
    counts[bucket("new-checkout", f"user-{i}") - 1] += 1
print(min(counts), max(counts))                    # 921 1081

Run it with Python 3 after pip install mmh3==5.2.0. You should see 75, False, True, and then 921 1081. Those last two numbers are the smallest and largest bucket counts across 100,000 users.

A missing user ID breaks stickiness. The Node SDK we tested falls back to a session ID, and then to a random number. With an empty context at 50%, it said on 477 times out of 1,000 calls.

flagd chose the other way: with no targeting key, the split returned the flag's default variant. Neither is wrong, but you should know which one your SDK does.

Think of exam roll numbers. Each student gets a roll number from 1 to 100 for each exam, and the rule says roll numbers 1 to 20 sit in hall A. Opening hall A to 40 adds students 21 to 40, and the first 20 never move.

Where the picture breaks: nobody hands out these numbers. Each server works out the same number from the name and the exam, every time. And a student's number for one exam tells you nothing about the next.

The fourth and fifth rows explain why the flag name goes into the hash. If you hash only the user ID, the same 10% of users get every new feature first. One unlucky group then absorbs every bug, and the effects of two experiments get mixed together.

Growing a percentage is safe, but changing the order or the middle of a split is not. Bucketing is a number line cut into slices, so anything that moves a slice's start moves its users.

Pause and predict. A flagd split gives on 10 and off 90. You swap the order to off 90 and on 10, with the same weights. How many of 100,000 users change their answer?

In flagd-core 5.0.0, 20,109 users changed. The on slice moved from the start of the line to the end. Almost every user who had on lost it, and a new 10% gained it. Growing on from 10 to 20 kept all 10,284 earlier users, whether on was listed first or last.

Middle slices are worse. With on 10, beta 10 and off 80, growing on to 20 pushed the beta slice along the line. All 9,921 beta users lost beta. GrowthBook 1.8.0 behaved the same way: reweighting a 50/50 test to 40/60 switched 9.96% of users.

GrowthBook's own FAQ says the same thing and advises treating running experiments as immutable. Raising its coverage setting kept all 19,959 users who were already in our test. Grow coverage instead of reweighting a running test.

With in-process evaluation, the app keeps using the rules it already has in memory. The real danger is a cold start: a new server that boots while the flag service is down has no rules at all.

The Node SDK we tested uses a fallback chain. It writes each fetched rule set to a backup file, in the system temp folder by default. You can also ship bootstrap data, a rule set packaged with the app, for the very first start.

The last layer is the default in your code. The OpenFeature specification says flag checks must not throw and must return the caller's default on any abnormal execution. That rule is requirement 1.4.10.

We tested each layer against a server address with nothing listening. The table shows what came back.

The last row is a gap between the spec and this SDK. Requirement 1.3.4 says the client should return the default on a type mismatch. In 1.23.0 the client passed the wrong type through, so validate values from custom providers yourself.

Choose defaults so that the fallback is the boring, proven path. For new-checkout, the default is off. For a kill switch that hides the slow widget, the default is also off, so the widget shows as normal when the system knows nothing.

Either each SDK polls the flag service on a timer, or the service streams changes to every SDK over a connection that stays open. Polling is simpler. Streaming is faster.

The Node SDK we tested, version 6.12.2, polls every 15 seconds by default. So a kill switch can take up to one polling interval, plus any server-side caching, to reach every server. Shorter intervals add load on the flag service.

The same SDK also has a streaming mode behind an option named experimentalMode. It uses server-sent events, which is one long HTTP response that the server keeps writing new messages into.

Its source switches back to polling at once on status codes such as 401, 404 or 429. It also switches after five network errors or responses such as 500, 502, 503 or 504 within 60 seconds. We found this in the source, not the SDK readme, as of 24 October 2026; pin your version.

flagd's in-process providers receive flag definitions over a sync stream built on gRPC, a remote-call framework. It is a long-lived connection where the server pushes updates. In RPC mode, flagd sends change notifications so clients can re-check their flags.

A kill switch is a long-lived flag that turns off a non-critical feature when it hurts the system. Pete Hodgson's article on Martin Fowler's site calls these ops toggles that operators use to degrade features under heavy load.

A good kill switch is a plain on or off flag with no percentage. It is evaluated locally, reaches servers by streaming where possible, and does not depend on the service it protects.

In the delivery app, kill-recommendations defaults to off. When the recommendations service stalls at dinner time, the on-call engineer flips it on. Checkout then skips the widget and never calls the slow service.

1. 1

   Create the switch before the incident

   Add kill-recommendations with default off, and wrap the widget call so that on means skip it. Mark its type as kill switch so nobody cleans it up as stale.

2. 2

   Make the skipped path cheap

   When the switch is on, return an empty widget without any network call. A kill switch that still calls the slow service does not help.

3. 3

   Measure how long a flip takes

   Flip it in staging and time when the first and last servers change. With polling, expect up to one interval, such as 15 seconds by default in the Node SDK we tested.

4. 4

   Practise the flip

   Run a drill each quarter. A switch nobody has flipped in a year may sit on a code path that no longer works.

5. 5

   Turn it back off slowly

   Recovery can overload the slow service again. Turn the switch off once the service is healthy, and watch latency for a few minutes.

The flag platform behind that Node SDK lists kill switch as its own flag type, with a permanent expected lifetime. That label matters for the next section, because every other flag type should die.

OpenFeature is a vendor-neutral API for checking flags. Your code calls one API, and a small adapter called a provider connects it to flagd or another flag backend.

It has been a CNCF project since 17 June 2022 and has been at the incubating level since 21 November 2023. The CNCF is the Cloud Native Computing Foundation, the home of Kubernetes.

The spec defines the evaluation context and the targeting key. It also defines hooks, code that runs around each check for logging or metrics, and reasons that say why a value came back.

• TARGETING_MATCH: a rule or user targeting picked the value.
• SPLIT: a pseudorandom assignment, random-looking but repeatable, picked it.
• DISABLED: the flag is switched off in the flag system.
• DEFAULT: the value fell back to a configured default.
• CACHED: the value came from a cache.
• STALE: the value may be out of date.
• ERROR: something failed, and an error code such as FLAG_NOT_FOUND or PROVIDER_NOT_READY says what.

Providers also emit events. PROVIDER_CONFIGURATION_CHANGED means new rules arrived. PROVIDER_STALE means cached rules may be out of date. Alert on stale events, because a fleet running on old rules ignores your kill switch.

One detail surprised us. flagd-core 5.0.0 reported TARGETING_MATCH for percentage-split answers, both inside the city rule and when the split was the whole rule. It never reported SPLIT in our runs, so log the variant as well as the reason.

Two details are easy to miss and change which users see what. Both come from the hash input, the exact bytes that go into the hash function.

A new hashing scheme is a breaking change

flagd v0.18.0, released on 15 October 2026, changed how the fractional split builds its hash input. It now encodes the flag key and targeting key as CBOR, a compact binary format, before hashing with murmur3.

The release notes warn that bucket assignment may change and advise upgrading all flagd parts together. We ran the same 50/50 split on flagd-core 4.0.1 and 5.0.0. 49.84% of users got a different variant.

That is what you would expect from two unrelated hashes: about half of a 50/50 split moves. If one service upgrades and another does not, the same user can see the new checkout on one page and the old one on the next.

Non-English IDs can land in different buckets

We bucketed five IDs with the platform's Node SDK 6.12.2 and its Python SDK 6.9.1, which uses the yggdrasil-engine 2.0.0 core. Plain IDs like user-42 matched exactly.

IDs with non-ASCII characters did not. The Devanagari ID for user-1 got bucket 29 in Node and 24 in Python. The ID café-9 got 36 and 86.

The source explains it. The Node SDK's murmur3 library keeps only the low 8 bits of each JavaScript character code. The Python engine gave the same buckets as Python's mmh3, which hashes the UTF-8 bytes. This is undocumented as of 24 October 2026; pin your versions.

Keep targeting keys to plain ASCII, such as a numeric user ID, if more than one language runtime evaluates the same flag. Note also that the platform's stickiness docs describe the bucket as 0 to 100, while the Node source returns 1 to 100.

Give every flag an owner, a type and an expiry date when you create it, and delete release flags soon after they reach 100%. Pete Hodgson's article calls flags inventory with a carrying cost.

Each on or off flag can double the paths through a piece of code. Ten independent on or off flags make 1,024 combinations, and nobody tests them all. Old flags also hide dead code that can wake up.

That is what happened at Knight Capital on 1 August 2012. The order from the SEC, the US market regulator, says new code reused a flag that once switched on old, unused code. One of eight servers missed the new code. Its system sent millions of unwanted orders over 45 minutes, and the firm lost over $460 million.

That platform marks a flag potentially stale once it passes its expected lifetime. Hodgson's article lists stricter habits. Teams create a removal task with each flag, add time bombs that fail a test after the expiry date, or cap the number of live flags.

Never reuse a flag name for a new meaning. Create a new flag, and delete the old flag and its code paths in the same pull request.

Lead with the split between control plane and data plane, then go deep on one hard part. Bucketing and failure behaviour are the two parts that show real depth.

1. 1

   State the requirements

   Flag checks add no network call, changes reach servers within seconds, rollouts are sticky per user, and the app works when the flag service is down.

2. 2

   Draw the two planes

   A dashboard, API and database with an audit log on one side. On the other, an SDK in every app server that holds the rules in memory.

3. 3

   Walk one flag check

   Flag enabled, then targeting rules top to bottom, then a hash of flag key and user ID into a bucket from 1 to 100.

4. 4

   Explain how updates travel

   Streaming for speed, polling as the fallback, and a relay, a proxy that caches rules near your servers, when thousands of SDKs connect.

5. 5

   Explain failure behaviour

   Last known rules in memory, a backup file or bootstrap for cold starts, and safe defaults in code that never throw.

6. 6

   Close with flag debt

   Flag types with expiry dates, stale flag reports from usage counts, and a rule never to reuse a flag name.