Feature flags, canary releases, and the art of not breaking production (too badly)

Posted on June 22, 2026  •  12 minutes  • 2468 words

Every team has its version of “we don’t deploy on Fridays” or “nobody touches payments at month end.”

It’s not in the internal handbook, but everyone treats it like it’s carved in stone. It didn’t come from theory — it came from long nights, heroic rollbacks, and that shared, unspoken “never again” feeling.

The problem is that the business doesn’t care about your trauma. It wants changes. Many, and often.

You want to keep your job (and your sanity), so you have to thread the needle: ship to production every day without turning every deployment into an extreme sport.

That’s where two of the best inventions of the past decade come in: feature flags and canary releases .

Decoupling “shipping code” from “turning it on”: the quiet revolution of feature flags

The old way was simple and unforgiving: a new version of the code, a deployment, and everyone crossing their fingers at the same time.

The day someone decided to separate when you ship the code from when you activate it, the game changed. That decision has a name: feature flag .

A feature flag is a switch that lives inside your application and decides whether a given feature behaves the old way or the new way.

The important part isn’t just the if — it’s what that implies: you can deploy code turned off, test it internally, turn it on for 1% of users, for a specific region, for employees, for a closed beta… all without redeploying .

Flag platform vendors (who obviously have a stake in your adoption) describe this as basically a superpower: continuous deployment without fear, instant kill switch if something breaks, A/B testing and experiments without branch nightmares.

And they’re right… up to a point. Because like any useful power, flags come with real risk baked in .

Recent posts from LaunchDarkly , Unleash , Flagsmith , and PostHog have entire pages dedicated to the classic mistakes that turn a flag system into a nightmare.

The first one is almost poetic in how predictable it is — flagrantly ignoring the single responsibility principle: trying to make one flag do two or three things at once.

“This flag activates the new screen AND ALSO the new price calculation AND WHILE WE’RE AT IT the checkout experiment.” If describing a flag requires the word “and,” you’re planting landmines for your future self.

Another classic: mixing business logic with flag logic until everything turns into a tangled mess — nested conditions, checks scattered across five different places, business rules hiding inside if (flagActive) statements spread across half the codebase.

PostHog covers this in gory detail: badly placed flags have caused serious outages, mountains of technical debt, and in one famous case, nearly sank a company because nobody could figure out which combination of flags made the system behave correctly anymore.

Modern best-practice guides keep hammering on things that sound (and are) boring — and work precisely because of it: every flag should have one clear responsibility, live as close to the boundary as possible (at the entry point to a flow, not buried 20 layers deep), and come with an exit plan.

Unleash and others put it plainly: flags are technical debt with an expiration date. If you don’t clean them up once they’ve done their job, they’ll come back to bite you.

When done right, the play is actually elegant: you put the flag at the boundary (“use the old checkout code or the new one”), deploy, test, run a canary with a subset of users, collect metrics… and once you’re confident, delete the flag and the old code path.

What was open-heart surgery becomes pulling a dead code stitch.

Canary releases: sending in the brave ones first

If feature flags decide which code path runs, canary releases decide who sees which version.

The idea comes from the old coal mine stories where a canary was sent in before the miners: if something was wrong, you’d find out at canary scale, not human scale. Not great for the bird, but it works remarkably well in practice.

In a canary, instead of serving the new version to everyone at once, you serve it to a small fraction of traffic (1%, 5%, sometimes just a handful of internal users) and watch.

Watching isn’t just “ship and pray” — it’s wiring the canary into your metrics and logs, to the point where many tools automate the decision. If error rates spike or latency crosses a threshold, the system cuts the experiment and rolls back to the previous version .

Two different philosophies for the same problem. Blue/green bets on cleanliness: you keep two full environments (blue, the current one; green, the new one) and switch instantly between them, with rollback just as fast. The cost is that it requires doubling your infrastructure, and when you flip the switch, you flip it for everyone at once.

Canary bets on the opposite: it reuses the same infrastructure and plays with percentages — slower, more granular, with the ability to collect real-world data without putting your entire user base at risk (canary vs. blue/green comparison on Stack Overflow ).

Some vendors have formalized this so much they now give you full recipes for “staged canary”: first 1%, then 10%, then 25%… with automatic promotion conditions based on error rates and performance. What used to be “let’s see what happens” becomes an experiment with defined exit criteria, like those offered by Flagger .

And this is where feature flags and canaries shake hands: the canary decides what version of the app a percentage of your traffic sees; the flag decides what functionality within that version a given type of user sees. An Unleash article puts it well: flags bring the canary from the infrastructure level down to the application level, letting you do fine-grained gradual rollouts by segment, region, or even individual user.

How to cut risk without killing velocity (and without building a flag hellscape)

The pitch is tempting: safe deployments, fewer incidents, the freedom to experiment without white-knuckling it. But the people who know these tools best — the ones who sell and operate them day in and day out — have been sounding the same alarm for years: used carelessly, flags and canaries can cause more damage than they prevent, as ConfigCat makes clear.

And Flagsmith, ConfigCat, DevCycle, and company all share the same horror stories. Flags that never get deleted turn every code flow into a Christmas tree of untestable conditions.

Flags used to control critical business rules or permissions, without proper oversight or auditing, open the door to security incidents . And explosive combinations that nobody ever tested together make the number of possible system states explode until nobody knows which combination is “how it’s supposed to work.”

The most ironic part is that almost universally it started with good intentions: “let’s add a couple flags to be safe.” Two years later, the first thing every new developer has to do is decode which set of flags makes the system behave the way it’s supposed to.

Modern best practices talk a lot about flag lifecycle management: document why it exists, who owns it, when it should die. They also insist on visibility into which flags are active in which environments, and on banning the reuse of old flags for new purposes “because they were already there,” according to DevCycle .

Canaries have a similar pitfall: designing which metrics you’ll actually watch before launching makes the difference between “we mitigated risk” and “we created extra work.” If you only watch technical error rates but miss a business metric collapse (conversions, cart abandonment, etc.), you can declare victory while users are quietly walking out the door.

The core insight is that flags and canaries don’t replace judgment — they amplify it.

If the judgment is “ship this no matter what,” you’ll end up with flags used as quick glue and canaries run once and never again. If the goal is “I want to be able to test and revert without a fire drill,” they become part of the team’s language: “we ship it behind a flag, start with a 5% canary, measure, and let the data decide.”

Real examples (and cautionary tales)

Some public cases are practically textbook. Companies running massive UI rollouts (dark mode, flow redesigns) typically use flags to deploy the full code days or weeks ahead, activate it first for employees, then for a slice of beta users — and only when they’ve confirmed nothing important is sinking, for everyone.

If something goes wrong, the kill switch is instant: flip the flag off, the old UI comes back like nothing happened, and the team has time to fix the issue without any firefighting.

Then there are the horror stories, like the one PostHog describes of a company that nearly went under because of a badly managed flag combination: nested flags, tangled business logic, new changes built on top of flags nobody remembered — until an apparently innocent deployment triggered behavior nobody had anticipated.

Hence their personal crusade against “multipurpose flags” and codebases where if (flag) statements act as goto in disguise.

On the canary side, plenty of teams describe going from “deployments are always nerve-wracking” to “we ship several times a day” simply by routing a small initial percentage of traffic and giving SRE/product 30 minutes to check the dashboards before ramping to 100%.

Some platforms, like Flagger , even integrate automatic rules: if latency or error rates cross a threshold, the canary freezes or rolls back without anyone having to stare at a screen.

There are also the sketchy uses: canaries that in practice always start at 50%, get “monitored” by waiting to see if someone yells, and get promoted on pure inertia; flags used as permanent workarounds for design problems (“if it breaks, send the user back to the old flow”) until nobody dares touch anything because the map of possible code paths is a nightmare.

The difference between an example you want to copy and one you want to avoid usually comes down to two words: intent and discipline.

If the intent is to reduce risk and learn, and you clean up what you no longer need, flags and canaries become allies.

If the intent is to move fast and leave traps behind, they’ll blow up on a Friday at 6 PM.

The art (and ethics) of not breaking production “too badly”

Nobody can promise you’ll never break production. Not flags, not canaries, not blue/green, not whatever deployment trend is currently dominating the conference circuit.

What is within your power is shifting the problem from “inevitable global catastrophe” to “contained, reversible incident that — with any luck — is nearly invisible to users.”

Feature flags let you turn behavior on and off like flipping a light switch: if something looks off, you can kill just that piece without rolling back the entire deployment.

Canary releases let you test in the wild on a small slice of your traffic before committing to the rest.

Together, used well, they’re a form of technical humility: accepting that you won’t get everything right on the first try, and designing your system as if that were true.

The best modern guides on safe deployment don’t talk much about heroism — they talk about craft: simple flags with a limited lifespan, canaries with clear metrics, pipelines that catch the obvious stuff before it reaches production, and teams that understand that speed and caution aren’t mutually exclusive, as the best CI/CD guides describe.

In that world, “don’t break production (too badly)” stops being a dark joke you share over coffee and becomes a reasonable goal: yes, something will fail eventually — but it’ll fail in a controlled way, with awareness, and above all, with a plan to flip the flag off, wind down the canary, and roll back to a known state without anyone sacrificing their weekend.

And if, despite everything, things really do go sideways… you can at least console yourself knowing you’re no longer doing it with a git push origin main straight to every user on the planet.

That’s progress too.

Quick glossary

The vocabulary of safe deployment. Quick version.

• Feature flag: a switch in code that enables or disables a feature without redeploying. Lets you decouple when you ship code from when you turn it on.
• Canary release: a deployment strategy that exposes the new version to a small percentage of traffic before rolling it out broadly. Named after the canaries sent into mines to detect danger before the miners went in.
• Blue/green deployment: a technique that maintains two identical environments (one live, one on standby) to switch between versions instantly, with equally instant rollback.
• Rollback: reverting a deployment to the previous version when something goes wrong. The technical equivalent of undo.
• Kill switch: a mechanism to disable a feature immediately without deploying new code. A well-placed feature flag acts as a kill switch.
• A/B testing: an experiment that shows two variants to different groups of users to compare results and decide which one performs better.
• Technical debt: accumulated cost of quick decisions made without thinking about future maintenance. Flags that never get cleaned up are technical debt with interest.
• CI/CD pipeline: an automated chain of build, test, and deployment steps that runs on every code change. A good pipeline catches the obvious problems before they reach production.

Sources and references

Because nobody should take someone else’s if (flag) on faith. Here are the switches that powered these ideas.

• LaunchDarkly - What are feature flags? - LaunchDarkly. Official introduction to the feature flags concept.
• Unleash - What is a canary deployment - Unleash. Detailed explanation of canary releases.
• Feature flags: the secret to safe canary testing - FasterCapital. The relationship between flags and canary testing.
• Flagsmith - Pitfalls of feature flags - Flagsmith. Common mistakes when using feature flags.
• LaunchDarkly - Feature flag mistakes - LaunchDarkly. The most common mistakes in flag systems.
• PostHog - Feature flag mistakes - PostHog. Real-world cases of badly managed flags, including one near-catastrophic example.
• Unleash - Best practices for feature flags at scale - Unleash. Guide to flag lifecycle, ownership, and cleanup.
• Octopus Deploy - Feature flag best practices - Octopus Deploy. Security, control, and auditing of flags.
• Unleash - Blue/green vs canary release - Unleash. Detailed comparison between both strategies.
• Stack Overflow - Canary vs. blue/green - Stack Overflow. Technical discussion with pros and cons of each approach.
• Flagger - Deployment strategies - Flagger. Documentation on automated canary with metrics and promotion criteria.
• ConfigCat - Feature flag pitfalls - ConfigCat. Analysis of the dangers of flag overuse.
• DevCycle - Dos and don’ts of feature flags - DevCycle. Flag lifecycle: document, control, and retire.
• CI/CD best practices: branching, approvals, and rollouts - Skywork AI. Pipeline best practices for safe deployment.