Staged rollout

A staged rollout releases a change to a small, representative group of users or hosts first, compares that group's health with the rest for a set bake time, and widens the release only if nothing gets worse. If a signal regresses, distribution stops and the group returns to the last known good version.

Tests cannot reproduce every production setup, and security fixes are often rushed. In July 2019 one Cloudflare WAF rule with a backtracking regular expression exhausted CPU on every core serving HTTP worldwide; the postmortem notes that the rule change went "globally into production without a staged rollout". A canary ring limits that kind of failure to the hosts in it.

Who: Release, SRE or platform engineers who own the deployment pipeline, with the service owner choosing the halt signals
When: At every production release, including configuration, feature flags, detection rules and other content, not only code
How often: Every release, code or content; the rollback target checked before each promotion
You end up with: Defined rings with bake times, an automatic halt on regressions, and a record of a rollback that worked

Do this

Define rings and bake times
Split the fleet or user base into rings: internal users, a small canary that mirrors production's platforms, regions and configurations, then progressively larger rings. Give each ring a minimum bake time and a minimum size, set by how long your failures take to show: a crash at startup shows in minutes, a crash at the next reboot or a slow memory leak only much later.
Choose a few signals and a control group
Compare each ring with hosts or users that did not get the change, over the same period, rather than with a fixed threshold. Google's SRE Workbook suggests a small set of metrics, perhaps no more than a dozen, ranked by how well they reflect problems users see: errors, latency, crashes, failed health checks. For a security change, add the signal it should move, such as requests blocked by a new WAF rule.
Halt automatically, resume by hand
Make the gate stop distribution and roll the ring back on a regression without waiting for a person; a person decides when to resume. Kubernetes alone does not do this: a stalled Deployment only gets a ProgressDeadlineExceeded condition, so a pipeline step or rollout controller must act on it, for example with kubectl rollout undo. A plain rolling update is not a canary: nothing stops it unless you add a gate.
Stage content and configuration like code
Send feature flags, remote configuration, WAF and detection rules and other content through the same rings as code. If a file changes what privileged code does, treat it as code for release purposes. Keep an emergency path for active attacks that shortens bake times but still starts with a canary, and needs a second person to approve a global push.
Keep a safe rollback target
Keep the previous release deployable: Kubernetes keeps 10 old ReplicaSets by default, and revisionHistoryLimit: 0 removes rollback entirely. Keep database changes compatible with the previous version for at least one release. For agents, apps and devices, keep the last known good version on the host so rollback works offline, behind an anti-rollback floor so it never reinstalls a version with a known vulnerability.
Rehearse the rollback
Before promoting a release, prove on a test target that the rollback target is present, intact and allowed. Inject a failure into a test release and confirm that the pipeline halts at the canary and rolls back without anyone acting. Keep the result as evidence that the halt and the rollback work.
Let customers choose their ring
If customers run your software, let them place systems in an early, general or delayed ring, with a limit on how long a security fix may wait, and publish when a rollout halts. The SSDF 1.2 initial public draft adds practice PS.4, which asks for tiered rollouts, robust rollback and, preferably, customer control of updates; SSDF 1.1 has no practice for it.

Done when

Rollback has been exercised, and a bad release can be stopped before it reaches everyone.

Mark it in the gap check

Common mistakes

Treating urgent as global. A rushed fix pushed everywhere at once turns one bad change into an outage; shorten the bake times, keep the canary.
Staging code but pushing rules, flags and configuration straight to everyone. The Cloudflare WAF outage of 2019 and the CrowdStrike content update of July 2024 both came through such a fast path.
Using a canary that does not mirror the fleet. A failure that appears only on one operating system, region or configuration passes a ring that contains none of them.
Expecting Kubernetes to roll back a failed rollout on its own. It only reports ProgressDeadlineExceeded; something else has to act on it.
Judging a ring against a fixed threshold instead of a control group, so normal daily swings hide a regression or trigger false halts.
Rolling back below your security floor, which reinstalls the vulnerability the last release fixed.

An example ring plan

Ring sizes and bake times are examples; set yours from your own incident history and control-group baselines.

An example ring plan: table
Ring	Who	Example bake time	Moves on when
0: internal	Your own staff and test fleets, on every supported platform	Hours	No regression in errors, crashes or health checks; functional checks pass
1: canary	A small external slice that mirrors the fleet, including customers who opted in early	Hours, including at least one restart or reboot cycle	No regression against the control group, with enough hosts or users reporting
2 and later: broad	Progressively larger slices	Hours to a day each	The same gate on a larger sample
Late	Systems customers marked critical, or chose to delay	After broad rollout is complete	The customer's own schedule

Tools and guides

Staged rollout and rollback for security-critical updates
Ship security fixes, detection content and configuration quickly without taking the fleet down: rollout rings and canaries, content validation, kill switches and rollback that works on a broken host.
Case study: CrowdStrike Channel File 291 (July 2024)
How a CrowdStrike content update crashed Windows hosts in July 2024: a field-count mismatch hidden by wildcard tests, a missing bounds check, and no staged rollout.
Design firmware updates that can recover safely
Connect trusted boot, update authorization, rollback policy and end-of-life behavior to the device’s actual failure modes.

Standards

OWASP SAMM 2: Secure Deployment

Sources

Google SRE Workbook, chapter 16: Canarying Releases
Kubernetes: Deployments (rolling updates, rollback and progress deadlines). Kubernetes v1.37.
Cloudflare: Details of the Cloudflare outage on July 2, 2019. 12 July 2019.
CrowdStrike: External Technical Root Cause Analysis, Channel File 291. 6 August 2024.
NIST SP 800-218 Rev. 1 (initial public draft): Secure Software Development Framework version 1.2. Initial public draft, 17 December 2025.
The Update Framework specification
OWASP SAMM: Secure Deployment

Do this

Define rings and bake times

Choose a few signals and a control group

Halt automatically, resume by hand

Stage content and configuration like code

Keep a safe rollback target

Rehearse the rollback

Let customers choose their ring