Staged rollout and rollback for security-critical updates

Security teams are under pressure to ship fixes and detection content quickly, and that pressure is exactly what turns a bad update into a fleet-wide outage. On 2 July 2019 a new Cloudflare WAF rule with a regular expression that backtracked badly exhausted CPU on every core serving HTTP worldwide; Cloudflare's postmortem says 'The SOP allowed a non-emergency rule change to go globally into production without a staged rollout', while its software releases went through internal, canary and global stages that could take hours or days. On 19 July 2024 a CrowdStrike content update crashed Windows hosts running its sensor; the root cause analysis lists 'Template Instances should have staged deployment' among its findings (/domains/appsec/platform-security/case-study-crowdstrike-channel-file-291/).

In both cases the update was content, not code, the pipeline treated it as safe to push everywhere at once, and the code that interpreted it failed. In both cases the fix was the same: stage content the way code is staged, and keep a separate, still-staged path for emergencies. NIST has since drafted the practice into the Secure Software Development Framework: the SSDF 1.2 initial public draft adds PS.4, 'Ensure Software Updates Are Robust and Reliable'.

Google's SRE Workbook defines canarying as 'a partial and time-limited deployment of a change in a service and its evaluation.' A ring is one step of that: a defined population that gets the update, a bake time during which you watch it, and a gate that decides whether the next ring gets it. Ring membership matters as much as size: a canary must include the operating system versions, hardware, configurations and workloads the fleet has, or it will miss the failure that only appears on one of them.

Compare each ring with a control group that did not get the update, over the same period, rather than with a fixed threshold; otherwise a busy Monday looks like a regression and a quiet Sunday hides one. The Workbook suggests starting from service level indicators and keeping to a small set, 'perhaps no more than a dozen'. For agents and clients, the signals that matter most are crashes, failed boots or automatic rollbacks, hosts that stop sending heartbeats, CPU and memory, and error rates in the product's own telemetry.

Bake time has to cover how the failure would show. A crash at load shows in minutes; a crash at the next reboot only after hosts reboot; a memory leak over a day. Set minimums per ring from your own incident history, and require a minimum number of hosts as well as a minimum time.

Many products ship two kinds of change: code releases, which go through a long test and rollout process, and content, such as detection rules, signatures, channel or configuration files, feature flags, allow and block lists, or models, which is pushed through a fast path because it changes often and is 'just data'. CrowdStrike described its channel files as 'configuration data; it is not code or a kernel driver.' The file was data; its effect was to choose which array element kernel code read.

The test is what the change can do, not what it is called. If content can make privileged code crash, hang, consume CPU, block traffic or stop protecting the host, give it the same rings and gates as code. Cloudflare's 2019 rule was deployed in 'simulate' mode, which logs instead of blocking, but the rule still ran and still consumed CPU. A feature flag that turns on untested code is a code release with a different name.

Keep a fast path, because security content sometimes has to beat an active attack. Make the fast path faster staging, with shorter bake times and the canary always on, rather than a bypass. Cloudflare's post-incident procedure kept 'the ability to do emergency global deployment for active attacks'; write down who can declare that, and make the global push itself the exception that needs a second person.

Schema validation proves that a file is well formed. It does not prove the consuming code can handle it. CrowdStrike's validator checked new content against a definitions file that said a template had 21 input fields; the sensor code supplied 20. Generate the interface description from the consuming code at build time (field counts, types, ranges, limits), and validate content against that, so the two cannot drift apart.

Run the content through the consumer in CI as well: load each new content file into the real parser or interpreter, built from the release branch, with sanitizers on, and replay representative inputs. Exercise every field with a real value at least once; CrowdStrike's tests used a wildcard in the 21st field throughout, so the missing input was never read. Fuzz the parser that reads content (/domains/appsec/testing/parser-fuzzing/), keep runtime bounds checks in release builds, and for rules that run regular expressions, bound their cost, as the ReDoS guide (/domains/appsec/input-handling/redos-input-validation/) describes.

Sign content as you sign code and have the client verify it before loading, so the fast path is not also an unauthenticated path into privileged code. The trusted publishing guide (/domains/supply-chain/releases/trusted-publishing-and-updates/) covers signing and update metadata.

The gate below compares a ring with a control group and returns halt, hold or promote. Any boot-failure regression halts at once, whatever the bake time; a crash or missing-heartbeat rate above a multiple of the control rate halts; otherwise the ring holds until both the bake time and the minimum host count are met. The thresholds are examples to tune from your own baseline. Its four unit tests (halt on early boot failures, halt on a crash spike, hold when young, promote after bake time) ran with Python 3.14. Wire 'halt' to stop distribution and start rollback without waiting for a person.

"""Decide whether an update may move from one rollout ring to the next.

Compare the ring that has the update with a control group that does not
(same hardware and OS mix, same period). Halt on a boot, crash or heartbeat
regression; hold until the bake time and sample size are met; else promote.
The thresholds are examples: set them from your own fleet's baseline.
"""
from dataclasses import dataclass


@dataclass(frozen=True)
class Health:
    hosts: int  # hosts with the update (or control hosts)
    crashes: int  # crashes or agent restarts in the window
    boot_failures: int  # failed boots or automatic rollbacks
    heartbeats_missing: int  # hosts that stopped reporting


@dataclass(frozen=True)
class Policy:
    min_hours: float = 4.0
    min_hosts: int = 500
    max_crash_ratio: float = 1.5  # ring rate may be at most 1.5x the control rate
    max_missing_ratio: float = 2.0


def rate(count, hosts):
    return count / hosts if hosts else 0.0


def decide(ring: Health, control: Health, hours: float, policy: Policy = Policy()):
    if ring.boot_failures > rate(control.boot_failures, control.hosts) * ring.hosts:
        return "halt", "boot failures above the control group: stop and roll back"
    floor = 1 / max(control.hosts, 1)  # a zero baseline must not hide a spike
    checks = {
        "crash": (ring.crashes, control.crashes, policy.max_crash_ratio),
        "missing heartbeat": (ring.heartbeats_missing, control.heartbeats_missing, policy.max_missing_ratio),
    }
    for name, (ours, theirs, limit) in checks.items():
        ours, theirs = rate(ours, ring.hosts), rate(theirs, control.hosts)
        if ours > max(theirs, floor) * limit:
            return "halt", f"{name} rate {ours:.3%} vs control {theirs:.3%}: stop and roll back"
    if hours < policy.min_hours or ring.hosts < policy.min_hosts:
        return "hold", f"baked {hours:.1f} of {policy.min_hours} h on {ring.hosts} of {policy.min_hosts} hosts"
    return "promote", "healthy for the full bake time"

Have two switches and test both. A distribution halt stops the update service from offering a release or content version to any more hosts; the promotion gate should be able to throw it automatically. A client-side disable turns off one content channel or feature on hosts that already have it, so a bad rule can be neutralized without shipping a new one; it is only useful if the client checks it before loading the content, including early in boot.

A kill switch is also an attack surface: anyone who can throw it can turn off protection across the fleet. Authenticate and sign switch messages like updates, limit who can throw them, log every use, and decide in advance whether a disabled channel fails open or closed. Cloudflare's 2019 postmortem shows the other failure: its responders could not reach the internal control panel because their own access service was down, and used a 'bypass mechanism that wasn't frequently used'. Drill the switch and the access path to it, not just the code.

Rolling back at the source helps only hosts that can still fetch the next version. CrowdStrike reverted its bad content 78 minutes after release, but machines already crashing at boot needed someone to delete the file by hand. Design the client so it recovers without the network: keep the last-known-good release or content set on the host, count consecutive failed boots or failed health checks, and fall back automatically. Android's A/B updates are the model for system images: if an update 'fails to boot, the device will reboot back into the old partition and remains usable'. The embedded update guide (/domains/appsec/platform-security/embedded-update-security/) covers A/B slots and boot counters on devices.

Rollback must not reopen a vulnerability. TUF's specification sets the goal that 'An attacker cannot trick clients into installing software that is older than that which the client previously knew'. Keep an anti-rollback floor, the oldest version the client will accept, raise it when you ship a security fix, and make sure the last-known-good copy is always at or above it. The SSDF 1.2 draft asks for both: robust rollback (PS.4.3), with an example to 'Implement protection mechanisms that prevent unauthorized roll-back to vulnerable software versions'.

Rehearse it. Before each promotion, prove on a test host that the rollback target is present, intact and allowed.

This drill reads the release manifest the client trusts and checks that the last-known-good artifact is on disk, matches its recorded SHA-256, and is not below the anti-rollback floor. Run against a correct manifest it prints 'rollback target OK'; against one with a wrong hash and a version below the floor it reports both problems and exits 1. In production the manifest itself must be signed and verified first, as with TUF or your platform's update metadata.

"""Rollback drill: prove the last-known-good release can be restored.

Reads the release manifest the update client trusts and checks that the
last-known-good artifact is on disk, matches its recorded SHA-256, and is not
below the anti-rollback floor (versions with known vulnerabilities that the
client must refuse). Run it on a test host before each ring promotion.
"""
import hashlib
import json
import sys
from pathlib import Path


def version_tuple(v):
    return tuple(int(part) for part in v.split("."))


def check(manifest, root):
    lkg, floor = manifest["last_known_good"], manifest["anti_rollback_floor"]
    path = Path(root) / lkg["file"]
    if not path.is_file():
        return [f"missing last-known-good artifact {path}"]
    problems = []
    digest = hashlib.sha256(path.read_bytes()).hexdigest()
    if digest != lkg["sha256"]:
        problems.append(f"{path.name}: sha256 {digest} does not match the manifest")
    if version_tuple(lkg["version"]) < version_tuple(floor):
        problems.append(f"last-known-good {lkg['version']} is below the anti-rollback floor {floor}")
    return problems


if __name__ == "__main__":
    manifest = json.loads(Path(sys.argv[1]).read_text(encoding="utf-8"))
    problems = check(manifest, sys.argv[2])
    print("\n".join(problems) or "rollback target OK")
    sys.exit(1 if problems else 0)

Customers know which of their systems they can afford to lose for an hour; you do not. Let them assign hosts to an early, general or delayed ring, and to hold critical systems one version behind, with a cap on how far behind a security fix may lag. The SSDF 1.2 draft's PS.4 asks producers to implement update strategies 'preferably allowing customers to control any updates', and its PS.4.2 examples include letting customers identify systems 'considered safe to receive early updates' and staged rollouts 'with built-in wait times for customer feedback'. CrowdStrike added customer control over where and when its content deploys after July 2024.

Publish what changes and when: release notes for content as well as code, the current ring a version is in, and a status page entry when a rollout halts. Customers who see a halt can hold their own late ring instead of guessing.

SSDF 1.1 (NIST SP 800-218, February 2022) has no practice for how updates are delivered; its testing and release practices cover parts of it. The SSDF 1.2 initial public draft (SP 800-218 Rev. 1, December 2025) adds PS.4 'Ensure Software Updates Are Robust and Reliable', with four tasks: PS.4.1 'Thoroughly test all releases following all guidance in PW'; PS.4.2 'Use tiered update and release strategies, such as canaries, staged roll-out, and test deployments'; PS.4.3 'Include robust rollback mechanisms for updates'; and PS.4.4 'Maintain robust and reliable update engines and associated infrastructure for the delivery of updates'. As of October 2026 it is still a draft; cite it as one, and see the SSDF guide (/frameworks/ssdf-product-lifecycle/) for how the final 1.1 practices map to evidence.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. NIST SP 800-218 Rev. 1 (initial public draft): Secure Software Development Framework version 1.2. Initial public draft, December 2025.
2. NIST SP 800-218: Secure Software Development Framework version 1.1. Version 1.1 (February 2022).
3. Google SRE Workbook, chapter 16: Canarying Releases
4. Cloudflare: Details of the Cloudflare outage on July 2, 2019. 12 July 2019.
5. CrowdStrike: External Technical Root Cause Analysis, Channel File 291. 6 August 2024.
6. Android Open Source Project: A/B (seamless) system updates
7. The Update Framework specification. Version 1.0.37.

Reference information

Tags

Signing and software updatesFirmware and hardware securitySecure development lifecycle

SDLC phase

Release

Role

AppSec engineersPlatform engineers and SREs

Download this guide as Markdown

Continue exploring