Case study: CrowdStrike Channel File 291 (July 2024)

CrowdStrike's Falcon sensor runs a kernel driver on Windows. Its behavior is shaped by two kinds of content: Sensor Content, compiled into each sensor release, and Rapid Response Content, which CrowdStrike pushes from the cloud in channel files to detect new attacker techniques 'without requiring sensor code changes'. On Friday 19 July 2024 at 04:09 UTC it released a Rapid Response Content update, a new version of Channel File 291. Windows hosts with sensor 7.11 or later that were online and received it crashed, and some crashed again on restart before they could fetch a fix.

CrowdStrike reverted the content at 05:27 UTC, 78 minutes later. That did not help machines already stuck in a crash loop, because they could not stay up long enough to download the fix; they needed someone to delete the file from recovery mode or restore a snapshot. Microsoft estimated that 8.5 million Windows devices were affected. This was an availability failure. CrowdStrike's technical details post says it 'is not the result of or related to a cyberattack', and its root cause analysis (RCA) says the out-of-bounds read 'is not an arbitrary memory write issue and has been independently reviewed'.

This page follows CrowdStrike's 6 August 2024 RCA and its remediation guidance. It pairs with the guide to staged rollout and rollback for security-critical updates (/domains/supply-chain/releases/staged-rollout-rollback/).

The RCA describes the moving parts. Template Types are compiled into the sensor and define what data a kind of detection can look at. A Template Type Definitions file says how many inputs each type uses. Detection engineers write Template Instances, regular-expression matching criteria for a type, which are bundled into Rapid Response Content and delivered as channel files. A Content Validator checks channel files against the definitions file, and a Content Interpreter in the sensor's C++ code evaluates them.

CrowdStrike stresses that 'Rapid Response Content is configuration data; it is not code or a kernel driver.' That is true of the file and beside the point for release engineering. The content decided which array index kernel code would read, so a bad file could crash the machine as surely as a bad driver. Sensor releases went through an internal test gauntlet and Microsoft's Windows Hardware Quality Labs certification, which tested them with the channel files current at the time; content shipped afterward relied on the Content Validator and CrowdStrike's own tests, and it was not staged.

Treat anything that changes what privileged code does as code for release purposes: rules, signatures, channel and configuration files, feature flags and models. It needs tests against the real consumer, a canary, rings and a rollback path. The faster path for urgent content should be faster staging, not no staging.

The IPC Template Type 'defined 21 input parameter fields, but the integration code that invoked the Content Interpreter with Channel File 291's Template Instances supplied only 20 input values.' The mismatch survived sensor release testing, stress testing and several earlier content deployments 'in part' because every test and every earlier Template Instance used a wildcard for the 21st field. A wildcard matches without reading the value, so the missing input was never touched.

Three checks each compared the content with a description of the code instead of the code. The Content Validator 'based its assessment on the expectation that the IPC Template Type would be provided with 21 inputs', the RCA's 'logic error'. The automated tests used a static set of 12 cases with a wildcard in the 21st field for all Template Instances. And nothing at compile time compared the definitions file's 21 with the code's 20.

On 19 July one new Template Instance put a real criterion in the 21st field. At the next named-pipe notification, the Content Interpreter tried to read input index 20 of a 20-pointer array. The crash dump in the RCA, annotated by CrowdStrike with illustrative symbol names, shows the read in csagent!TemplateGetString with index 0x14: a page fault in kernel mode, which Windows cannot recover from. There was no runtime bounds check, so data the driver did not control decided whether it read valid memory.

Finding 6 of the RCA is one line: 'Each Template Instance should be deployed in a staged rollout.' On 19 July new content went to every online sensor, and customers had no control over when Rapid Response Content arrived. Because affected hosts crashed at the next named-pipe notification, a canary ring would most likely have shown the failure within minutes, before the content reached the wider fleet.

CrowdStrike's changes are a template for any content pipeline: Template Instances now pass canary testing and are 'successively promoted to wider deployment rings or rolled back if problems are detected', with bake-in time and telemetry at each ring; customers 'can choose where and when Rapid Response Content updates are deployed'; and content updates come with release notes. The draft of SSDF 1.2 (NIST SP 800-218 Rev. 1, initial public draft, December 2025) adds a practice for exactly this, PS.4 'Ensure Software Updates Are Robust and Reliable', with tasks for tiered rollouts such as canaries and staged roll-out (PS.4.2) and robust rollback (PS.4.3). It is still a draft.

Rollback at the source fixed every machine that had not yet crashed. For the rest, CrowdStrike's guidance was to reboot on a wired network in case the host fetched the reverted file before the crash; otherwise to boot into recovery and delete the channel files matching C-00000291*.sys, or, for cloud and virtual machines, attach the OS disk to another server and delete them there, or roll back to a snapshot from before 04:09 UTC. A channel file timestamped 05:27 UTC or later is the good version.

Encrypted disks made this slower: CrowdStrike notes that BitLocker-encrypted hosts may need a recovery key. The design lesson for anyone shipping updates is that rollback has to work when the update has broken the machine's ability to receive the next one: last-known-good content the agent falls back to after repeated failed boots, A/B partitions, or a boot counter that disables a content channel. The embedded update guide (/domains/appsec/platform-security/embedded-update-security/) covers the device version of this.

The check below compares three things the incident shows can drift apart: the field list a template type declares, the number of inputs the consuming code supplies, and the fields the content and the tests actually exercise. It fails if the counts differ, if any instance or test matches on a field index the code does not supply, and if the tests never use a non-wildcard value for some field. Its tests reproduce the 21-versus-20 case; they ran with Python 3.14. Generate inputs_supplied from the consuming code at build time, not by hand, or the check only compares two descriptions again.

"""Check a content file against the interface of the code that will run it.

The template type declares its fields; the consumer (the code that evaluates
the content) declares how many input values it actually supplies. A template
instance may only match on fields the consumer supplies, and the test suite
must exercise every field with a non-wildcard value at least once.
"""
import json
import sys

WILDCARD = "*"


def check(template_type, consumer, instances, test_instances):
    problems = []
    declared = len(template_type["fields"])
    supplied = consumer["inputs_supplied"]
    if declared != supplied:
        problems.append(
            f"{template_type['name']}: definition declares {declared} fields, "
            f"consumer supplies {supplied}"
        )
    for inst in instances + test_instances:
        for index, criterion in enumerate(inst["criteria"]):
            if criterion != WILDCARD and index >= supplied:
                problems.append(
                    f"{inst['id']}: field {index + 1} is matched but only {supplied} inputs exist"
                )
    tested = {
        i for inst in test_instances for i, c in enumerate(inst["criteria"]) if c != WILDCARD
    }
    for index in range(declared):
        if index not in tested:
            problems.append(f"tests never use a non-wildcard value for field {index + 1}")
    return problems


if __name__ == "__main__":
    with open(sys.argv[1], encoding="utf-8") as f:
        bundle = json.load(f)
    found = check(bundle["template_type"], bundle["consumer"], bundle["instances"], bundle["tests"])
    print("\n".join(found) or "OK")
    sys.exit(1 if found else 0)

Validation upstream reduces bad content; it does not make the interpreter safe. Where data from a file chooses an array index in privileged code, check the index against the array's real length every time and fail closed. The function below is illustrative, not CrowdStrike's code; it compiled cleanly with Clang using -Wall -Wextra -fsanitize=address,undefined, and the test call for the 21st field returns NULL instead of reading past the array. Fuzz the interpreter with sanitizers on as well (/domains/appsec/testing/parser-fuzzing/), and see the C and C++ memory-safety guide (/domains/appsec/platform-security/memory-safety-c-cpp/).

#include <stddef.h>
#include <stdio.h>

struct input { const char *buf; size_t len; };

/* Return input `index`, or NULL when the content asks for a field the caller
   did not supply. Illustrative only: not CrowdStrike's code. */
const struct input *get_input(const struct input *inputs, size_t count, size_t index) {
    if (inputs == NULL || index >= count) return NULL;
    return &inputs[index];
}

int main(void) {
    struct input inputs[20] = {{"a", 1}};
    const struct input *in = get_input(inputs, 20, 20); /* the 21st field */
    puts(in == NULL ? "rejected out-of-range field 21" : "BUG: read past the array");
    return in == NULL ? 0 : 1;
}

Most of these are CrowdStrike's own remediations from the RCA. SSDF 1.1 (NIST SP 800-218, February 2022) has no practice for staged delivery; the PS.4 entries refer to the SSDF 1.2 initial public draft and may change.

• If a file changes what privileged code does, release it like code: canary, rings, bake time, automatic halt and rollback.
• Validate content against the consuming code itself, generated from that code, not against a separate description that can drift.
• Wildcards and defaults in tests hide unused paths. Exercise every field with a real value.
• Keep bounds checks in the interpreter even when upstream validation exists; validation is a filter, not a guarantee.
• Design rollback for the worst case: a host that crashes before it can fetch the fix.
• Give customers control over when updates reach their critical systems, and tell them what changed.

The RCA gives no count of affected devices; the widely cited 8.5 million is Microsoft's estimate from 20 July 2024. Financial losses reported in the press come from third-party estimates and are not part of the primary record. The RCA explains what the Content Validator checked but not why the definitions file and the sensor code disagreed on the input count in the first place, beyond the absence of a compile-time check. It does not say how many hosts recovered by downloading the reverted file before crashing again versus by manual repair. CrowdStrike's planned dates in the RCA (hotfixes by 9 August, validator checks by 19 August 2024) are commitments as of 6 August 2024; this page does not verify their completion.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. CrowdStrike: External Technical Root Cause Analysis, Channel File 291. 6 August 2024.
2. CrowdStrike: Remediation and guidance hub, Channel File 291 incident (includes the preliminary Post Incident Review). PIR section updated 24 July 2024.
3. CrowdStrike: Technical details, Falcon content update for Windows hosts. 20 July 2024.
4. Microsoft: Helping our customers through the CrowdStrike outage. 20 July 2024.
5. NIST SP 800-218 Rev. 1 (initial public draft): SSDF version 1.2, practice PS.4. Initial public draft, December 2025.
6. NIST SP 800-218: Secure Software Development Framework version 1.1. Version 1.1 (February 2022).
7. MITRE CWE-125: Out-of-bounds Read

Reference information

Tags

Memory safetySigning and software updatesIncident responseDesktop and native apps

SDLC phase

DevelopmentReleaseMaintenance

Role

DevelopersPlatform engineers and SREs

Download this guide as Markdown

Continue exploring