Case study: Storm-0558 and the forged Microsoft tokens (2023)

From 15 May 2023, a threat actor Microsoft tracks as Storm-0558 read email in Exchange Online, Microsoft's hosted mail service, without stealing a single password. It presented authentication tokens it had forged and signed with a Microsoft account (MSA) signing key, the kind used for consumer services such as Outlook.com. The Cyber Safety Review Board (CSRB), which reviewed the incident for the US Department of Homeland Security, counted 22 enterprise organizations and 503 related personal accounts, including senior US officials; the actor downloaded about 60,000 emails from State Department alone.

The key had been created in 2016. It should only have been able to sign tokens for consumer Outlook Web Access, and it was meant to be retired in March 2021. Two separate failures turned it into a key to enterprise mailboxes: rotation of consumer keys had stopped, so the key was still valid; and enterprise mail accepted tokens signed by a consumer key, because the code validating them never checked which issuer the key belonged to.

This page follows the CSRB report and Microsoft's own posts, and it marks where they disagree or have been corrected. The board's verdict was blunt: 'this intrusion was preventable and should never have occurred.' How the actor obtained the key is still unknown.

Microsoft runs two identity systems: consumer Microsoft accounts and the enterprise system now called Microsoft Entra ID (formerly Azure AD). Each signs tokens with its own keys, and a relying service should accept a token only if it was signed by a key of the issuer it trusts. In September 2018, responding to customer requests, Microsoft introduced a common OpenID Connect metadata endpoint that listed the active signing keys of both systems. According to MSRC, it provided an API to check signatures but did not update its libraries to validate scope automatically, so a caller had to check for itself which issuer a key belonged to.

When the mail system moved to the common endpoint in 2022, its 'Developers in the mail system incorrectly assumed libraries performed complete validation.' The result, in MSRC's words, was that the mail system 'would accept a request for enterprise email using a security token signed with the consumer key.' The CSRB described the same flaw from the other side: Microsoft 'had not adequately updated the software development kits (SDKs)' to tell the two key sets apart. Microsoft's fix was to release 'enhanced libraries to automate key scope validation in authentication libraries'.

Every team that validates tokens from more than one issuer has this exposure. A JSON Web Key Set (JWKS) is a list of keys, not a statement of what each key may sign. The verifier has to bind each key to the issuer whose own configuration published it and reject a token whose signing key belongs to a different issuer, even when the signature is perfect. ASVS 5.0 requirement V9.1.3 states the rule: key material must come 'from trusted pre-configured sources for the token issuer'. The JWT validation guide (/domains/identity-crypto/authentication/jwt-validation-rollover/) covers key selection and rollover in detail.

The enterprise identity system had automated key rotation; the consumer system never caught up. The CSRB found that Microsoft 'continued to rotate consumer MSA keys infrequently and manually until it stopped the rotation entirely in 2021 following a major cloud outage linked to the manual rotation process.' It also had no 'automated alerting system to notify the appropriate Microsoft teams about the age of active signing keys.' The 2016 key had been scheduled for retirement in March 2021; its removal was delayed while the consumer key systems were hardened.

The board spelled out the counterfactual: 'If Microsoft had not paused manual rotation of keys; if it had completed the migration of its MSA environment to rotate keys automatically; if it had put in place a technical or other control to generate alerts for aging keys, the 2016 MSA key would not have been valid in 2023.' A rotation process that breaks things gets paused; a paused process with no alarm becomes permanent. Automate rotation so it is routine, make a pause an exception with an owner and an end date, and alert on key age independently of the rotation job. The key rotation guide (/domains/identity-crypto/key-management-pki/envelope-encryption-rotation/) and the crypto inventory guide (/domains/identity-crypto/key-management-pki/crypto-inventory-pqc/) cover the inventory this needs.

Microsoft did not detect the intrusion. On 15 June 2023 State Department's security operations center saw anomalies, and the next day a custom alert rule it called 'Big Yellow Taxi' fired on the MailItemsAccessed log, which records access to Exchange Online mailbox items. According to CISA and the FBI, the rule flagged mailbox access by an application ID that did not normally read mail in State's environment. State had that log because it bought the G5 licence that includes Purview Audit (Premium); the CSRB notes that 'The MailItemsAccessed log was not accessible without that premium service.'

Other victims paid for the gap. Microsoft enabled enhanced logging for identified victims without the licence, but 'could not give historical logs to customers unless they already had the premium licenses at the time of the intrusion,' so they could not see what had already happened. CISA and the FBI published the detection approach in AA23-193A on 12 July 2023. On 19 July 2023 Microsoft announced that Audit (Standard) customers would get detailed logs of email access and more than 30 other log types, with default retention raised from 90 to 180 days, rolling out from September 2023.

The CSRB's Recommendation 4 says security-related logging offered only in paid tiers 'should stop', and Recommendation 10 asks cloud providers to log all access to customer business data, including the provider's own, at no extra charge with at least six months' retention. If you run a service, the logs your customers need to investigate you are part of the product, not an upsell. The security logging guide (/domains/runtime/detection/security-event-logging/) covers what such events should contain.

The CSRB concluded that 'Microsoft's security culture was inadequate and requires an overhaul.' Its reasons included the 'cascade of Microsoft's avoidable errors', Microsoft's failure to detect the theft of its 'cryptographic crown jewels' on its own, practices that other cloud providers had and Microsoft lacked, and the handling of its public statements.

That last point matters for any team that publishes incident reports. On 6 September 2023 MSRC wrote that the most probable route for the key was a 2021 crash dump in which a race condition had left key material. The CSRB reports that Microsoft told the board in November 2023 that the post was inaccurate, but did not correct it until 12 March 2024, 'as the Board was concluding its review'. The addendum says Microsoft has not found a crash dump containing the key. Recommendation 16 asks providers to 'promptly correct significant factual inaccuracies'. The product incident response guide (/domains/appsec/vulnerability-response/product-incident-response/) covers updating a published statement.

The board also listed practices other providers used: automated regular key rotation, keys kept in isolated systems such as hardware security modules, stateful token validation, keys scoped narrowly, and tokens bound to a session or operation instead of broad bearer tokens. Those are the architecture behind the controls table below.

The test below uses PyJWT 2.15 and cryptography 50, and it ran with Python 3.14. A verifier for the enterprise issuer loads only that issuer's keys, so a token signed with a consumer key fails even though it claims the enterprise issuer and its signature is valid. The last test shows the flaw: a verifier that looks the kid up in one combined key set accepts the forgery. Add the equivalent negative test to every service that validates tokens from more than one issuer.

# Bind every signing key to the one issuer that publishes it (PyJWT 2.x).
# A verifier that looks keys up in one combined set accepts a token signed by
# any issuer's key; this test shows the difference.
import time
import unittest

import jwt
from cryptography.hazmat.primitives.asymmetric import rsa

ENTERPRISE = "https://login.example.com/enterprise/v2.0"
CONSUMER = "https://login.example.com/consumer/v2.0"
AUDIENCE = "api://mail"


def new_key():
    return rsa.generate_private_key(public_exponent=65537, key_size=2048)


def verify(token, keys_by_issuer, trusted_issuer):
    """Verify with only the keys the trusted issuer publishes."""
    kid = jwt.get_unverified_header(token).get("kid")
    key = keys_by_issuer[trusted_issuer].get(kid)
    if key is None:
        raise jwt.InvalidTokenError(f"key {kid!r} is not published by {trusted_issuer}")
    return jwt.decode(
        token,
        key,
        algorithms=["RS256"],
        audience=AUDIENCE,
        issuer=trusted_issuer,
        options={"require": ["exp", "iss", "aud"]},
    )


def verify_combined(token, combined_keys, trusted_issuer):
    """The flaw: any key from a combined set is accepted for any issuer."""
    kid = jwt.get_unverified_header(token).get("kid")
    return jwt.decode(
        token, combined_keys[kid], algorithms=["RS256"], audience=AUDIENCE, issuer=trusted_issuer
    )


class IssuerKeyBinding(unittest.TestCase):
    def setUp(self):
        self.ent_key, self.con_key = new_key(), new_key()
        self.keys_by_issuer = {
            ENTERPRISE: {"ent-1": self.ent_key.public_key()},
            CONSUMER: {"con-2016": self.con_key.public_key()},
        }
        self.combined = {"ent-1": self.ent_key.public_key(), "con-2016": self.con_key.public_key()}
        self.claims = {"iss": ENTERPRISE, "aud": AUDIENCE, "sub": "user", "exp": int(time.time()) + 300}

    def sign(self, key, kid):
        return jwt.encode(self.claims, key, algorithm="RS256", headers={"kid": kid})

    def test_enterprise_token_is_accepted(self):
        token = self.sign(self.ent_key, "ent-1")
        self.assertEqual(verify(token, self.keys_by_issuer, ENTERPRISE)["sub"], "user")

    def test_consumer_key_cannot_sign_for_the_enterprise_issuer(self):
        forged = self.sign(self.con_key, "con-2016")  # claims the enterprise issuer
        with self.assertRaises(jwt.InvalidTokenError):
            verify(forged, self.keys_by_issuer, ENTERPRISE)

    def test_a_combined_key_set_accepts_the_forgery(self):
        forged = self.sign(self.con_key, "con-2016")
        self.assertEqual(verify_combined(forged, self.combined, ENTERPRISE)["sub"], "user")


if __name__ == "__main__":
    unittest.main()

Rotation can stop silently; an age check run on its own schedule catches that. The script reads a key inventory exported from your key management system or identity provider and exits non-zero when an active key is older than your policy allows. Run against the sample inventory with a 90-day limit, it reports the 2016 key as 3,841 days old. Wire it to an alert, not a dashboard.

"""Flag signing keys that are still trusted past the rotation limit.

Input: a JSON inventory of signing keys, for example exported from your KMS or
identity provider: [{"kid": ..., "issuer": ..., "created": "YYYY-MM-DD",
"retired": "YYYY-MM-DD" or null}]. Exit status 1 if any active key is too old.
"""
import json
import sys
from datetime import date


def stale_keys(inventory, max_age_days, today):
    for key in inventory:
        if key.get("retired"):
            continue
        age = (today - date.fromisoformat(key["created"])).days
        if age > max_age_days:
            yield key["issuer"], key["kid"], age


if __name__ == "__main__":
    path, max_age = sys.argv[1], int(sys.argv[2]) if len(sys.argv) > 2 else 90
    with open(path, encoding="utf-8") as f:
        found = list(stale_keys(json.load(f), max_age, date.today()))
    for issuer, kid, age in found:
        print(f"STALE {issuer} {kid}: {age} days old, limit {max_age}")
    sys.exit(1 if found else 0)

The controls split between the token issuer and operator (here Microsoft) and the services and customers that rely on it. The SSDF column names NIST SP 800-218 version 1.1 practices; the ASVS column names ASVS 5.0 requirements.

• A valid signature is not enough. Bind each key to its issuer and test that a key from another issuer is rejected; do not assume a library does this unless you have read its code or tested it.
• Rotation that hurts gets turned off. Automate it, alert on key age separately, and treat a pause as a dated exception someone owns.
• The logs your customers need to investigate you belong in the base product. Detection here depended on one customer paying for a premium tier and writing its own rule.
• Keep signing keys where crash dumps, debuggers and engineers' tools cannot reach them, and scope each key as narrowly as the design allows.
• Publish what you know and what you do not, and correct a wrong root cause as soon as you learn it is wrong.

The public record does not establish how or when Storm-0558 obtained the 2016 key. Microsoft's September 2023 crash-dump explanation was corrected in March 2024, and the CSRB says Microsoft did not have the logs to determine how the key was stolen; Microsoft's leading hypothesis, that operational errors let key material leave the signing environment and a compromised engineering account later reached it, is a hypothesis. Victim counts changed between Microsoft's July 2023 post (about 25 organizations) and the CSRB report (22 organizations and 503 personal accounts). The link to the People's Republic of China is the assessment of Microsoft and the CSRB, not something the technical findings above demonstrate, and the CSRB says the possibility that the actor accessed other keys and sensitive data as well remains open.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. Cyber Safety Review Board: Review of the Summer 2023 Microsoft Exchange Online Intrusion. Dated 20 March 2024; released 2 April 2024.
2. CISA: Cyber Safety Review Board report on the Summer 2023 Microsoft Exchange Online incident (landing page)
3. MSRC: Results of major technical investigations for Storm-0558 key acquisition. 6 September 2023, addendum 12 March 2024.
4. Microsoft Threat Intelligence: Analysis of Storm-0558 techniques for unauthorized email access. 14 July 2023.
5. MSRC: Microsoft mitigates China-based threat actor Storm-0558 targeting of customer email. 11 July 2023.
6. CISA and FBI AA23-193A: Enhanced monitoring to detect APT activity targeting Outlook Online. 12 July 2023.
7. Microsoft Security blog: Expanding cloud logging to give customers deeper security visibility. 19 July 2023.
8. OWASP ASVS 5.0.0: V9 Self-contained Tokens. Version 5.0.0.

Reference information

Tags

JSON Web Tokens (JWT)Key managementLogging and detectionIncident responseAppSec programs

SDLC phase

DevelopmentMaintenanceSecurity program

Role

DevelopersPlatform engineers and SREs

ASVS 5.0.0 requirements

v5.0.0-9.1.3 · v5.0.0-11.1.1 · v5.0.0-13.3.4 · v5.0.0-16.3.2

CAPEC attack patterns

• CAPEC-196 at MITRE

Download this guide as Markdown

Continue exploring