--- key: "case-study-solarwinds" title: "Case study: the SolarWinds Orion build compromise (SUNBURST)" url: "https://softwaresecurity.com/domains/supply-chain/builds/case-study-solarwinds/" api: "https://softwaresecurity.com/api/v1/articles/case-study-solarwinds.json" domain: "supply-chain" topic: "Software supply chain" type: "Case study" identifiers: ["CWE-506","CWE-912","A03:2025","A08:2025","CAPEC-444","CAPEC-511","CAPEC-206"] tags: ["Dependency management","Supply chain attacks","Build integrity and provenance","Signing and software updates"] stages: ["release","respond"] roles: ["appsec-engineer","platform-sre"] license: "CC-BY-4.0" license_url: "https://creativecommons.org/licenses/by/4.0/" code_license: "MIT" --- # Case study: the SolarWinds Orion build compromise (SUNBURST) How an implant on SolarWinds' build server inserted the SUNBURST backdoor into signed Orion updates in 2020, how it was found, why code signing and review could not catch it, and which build controls would have. Canonical URL: https://softwaresecurity.com/domains/supply-chain/builds/case-study-solarwinds/ ## In short In 2020 an implant called SUNSPOT on SolarWinds' build server swapped one source file while Orion was being compiled, so the official, signed Orion updates released from March to June 2020 carried the SUNBURST backdoor. SolarWinds estimated that fewer than 18,000 customers installed them; the attackers chose a much smaller number for follow-on intrusion. FireEye found it in December 2020 while investigating its own breach. Code signing and source review could not catch it; isolated, reproducible builds with provenance and least-privilege, egress-restricted deployments of management tools could have. - The repository was clean and the release was signed; the change happened on the build server, during compilation, and was undone afterward. - Affected: Orion Platform 2019.4 HF5, 2020.2 (no hotfix) and 2020.2 HF1, released between March and June 2020. - No CVE covers the SUNBURST backdoor itself; CVE-2020-10148 is a separate Orion API flaw used by the SUPERNOVA web shell. - SolarWinds estimated fewer than 18,000 customers installed an affected version; follow-on intrusions, including forged SAML tokens, targeted far fewer. - Build isolation, reproducible builds and provenance address the root cause; egress restriction and least privilege for monitoring tools limit the blast radius. - The US government attributed the campaign to Russia's SVR in April 2021; attribution is a government statement, not something the technical evidence on this page proves. ## Incident at a glance - Date: 2020-02-20 to 2020-12-13 - Category: Supply-chain compromise - Organizations: SolarWinds (Orion Platform); FireEye (now Mandiant) - Impact: An implant on SolarWinds' build server inserted the SUNBURST backdoor into signed Orion updates released from March to June 2020. SolarWinds estimated that fewer than 18,000 customers installed one; the attackers chose far fewer for follow-on intrusions, including forged SAML tokens. - How it was found: FireEye, investigating its own breach, traced the intrusion to its Orion server and published the SUNBURST analysis with indicators on 2020-12-13. ### Timeline 1. 2019-09: Earliest suspicious activity on SolarWinds' internal systems. 2. 2019-10: An Orion release carries test modifications, without the backdoor. 3. 2020-02-20: SUNSPOT begins inserting SUNBURST into Orion builds. 4. 2020-03-24: A trojanized Orion update that Mandiant later analyzed is signed. 5. 2020-06: The attackers remove the injection code from SolarWinds' environment. 6. 2020-12-08: FireEye discloses that it was breached and its red-team tools stolen. 7. 2020-12-13: Mandiant publishes the SUNBURST analysis; CISA issues Emergency Directive 21-01. 8. 2020-12-14: SolarWinds files an 8-K: about 33,000 customers notified, fewer than 18,000 may have installed an affected version. 9. 2021-01-11: CrowdStrike publishes its SUNSPOT analysis and SolarWinds its investigation findings. 10. 2021-04-15: The US government attributes the campaign to Russia's SVR. ## Recommended defaults | Setting | Value | Note | Source | | --- | --- | --- | --- | | Backdoored releases | Orion Platform 2019.4 HF5, 2020.2 RC1, RC2, 2020.2 and 2020.2 HF1 | Released March to June 2020. File versions are in CISA AA20-352A, Appendix A. | SolarWinds security advisory; CISA AA20-352A | | Malicious component | `SolarWinds.Orion.Core.BusinessLayer.dll` (SUNBURST) | Signed by SolarWinds; the first trojanized update Mandiant analyzed was signed on 2020-03-24. | Mandiant SUNBURST report | | Build implant | SUNSPOT, build timestamp 2020-02-20 | Replaced `InventoryManager.cs` while MsBuild compiled Orion, then restored it. | CrowdStrike SUNSPOT analysis | | CVE | None for SUNBURST | CVE-2020-10148 (Orion API authentication bypass, in KEV since 2021-11-03) relates to SUPERNOVA, a separate web shell. | CISA ED 21-01; CISA KEV | | Customers who installed an affected version | Fewer than 18,000 (SolarWinds estimate) | About 33,000 active Orion maintenance customers were notified. | SolarWinds Form 8-K, 2020-12-14 | | Response | Disconnect, image and rebuild on a release without SUNBURST | CISA ED 21-01, since closed, required federal agencies to disconnect or power down affected servers; SolarWinds' advisory directs customers to 2020.2.5 or later. | CISA ED 21-01; SolarWinds advisory | ## Verify it 1. Inventory every SolarWinds Orion server and check the business-layer DLL's file version against the AA20-352A list, keeping the result even if the server was rebuilt. 2. Confirm your build servers are ephemeral, isolated from developer workstations and the corporate network, and rebuilt from a known image for each release. 3. Rebuild one recent release on an independent builder from the tagged source and compare the artifacts byte for byte. 4. Check that every release ships provenance naming the source commit and builder, and that your release gate verifies it. 5. Review management and monitoring servers for internet egress and domain-admin-equivalent privileges, and remove both where the tool does not need them. ## What you will learn - Explain how SUNSPOT altered Orion during compilation without changing the repository, and why signing could not detect it. - Separate installing an affected version from being a second-stage target when scoping a vendor backdoor. - Map the build controls that would have caught it to SSDF practices and SLSA requirements. - Separate documented facts from attribution in the public record. Audience: Release, build and platform engineers who own build systems, and AppSec and IT teams who assess vendor software with privileged network access Before you start: - Know how a CI or build server turns source into a signed release. - Understand what a code-signing signature proves about an artifact. - For the code: Windows PowerShell for the version check, and a shell for the log search. Time: About 28 minutes: 8 to read and 20 for the exercise ## A clean repository, a signed release, and a backdoor in between SolarWinds Orion is network and infrastructure monitoring software that customers install on their own servers, usually with broad administrative access to the systems it watches. Between March and June 2020, SolarWinds shipped Orion updates that contained a backdoor, later named SUNBURST, inside the legitimately signed `SolarWinds.Orion.Core.BusinessLayer.dll`. Customers installed them through the normal update process. The source repository did not contain the backdoor and the signing process worked as designed. An implant on SolarWinds' build server, which CrowdStrike named SUNSPOT, watched for the compiler building Orion, replaced one source file for the length of the build and put the original back afterward. Everything downstream of compilation, including signing, saw a binary that looked like an ordinary release. That is why this incident reshaped supply-chain security. It showed that a signature proves who released an artifact, not that the artifact was built from the reviewed source, and it led directly to build integrity becoming a requirement rather than an aspiration. ## Timeline | When | Event | Source | | --- | --- | --- | | September 2019 | Earliest suspicious activity on SolarWinds' internal systems. | SolarWinds, 'New findings from our investigation of SUNBURST' (2021-01-11) | | October 2019 | An Orion release contains modifications that test whether the attackers can insert code into builds undetected. | SolarWinds, 2021-01-11 | | 2020-02-20 | SUNSPOT begins inserting SUNBURST into Orion builds (its binary carries this build timestamp). | SolarWinds, 2021-01-11; CrowdStrike | | March-June 2020 | Trojanized Orion releases (2019.4 HF5, 2020.2 RC1, RC2, 2020.2, 2020.2 HF1) are signed and published; Mandiant analyzed one signed on 2020-03-24. | SolarWinds advisory; Mandiant | | June 2020 | The attackers remove the SUNBURST injection code from SolarWinds' environment. | SolarWinds, 2021-01-11 | | 2020-12-08 | FireEye discloses that it was breached and its red-team tools were stolen. | FireEye | | 2020-12-12 | SolarWinds is informed of the attack. | SolarWinds, 2021-01-11 | | 2020-12-13 | Mandiant (then FireEye) publishes the SUNBURST analysis; CISA issues Emergency Directive 21-01. | Mandiant; CISA ED 21-01 | | 2020-12-14 | SolarWinds files an 8-K: about 33,000 customers notified, fewer than 18,000 may have installed an affected version. | SolarWinds Form 8-K | | 2020-12-17 | CISA alert AA20-352A notes initial access vectors other than Orion and SAML token forgery. | CISA AA20-352A | | 2021-01-11 | CrowdStrike publishes the SUNSPOT analysis; SolarWinds publishes its investigation findings. | CrowdStrike; SolarWinds | | 2021-04-15 | The US government names Russia's Foreign Intelligence Service (SVR) as responsible. | White House fact sheet | ## How did SUNSPOT change the build without changing the source? CrowdStrike's analysis of the implant recovered from a SolarWinds build server describes a small, careful tool. It checked running processes every second for `MsBuild.exe`, read each build's command line to see whether the Orion solution was being compiled, and only then replaced `InventoryManager.cs` with a backdoored version. Before swapping, it compared MD5 hashes of the original and replacement files, so a source change by developers would not cause a failed or suspicious build. When `MsBuild.exe` exited, it restored the original file and deleted its temporary copies. The result was a release whose binary did not match its source, built on infrastructure that SolarWinds controlled and signed with SolarWinds' key. Any control that looked only at the repository, the code review record or the signature would pass it. The controls that would have noticed are the ones that compare the artifact with an independent derivation of it: a second, isolated build from the same commit, or provenance from a builder the attacker could not reach. ## What SUNBURST did on customer networks Mandiant's report describes a backdoor built to blend in. After installation it stayed dormant for up to two weeks, then resolved subdomains of `avsvmcloud[.]com` generated from the victim's identity and domain, disguised its traffic as Orion Improvement Program telemetry and stored data in plugin configuration files. Based on the DNS responses, the operators decided which victims to pursue; in those, they deployed second-stage tools such as the memory-only TEARDROP dropper, which loaded Cobalt Strike BEACON. In the networks the operators chose, CISA's alert AA20-352A describes escalation to Active Directory privileges, theft of the SAML token-signing certificate and forging of valid tokens to reach cloud services such as Microsoft 365. CISA also noted evidence of initial access vectors other than Orion, so an organization without Orion was not automatically out of scope. SolarWinds estimated that fewer than 18,000 customers installed an affected version. Installing the update gave the attackers a foothold; it did not mean they used it. That difference between carrying the backdoor and being an actual target is the main scoping question in the response. ## How was it found? Not by any supply-chain control. FireEye was investigating its own breach, disclosed on 2020-12-08, and traced the intrusion to its Orion server. The analysis it published on 2020-12-13 gave defenders the DLL name, the C2 domain and network signatures, and SolarWinds learned of the attack the day before. The backdoored releases had been on customer networks for up to nine months. Monitoring products are given network-wide credentials and are usually allowed to talk to the internet for updates and telemetry, which is exactly what made the traffic look normal. A detection story that depended on one targeted security company noticing an anomaly is a near miss, not a control. ## Check a server for an affected version The affected releases are long superseded, but the check below shows the scoping step for any vendor backdoor: find the component on disk, compare its exact version with the advisory, and record that the signature is valid, because in this incident the signature was valid. A DNS log search for the published domain covers the other half of the question: whether an affected server ever called out. ```powershell # Find the Orion business-layer DLL and compare its file version with the # versions CISA AA20-352A lists as carrying SUNBURST. $affected = @( '2019.4.5200.9083', # 2019.4 HF5 '2020.2.100.12219', # 2020.2 RC1 '2020.2.5200.12394', # 2020.2 RC2 '2020.2.5300.12432' # 2020.2 and 2020.2 HF1 ) Get-ChildItem -Path "$env:ProgramFiles*\SolarWinds" -Recurse ` -Filter 'SolarWinds.Orion.Core.BusinessLayer.dll' -ErrorAction SilentlyContinue | ForEach-Object { $version = $_.VersionInfo.FileVersion [pscustomobject]@{ Path = $_.FullName FileVersion = $version Affected = $affected -contains $version Signer = (Get-AuthenticodeSignature $_.FullName).SignerCertificate.Subject } } # A valid SolarWinds signature on an affected version is expected: the # backdoored DLL was built and signed by SolarWinds' own pipeline. ``` ## Search resolver logs for the first-stage domain A hit from an Orion server inside the exposure window means the backdoor was active, not that the operators chose that network. The DNS responses and any HTTPS connections to the CNAME targets that followed are what separate a dormant install from a second-stage intrusion, which is why investigators asked for full DNS and proxy logs from March 2020 onward. ```bash # SUNBURST's first stage resolved subdomains of avsvmcloud[.]com. # Search resolver logs for the whole exposure window, March 2020 onward. grep -hi 'avsvmcloud\.com' dns-*.log | sort -u > avsvmcloud-hits.txt wc -l < avsvmcloud-hits.txt # Per log file, how many lookups? Then check which clients asked: only Orion # servers should appear, and a hit is followed up with the CNAME answers and # proxy logs for the same hosts. grep -ci 'avsvmcloud\.com' dns-*.log ``` ## Which controls would have stopped or limited it? The root cause was a build environment that an intruder could reach and persist in, producing artifacts nobody independently checked. The SLSA Build track and NIST's Secure Software Development Framework (SSDF, SP 800-218), which NIST updated in 2022 in response to Executive Order 14028, describe the controls. The table maps them to the guides on this site. | Control | SSDF practice or SLSA requirement | Would it have helped? | Guide | | --- | --- | --- | --- | | Ephemeral, isolated build environments rebuilt for each release | SSDF PO.5 (secure development environments); SLSA Build L3 isolated builds | Yes: SUNSPOT relied on persisting on a long-lived build server. | /domains/supply-chain/ci-cd/build-integrity/ | | Reproducible builds checked by an independent builder | SSDF PS.2, PW.6; SLSA verification | Yes: a second build from the same commit would not contain SUNBURST. | /domains/supply-chain/builds/reproducibility-and-hermeticity/ | | Signed provenance from a builder the attacker cannot reach, verified before release | SLSA Build L2-L3 provenance | Partly: it ties the artifact to a builder and commit, but a compromised builder can sign false provenance. | /domains/supply-chain/provenance/read-and-verify-slsa/ | | Signing keys in an HSM, used only by the isolated release step | SSDF PS.2 | Partly: the attackers did not need the key, because they changed what was signed. | /domains/supply-chain/provenance/artifact-signature-policy/ | | Least privilege and egress allow-lists for monitoring servers | Customer-side deployment hardening | Limits impact: no internet egress means no C2; less privilege means less to steal. | /frameworks/supplier-software-assurance/ | | Detection on build systems and on vendor tools' network behavior | SSDF PO.5, RV.1 | Could have shortened the nine-month window. | /domains/runtime/detection/runtime-detection-response/ | ## Lessons that still apply - Treat the build system as production: isolate it, rebuild it from a known image for each release, and restrict who and what can reach it. - Verify artifacts, not just signatures. An independent rebuild or provenance from a hardened builder answers 'was this built from the reviewed source?'; a signature does not. - Monitor build infrastructure as closely as customer-facing systems, including unexpected processes that read compiler command lines or touch source trees during builds. - As a customer, give management software only the access and egress it needs, and keep the logs that would let you scope a vendor compromise months later. - Separate facts from attribution: the build mechanism and affected versions are documented by the vendor and responders; responsibility comes from a government statement. ## What the public evidence does not establish The public record does not establish how the attackers first entered SolarWinds' network; the company's investigation reported activity from September 2019 but did not publish a confirmed initial access vector. The number of organizations that received second-stage intrusions is not public in full; government statements and press reporting name some victims, and SolarWinds said the number actually exploited was substantially lower than the number that installed the update. Attribution to the SVR rests on the US and UK government statements rather than on evidence published with the technical reports. ## Guided practice It is December 2020 and you are scoping SUNBURST across a company's estate. For each item, decide whether it carried the backdoor, whether there is evidence it was used, and what to do. Treat the facts as true. ### Supplied material ```text S1: Orion server, business-layer DLL file version 2019.4.5200.9083, installed 2020-04-02; outbound DNS and HTTPS allowed; resolver logs show avsvmcloud[.]com lookups from 2020-04-16, followed by CNAME responses and HTTPS connections to the CNAME target. S2: Orion server, file version 2020.2.5300.12432 (2020.2 HF1); outbound HTTPS blocked by policy; resolver logs show avsvmcloud[.]com lookups but no CNAME responses and no outbound HTTPS. S3: Orion server on 2019.4 HF4, never updated in 2020. S4: Domain controllers and the AD FS server used by Microsoft 365 sign-in; S1's service account has domain admin rights. S5: Your own company's build server, a long-lived Windows VM that compiles and signs your product, reachable over RDP from the IT admin network. ``` ### 1. Classify S1 Decide S1's status and first actions. Worked answer: Affected version (2019.4 HF5) and evidence of second-stage activity: CNAME responses and HTTPS connections to the C2 target. Treat it as an active intrusion: disconnect and image it, block egress, and start incident response that assumes the attacker used its domain admin service account. ### 2. Classify S2 Decide S2's status. Worked answer: Affected version (2020.2 HF1) and the backdoor was active, because it resolved the first-stage domain, but there is no evidence of second-stage contact, which blocked HTTPS would also have prevented. Disconnect, image and rebuild it on a fixed release, and keep the DNS evidence; do not declare it clean on DNS alone. ### 3. Clear S3 Decide whether S3 is affected. Worked answer: Not affected: 2019.4 HF4 predates the backdoored releases. Patch it normally and note that the October 2019 release contained only test modifications, not SUNBURST. ### 4. Scope S4 What does S1 mean for the identity systems? Worked answer: Because S1 had a domain admin service account and second-stage activity, assume the attacker could reach AD and the AD FS token-signing certificate. Look for new federation trusts and anomalous token use as AA20-352A describes, rotate the token-signing certificate and privileged credentials once the attacker is evicted, and review cloud sign-in logs for forged tokens. ### 5. Fix S5 Name three changes that would stop a SUNSPOT-style implant on your own build. Worked answer: Replace the long-lived VM with ephemeral builders created from a known image for each release and isolated from admin networks; reproduce each release on an independent builder and compare hashes before signing; and generate signed provenance from a build platform the build steps cannot tamper with, verified at release. Keep the signing key in an HSM used only by the release step. Deliverable: A scoping table for S1 to S4 (carried the backdoor, evidence of use, action) and a one-paragraph build-hardening plan for S5. Scope: The fleet is invented and the evidence is simplified; real scoping used full DNS, proxy and identity logs and vendor indicator lists that changed as the investigation progressed. A DNS hit alone does not tell you what the operators did next. Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security. ## Sources - [SolarWinds: Security advisory (SUNBURST and SUNSPOT)](https://www.solarwinds.com/sa-overview/securityadvisory). Recent as of 6 April 2021. - [SolarWinds Form 8-K with 'New findings from our investigation of SUNBURST' (2021-01-11)](https://www.sec.gov/Archives/edgar/data/1739942/000173994221000015/swi-20210111.htm) - [SolarWinds Form 8-K (2020-12-14): customers notified and affected estimate](https://www.sec.gov/Archives/edgar/data/1739942/000162828020017451/swi-20201214.htm) - [CrowdStrike: SUNSPOT, an implant in the build process (2021-01-11)](https://www.crowdstrike.com/en-us/blog/sunspot-malware-technical-analysis/) - [Mandiant: Highly evasive attacker leverages SolarWinds supply chain to compromise multiple global victims with SUNBURST backdoor (2020-12-13)](https://cloud.google.com/blog/topics/threat-intelligence/evasive-attacker-leverages-solarwinds-supply-chain-compromises-with-sunburst-backdoor) - [CISA Emergency Directive 21-01: Mitigate SolarWinds Orion code compromise (closed)](https://www.cisa.gov/news-events/directives/ed-21-01-mitigate-solarwinds-orion-code-compromise-closed). Issued 13 December 2020. - [CISA AA20-352A: APT compromise of government agencies, critical infrastructure and private sector organizations](https://www.cisa.gov/news-events/cybersecurity-advisories/aa20-352a). 17 December 2020, revised 15 April 2021. - [White House fact sheet: Imposing costs for harmful foreign activities by the Russian government (2021-04-15)](https://bidenwhitehouse.archives.gov/briefing-room/statements-releases/2021/04/15/fact-sheet-imposing-costs-for-harmful-foreign-activities-by-the-russian-government/) - [NIST SP 800-218: Secure Software Development Framework](https://csrc.nist.gov/pubs/sp/800/218/final). Version 1.1 (February 2022).