Case study: the SolarWinds Orion build compromise (SUNBURST)

SolarWinds Orion is network and infrastructure monitoring software that customers install on their own servers, usually with broad administrative access to the systems it watches. Between March and June 2020, SolarWinds shipped Orion updates that contained a backdoor, later named SUNBURST, inside the legitimately signed SolarWinds.Orion.Core.BusinessLayer.dll. Customers installed them through the normal update process.

The source repository did not contain the backdoor and the signing process worked as designed. An implant on SolarWinds' build server, which CrowdStrike named SUNSPOT, watched for the compiler building Orion, replaced one source file for the length of the build and put the original back afterward. Everything downstream of compilation, including signing, saw a binary that looked like an ordinary release.

That is why this incident reshaped supply-chain security. It showed that a signature proves who released an artifact, not that the artifact was built from the reviewed source, and it led directly to build integrity becoming a requirement rather than an aspiration.

CrowdStrike's analysis of the implant recovered from a SolarWinds build server describes a small, careful tool. It checked running processes every second for MsBuild.exe, read each build's command line to see whether the Orion solution was being compiled, and only then replaced InventoryManager.cs with a backdoored version. Before swapping, it compared MD5 hashes of the original and replacement files, so a source change by developers would not cause a failed or suspicious build. When MsBuild.exe exited, it restored the original file and deleted its temporary copies.

The result was a release whose binary did not match its source, built on infrastructure that SolarWinds controlled and signed with SolarWinds' key. Any control that looked only at the repository, the code review record or the signature would pass it. The controls that would have noticed are the ones that compare the artifact with an independent derivation of it: a second, isolated build from the same commit, or provenance from a builder the attacker could not reach.

Mandiant's report describes a backdoor built to blend in. After installation it stayed dormant for up to two weeks, then resolved subdomains of avsvmcloud[.]com generated from the victim's identity and domain, disguised its traffic as Orion Improvement Program telemetry and stored data in plugin configuration files. Based on the DNS responses, the operators decided which victims to pursue; in those, they deployed second-stage tools such as the memory-only TEARDROP dropper, which loaded Cobalt Strike BEACON.

In the networks the operators chose, CISA's alert AA20-352A describes escalation to Active Directory privileges, theft of the SAML token-signing certificate and forging of valid tokens to reach cloud services such as Microsoft 365. CISA also noted evidence of initial access vectors other than Orion, so an organization without Orion was not automatically out of scope.

SolarWinds estimated that fewer than 18,000 customers installed an affected version. Installing the update gave the attackers a foothold; it did not mean they used it. That difference between carrying the backdoor and being an actual target is the main scoping question in the response.

Not by any supply-chain control. FireEye was investigating its own breach, disclosed on 2020-12-08, and traced the intrusion to its Orion server. The analysis it published on 2020-12-13 gave defenders the DLL name, the C2 domain and network signatures, and SolarWinds learned of the attack the day before.

The backdoored releases had been on customer networks for up to nine months. Monitoring products are given network-wide credentials and are usually allowed to talk to the internet for updates and telemetry, which is exactly what made the traffic look normal. A detection story that depended on one targeted security company noticing an anomaly is a near miss, not a control.

The affected releases are long superseded, but the check below shows the scoping step for any vendor backdoor: find the component on disk, compare its exact version with the advisory, and record that the signature is valid, because in this incident the signature was valid. A DNS log search for the published domain covers the other half of the question: whether an affected server ever called out.

# Find the Orion business-layer DLL and compare its file version with the
# versions CISA AA20-352A lists as carrying SUNBURST.
$affected = @(
  '2019.4.5200.9083',   # 2019.4 HF5
  '2020.2.100.12219',   # 2020.2 RC1
  '2020.2.5200.12394',  # 2020.2 RC2
  '2020.2.5300.12432'   # 2020.2 and 2020.2 HF1
)
Get-ChildItem -Path "$env:ProgramFiles*\SolarWinds" -Recurse `
    -Filter 'SolarWinds.Orion.Core.BusinessLayer.dll' -ErrorAction SilentlyContinue |
  ForEach-Object {
    $version = $_.VersionInfo.FileVersion
    [pscustomobject]@{
      Path        = $_.FullName
      FileVersion = $version
      Affected    = $affected -contains $version
      Signer      = (Get-AuthenticodeSignature $_.FullName).SignerCertificate.Subject
    }
  }
# A valid SolarWinds signature on an affected version is expected: the
# backdoored DLL was built and signed by SolarWinds' own pipeline.

A hit from an Orion server inside the exposure window means the backdoor was active, not that the operators chose that network. The DNS responses and any HTTPS connections to the CNAME targets that followed are what separate a dormant install from a second-stage intrusion, which is why investigators asked for full DNS and proxy logs from March 2020 onward.

# SUNBURST's first stage resolved subdomains of avsvmcloud[.]com.
# Search resolver logs for the whole exposure window, March 2020 onward.
grep -hi 'avsvmcloud\.com' dns-*.log | sort -u > avsvmcloud-hits.txt
wc -l < avsvmcloud-hits.txt

# Per log file, how many lookups? Then check which clients asked: only Orion
# servers should appear, and a hit is followed up with the CNAME answers and
# proxy logs for the same hosts.
grep -ci 'avsvmcloud\.com' dns-*.log

The root cause was a build environment that an intruder could reach and persist in, producing artifacts nobody independently checked. The SLSA Build track and NIST's Secure Software Development Framework (SSDF, SP 800-218), which NIST updated in 2022 in response to Executive Order 14028, describe the controls. The table maps them to the guides on this site.

• Treat the build system as production: isolate it, rebuild it from a known image for each release, and restrict who and what can reach it.
• Verify artifacts, not just signatures. An independent rebuild or provenance from a hardened builder answers 'was this built from the reviewed source?'; a signature does not.
• Monitor build infrastructure as closely as customer-facing systems, including unexpected processes that read compiler command lines or touch source trees during builds.
• As a customer, give management software only the access and egress it needs, and keep the logs that would let you scope a vendor compromise months later.
• Separate facts from attribution: the build mechanism and affected versions are documented by the vendor and responders; responsibility comes from a government statement.

The public record does not establish how the attackers first entered SolarWinds' network; the company's investigation reported activity from September 2019 but did not publish a confirmed initial access vector. The number of organizations that received second-stage intrusions is not public in full; government statements and press reporting name some victims, and SolarWinds said the number actually exploited was substantially lower than the number that installed the update. Attribution to the SVR rests on the US and UK government statements rather than on evidence published with the technical reports.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. SolarWinds: Security advisory (SUNBURST and SUNSPOT). Recent as of 6 April 2021.
2. SolarWinds Form 8-K with 'New findings from our investigation of SUNBURST' (2021-01-11)
3. SolarWinds Form 8-K (2020-12-14): customers notified and affected estimate
4. CrowdStrike: SUNSPOT, an implant in the build process (2021-01-11)
5. Mandiant: Highly evasive attacker leverages SolarWinds supply chain to compromise multiple global victims with SUNBURST backdoor (2020-12-13)
6. CISA Emergency Directive 21-01: Mitigate SolarWinds Orion code compromise (closed). Issued 13 December 2020.
7. CISA AA20-352A: APT compromise of government agencies, critical infrastructure and private sector organizations. 17 December 2020, revised 15 April 2021.
8. White House fact sheet: Imposing costs for harmful foreign activities by the Russian government (2021-04-15)
9. NIST SP 800-218: Secure Software Development Framework. Version 1.1 (February 2022).

Reference information

Tags

Dependency managementSupply chain attacksBuild integrity and provenanceSigning and software updates

SDLC phase

ReleaseMaintenance

Role

AppSec engineersPlatform engineers and SREs

CAPEC attack patterns

• CAPEC-444 at MITRE
• CAPEC-511 at MITRE
• CAPEC-206 at MITRE

Download this guide as Markdown

Continue exploring