---
key: "case-study-certificate-expiry"
title: "Case study: certificate expiry outages at Ericsson (2018) and Microsoft Teams (2020), and Equifax's blind spot"
url: "https://softwaresecurity.com/domains/identity-crypto/key-management-pki/case-study-certificate-expiry/"
api: "https://softwaresecurity.com/api/v1/articles/case-study-certificate-expiry.json"
domain: "identity-crypto"
topic: "Identity \u0026 cryptography"
type: "Case study"
identifiers: ["CWE-298","CWE-324","CWE-295","CWE-778","A02:2025","A04:2025","A09:2025","V12.1.1","V12.2.2","V13.3.4"]
tags: ["PKI and certificates","TLS","Logging and detection","Incident response"]
stages: ["operate","respond"]
roles: ["developer","appsec-engineer","platform-sre"]
license: "CC-BY-4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
code_license: "MIT"
---

# Case study: certificate expiry outages at Ericsson (2018) and Microsoft Teams (2020), and Equifax's blind spot

How an expired certificate inside Ericsson's core-network software cut mobile data for millions in about 11 countries, how an expired authentication certificate took Microsoft Teams offline, how an expired inspection certificate left Equifax blind for 19 months, through its 2017 breach, and the inventory, automation and testing that prevent each.

Canonical URL: https://softwaresecurity.com/domains/identity-crypto/key-management-pki/case-study-certificate-expiry/

## In short

On 6 December 2018 a certificate inside two versions of Ericsson's SGSN-MME core-network software expired, cutting mobile data for O2 in the UK, SoftBank in Japan and operators in about 11 countries until the software was rolled back. On 3 February 2020 Microsoft Teams went down because an authentication certificate had expired. At Equifax, an expired certificate on a traffic-inspection device left monitoring blind for 19 months, through the 2017 breach. Inventory every certificate, including those inside products, automate renewal, alert well before expiry and test with the clock moved forward.

- Ericsson's certificate was built into software that had run for about nine months; nothing in the operators' own renewal processes could see it.
- SoftBank's 4G service was down from 13:39 to 18:04 JST and recovered by reverting to an older software version; O2 restored 4G the next morning.
- Microsoft's status message on 3 February 2020 said an authentication certificate had expired, and Teams users could not use the service until a new one was deployed.
- Equifax's SSL inspection device had an expired certificate for 19 months, so the 76-day exfiltration in 2017 went unseen; over 300 of its certificates had expired.
- Root expiry breaks clients, not servers: when DST Root CA X3 expired on 30 September 2021, clients without ISRG Root X1 or with OpenSSL 1.0.2 failed.
- Public TLS certificates now last at most 200 days and will last 47 days from March 2029, so manual renewal fails more often every year.

## Incident at a glance

- Date: 2016-01-31 to 2021-09-30
- Category: Outage
- Organizations: Equifax; Ericsson; SoftBank; O2 (UK); Microsoft; Let’s Encrypt
- Impact: Expired certificates cut mobile data for operators in about 11 countries (Ericsson, 2018) and blocked access to Microsoft Teams (2020). At Equifax an expired inspection certificate left monitoring blind for 19 months, through the 2017 breach.
- How it was found: Each failure surfaced only at or after expiry: services stopped when the certificates lapsed, and Equifax saw the breach traffic only after renewing its certificate on 29 July 2017.

### Timeline

1. 2016-01-31: The certificate on Equifax’s SSL Visibility device for its dispute portal expires, and the device stops inspecting traffic.
2. 2017-07-29: Equifax renews the certificate and immediately sees suspicious traffic, which reveals the breach.
3. 2018-12-06: An expired certificate in two Ericsson SGSN-MME software versions cuts mobile data for SoftBank, O2 and operators in about 11 countries.
4. 2020-02-03: Users cannot access Microsoft Teams because an authentication certificate has expired.
5. 2021-09-30: IdenTrust’s DST Root CA X3 expires; clients that do not trust ISRG Root X1 start rejecting Let’s Encrypt certificates.
6. 2025-04-11: The CA/Browser Forum adopts ballot SC-081, which shortens public TLS certificate lifetimes in steps to 47 days.
7. 2026-03-15: The maximum validity of a public TLS certificate drops to 200 days.

## Recommended defaults

| Setting | Value | Note | Source |
| --- | --- | --- | --- |
| Ericsson outage | 6 December 2018 | Two SGSN-MME software versions; an expired certificate per Ericsson's initial root-cause analysis. | Ericsson press release; SoftBank press release |
| SoftBank outage window | 13:39 to 18:04 JST | Recovered by reverting all packet switches to older software. | SoftBank press release |
| Microsoft Teams outage | 3 February 2020, from about 8:30 a.m. ET | An expired authentication certificate. | TechCrunch, quoting Microsoft 365 Status |
| Equifax monitoring gap | 19 months with an expired certificate on the SSL inspection device | Over 300 expired certificates, 79 for monitoring business-critical domains. | US House Oversight Committee report, December 2018 |
| Maximum public TLS certificate validity | 200 days now; 100 days from 15 March 2027; 47 days from 15 March 2029 | — | CA/Browser Forum ballot SC-081v3 |
| Renewal timing | At the ACME ARI window, or at two-thirds of the lifetime | — | RFC 9773; Let's Encrypt |
| Alert threshold | Served certificate with less than 30 days, or less than a third of its lifetime, left | — | — |

## Verify it

1. Run the checker on this page against every endpoint you serve and fail the job when any certificate has less than 30 days left.
2. List the certificates inside products and devices you ship or buy, with their expiry dates, and ask vendors for the dates you cannot read yourself.
3. Start a test environment with the clock set past the earliest embedded certificate's expiry and confirm the product either renews or fails with a clear alert.
4. Confirm that every TLS-inspection, monitoring and VPN device alerts on its own certificate's expiry, and that an expired one raises an incident rather than silently passing traffic uninspected.
5. Check that clients trust the current roots of the CAs you use and that no client pins a root or intermediate that will expire.

## What you will learn

- Explain three ways an expired certificate causes harm: a server outage, a client trust failure and lost monitoring.
- Build a certificate inventory that includes certificates inside products and devices.
- Monitor served certificates and fail a pipeline before expiry.
- Test software for certificate time bombs by moving the clock forward.

Audience: Site reliability, platform and security engineers who run certificates in production, and engineers who build products or devices that contain certificates.

Before you start:

- Know what a certificate's notAfter date is and how a TLS client validates a chain.
- For the code: bash and OpenSSL 1.1.1 or later.

Time: About 28 minutes: 6 to read and 22 for the exercise

## Ericsson 2018: a certificate inside the software

On 6 December 2018 Ericsson reported network disturbances for customers in several countries running two specific software versions of its SGSN-MME, the core-network node that handles mobile data sessions. Its initial root-cause analysis found that the main issue was an expired certificate in those software versions. Chief executive Börje Ekholm apologized and said the faulty software was being decommissioned.

SoftBank's account is the most precise primary record. Its 4G LTE service for the SoftBank and Y!mobile brands, its home phone service and part of SoftBank Air were unavailable across Japan from 13:39 to 18:04 JST. Ericsson told SoftBank the software had been in operation for about nine months and that carriers in 11 countries using the same equipment had failed at the same time. SoftBank restored service by reverting every packet switch to an older software version. In the UK, millions of O2 customers lost mobile data; O2 said its 4G network was restored the next morning.

The operators did nothing wrong in their own certificate management: the certificate was inside a vendor's software, invisible to their inventories, and it expired everywhere at once. The defenses are to ask suppliers which certificates and time limits their products contain, to test products with the clock moved forward before deployment, and to design software that reports an approaching expiry rather than failing at the moment it is reached.

## Microsoft Teams 2020: an authentication certificate

On the morning of 3 February 2020 Microsoft Teams stopped working for many users from about 8:30 a.m. ET. Microsoft's status account said: 'We've determined that an authentication certificate has expired causing, users to have issues using the service.' Microsoft then deployed an updated certificate, and service recovered over the following hours.

The cause is the most common one: a certificate whose renewal depended on someone remembering, in a service that depended on it for sign-in. The fix is the same at any scale: automated renewal with ACME, monitoring of the certificate actually served by each endpoint, and an alert early enough to act.

## Equifax 2017: an expired certificate that blinded monitoring

An expired certificate does not always cause an outage. At Equifax, traffic to the online dispute portal (ACIS) passed through an SSL Visibility appliance that decrypted it for inspection. Its certificate had expired, and according to the US House Oversight Committee's December 2018 report, the device had been inactive for 19 months because of it. Attackers who entered through an unpatched Apache Struts flaw queried databases and exfiltrated personal data for 76 days without being seen.

On 29 July 2017 Equifax renewed the certificate and immediately saw suspicious traffic, which is how the breach was discovered. The committee found that Equifax had let more than 300 security certificates expire, including 79 for monitoring business-critical domains, and that it knew it lacked a process for updating them. A security control that fails open when its certificate expires should page someone, not quietly stop inspecting.

## Let's Encrypt 2021: when a root certificate expires

Root expiry breaks clients instead of servers. Let's Encrypt's certificates chained to its own ISRG Root X1 and, for older devices, through a cross-signature to IdenTrust's DST Root CA X3, which expired on 30 September 2021. Devices that did not trust ISRG Root X1 began to fail, and clients using OpenSSL 1.0.2 could reject valid certificates because they built the chain through the expired root. Let's Encrypt told API and IoT operators that every client must trust ISRG Root X1 and that OpenSSL clients must use version 1.1.0 or later.

The defense is on the client side: keep trust stores updated through the platform, never pin a root or intermediate you cannot update, and test every client family you support, including embedded devices and old language runtimes, before a CA's published expiry dates.

## Why certificates still expire unnoticed



- No inventory: certificates live on load balancers, service meshes, appliances, devices and inside vendor software, and no single list holds them all.
- Renewal is manual or half-automated: the certificate renews but the listener is never reloaded, or a cron job assumes a fixed lifetime.
- Monitoring checks the wrong thing: a file on disk or a renewal job's exit code instead of the certificate the endpoint serves.
- Nobody owns the expiry date of a certificate embedded in a product, and nobody tests with the clock moved forward.
- Lifetimes keep shrinking: 200 days now, 100 days from March 2027 and 47 days from March 2029, so every manual step fails more often.

## What to do today: inventory, automate, monitor, test

Build the inventory from several directions: scan your address ranges and service registry, read Certificate Transparency logs for your domains, and ask vendors for the certificates and validity periods inside their products. The cryptographic inventory guide at /domains/identity-crypto/key-management-pki/crypto-inventory-pqc/ builds its list from the same sources. Automate issuance, deployment and reload with ACME and let the CA's ACME Renewal Information (RFC 9773) choose the renewal time; without it, renew at about two-thirds of the lifetime. The TLS operations guide at /domains/identity-crypto/key-management-pki/tls-pki-operations/ covers configuration and lifetimes in detail.

Monitor what each endpoint actually serves, from a fresh handshake, and fail loudly before expiry. The checker below reads a list of endpoints and exits with status `1` if any served certificate expires within the threshold, or `2` if one cannot be reached, so it can run in CI or a scheduler. It was tested with OpenSSL 3.6.4 against local servers presenting a 400-day and a 10-day certificate. For products that embed certificates, run a test environment with the clock set past the earliest expiry, for example with libfaketime, before the date arrives in production.

```bash
#!/usr/bin/env bash
# Report each endpoint whose served certificate expires within DAYS (default 30).
# Usage: DAYS=30 ./cert-expiry.sh endpoints.txt   (one host or host:port per line)
set -u
days=${DAYS:-30}
status=0
while read -r target; do
  [ -z "$target" ] && continue
  host=${target%%:*}
  port=${target##*:}
  [ "$port" = "$target" ] && port=443
  # Fetch the certificate the listener serves now, from a fresh handshake.
  cert=$(openssl s_client -connect "$host:$port" -servername "$host" </dev/null 2>/dev/null |
    openssl x509 2>/dev/null)
  if [ -z "$cert" ]; then
    echo "ERROR    $target: no certificate (connection or handshake failed)"
    status=2
    continue
  fi
  end=$(printf '%s\n' "$cert" | openssl x509 -noout -enddate | cut -d= -f2)
  if printf '%s\n' "$cert" | openssl x509 -noout -checkend $((days * 86400)) >/dev/null; then
    echo "OK       $target: expires $end"
  else
    echo "EXPIRING $target: expires $end, within $days days"
    status=1
  fi
done <"$1"
exit "$status"

```

## Guided practice

It is 10 October 2026. You own the certificate inventory below. Decide which entries need action now, which will break as lifetimes shrink, and write the monitoring rule and the test that would have caught the Ericsson failure. Produce the answers before reading the worked ones.

### Supplied material

```text
id, location, notAfter (UTC), issued, lifetime, renewal
C1, api.example.com (public load balancer), 2026-10-27, 2026-04-10, 200 days, ACME with ARI; listener reloads automatically
C2, intranet.example.internal (private CA), 2026-11-02, 2025-11-02, 365 days, manual ticket
C3, payments gateway appliance (TLS inspection), 2026-10-15, 2024-10-15, 730 days, vendor console, manual
C4, client certificate inside firmware 4.2 on 30,000 field devices, 2027-03-01, 2022-03-01, 5 years, firmware update only
C5, www.example.com, 2027-01-05, 2026-07-09, 180 days, cron job every 150 days copies a new file; no reload
```

### 1. Triage by date

List which certificates expire within 30 days of 10 October 2026, and which within 6 months.

Worked answer: Within 30 days: C3 (5 days, 15 October), C1 (17 days, 27 October) and C2 (23 days, 2 November). Within six months, add C5 (87 days, 5 January 2027) and C4 (142 days, 1 March 2027).

### 2. Find the real risks

For each certificate, say whether its renewal process will actually work, and why.

Worked answer: C1 is fine: ACME with ARI renews in time and the listener reloads. C3 is the most urgent: manual, five days left, and like Equifax's device an inspection appliance may fail open, so expiry could silently stop inspection. C2 depends on a ticket. C5 renews a file every 150 days but never reloads the server, so the served certificate can still expire; and the fixed 150-day cron breaks once public lifetimes drop to 100 days in March 2027. C4 can only be fixed by shipping firmware to 30,000 devices, which needs months: it is the Ericsson pattern.

### 3. Write the monitoring rule

Write the rule that alerts on all five, using the served certificate rather than the files.

Worked answer: Check every endpoint's served certificate from a fresh handshake daily (the page's checker with `DAYS=30`, plus a warning at one third of the lifetime remaining), page on any result of `1` or `2`, and add the embedded C4 expiry to the same inventory as a dated task with an owner, because no handshake will show it before the devices fail.

### 4. Write the time-bomb test

Describe a test that would have caught Ericsson's failure before deployment, applied to C4.

Worked answer: Run firmware 4.2 in a test rig with the clock set after 2027-03-01 (for example with libfaketime or the device's own clock setting) and confirm the device either uses a renewed certificate or reports the coming expiry clearly while still able to receive an update. Repeat for every release, and require suppliers to state embedded certificate expiry dates.

### 5. Fix C5 for 47-day certificates

Rewrite C5's renewal so it survives lifetimes of 100 and 47 days.

Worked answer: Replace the fixed 150-day cron with an ACME client that asks for ARI renewal windows (or renews at two-thirds of the actual lifetime), reload the web server after each renewal, and verify the served certificate's new serial number and `notAfter` from a fresh handshake.

Deliverable: A triage table for C1 to C5 with days remaining, risk and action, the monitoring rule, a clock-forward test plan for C4 and a corrected renewal design for C5.

Scope: The inventory is synthetic. Real inventories also need certificates you do not know about, which only scanning and Certificate Transparency monitoring will find.

Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security.

## Sources

- [Ericsson: update on software issue impacting certain customers (6 December 2018)](https://www.ericsson.com/en/press-releases/2018/12/update-on-software-issue-impacting-certain-customers)
- [SoftBank: apology for mobile communication service troubles (6 December 2018)](https://www.softbank.jp/en/corp/news/press/sbkk/2018/20181206_02/)
- [TechCrunch: here's what caused yesterday's O2 and SoftBank outages (7 December 2018)](https://techcrunch.com/2018/12/07/heres-what-caused-yesterdays-o2-and-softbank-outages/)
- [TechCrunch: Microsoft Teams has been down this morning (3 February 2020)](https://techcrunch.com/2020/02/03/microsoft-teams-has-been-down-this-morning/)
- [US House Oversight Committee: the Equifax data breach, majority staff report (December 2018)](https://oversight.house.gov/wp-content/uploads/2018/12/Equifax-Report.pdf)
- [Let's Encrypt: DST Root CA X3 expiration (September 2021)](https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/)
- [CA/Browser Forum ballot SC-081v3: reducing validity and data reuse periods](https://cabforum.org/2025/04/11/ballot-sc081v3-introduce-schedule-of-reducing-validity-and-data-reuse-periods/). Adopted 11 April 2025.
- [RFC 9773: ACME Renewal Information (ARI)](https://www.rfc-editor.org/rfc/rfc9773.html). June 2025.
- [Let's Encrypt: decreasing certificate lifetimes to 45 days](https://letsencrypt.org/2025/12/02/from-90-to-45/). 2 December 2025.