Anatomy of Real Incidents
Four incidents, four different stages of the pipeline, four different detection stories. Two of them were found by accident. One sat in production for sixty-five days. One had been in preparation for roughly two years. None of them were caught by a scanner.
The Concept Explained
These incidents are widely cited and rarely dissected. The interesting part is never the headline, it is the mechanism: which stage was subverted, why the existing controls could not see it, and what specifically would have been required to detect it.
Each of the four maps to a different stage from Lesson 1.2, which is why this set is worth studying together rather than individually.
Read each one asking two questions: which trust handoff was occupied, and what would have had to exist for this to be caught deliberately rather than by luck? The second question is the one that generates controls. In three of these four cases, the answer points to something this course covers, and in the fourth it points to a control that is still genuinely hard.
How It Works
SolarWinds: the build stage
What happened. Attackers gained access to SolarWinds' build environment and injected malicious code into the Orion platform during compilation. The resulting artifact was signed with SolarWinds' legitimate certificate and shipped through normal release channels. The campaign had been underway since at least March 2020.
The numbers that matter. Up to 18,000 customers downloaded an affected version. Fewer than 100 had servers that actually communicated with the threat actor. That ratio is the signature of a targeted supply chain attack: mass distribution, selective exploitation.
Why nothing caught it. The source repository was not modified, so every source-stage control was inspecting clean code. The signature was valid, because the legitimate key signed the compromised output. Customers verifying the download were verifying, correctly, that they had received exactly what SolarWinds built.
What would have been required. Provenance describing how the artifact was built, produced by a build service the build itself could not influence, plus consumers verifying that provenance against expectations. That is precisely SLSA Build Level 3 (Module 4), and it is the incident that motivated much of the framework's design.
Codecov: the artifact and distribution stage
What happened. An error in Codecov's Docker image creation process allowed an attacker to extract a credential for the Google Cloud Storage bucket hosting the Bash Uploader script. They modified the script so that it exfiltrated environment variables from customer CI runs to a third-party server. Customer CI environments are full of credentials, so the payload was a harvest of secrets across every organisation running the uploader.
The timeline. Modified from 31 January 2021, reported 1 April 2021: roughly 65 days in production.
How it was found. A customer verified the shasum of the downloaded script against the shasum published on GitHub, saw a discrepancy, and reported it. Detection came from a consumer doing integrity verification that was available to everyone and performed by almost nobody.
What would have been required. Consumers verifying the artifact against a published, independently hosted digest, which was possible here and rarely done. This one is a strong argument for verification being worthless unless something actually performs it (Module 7), and for the specific practice of never piping a remote script into a shell without checking what you received.
event-stream: the dependency stage
What happened. The maintainer of a widely used npm package, no longer interested in maintaining it, accepted an offer of help from a volunteer and transferred publishing rights. The new maintainer added a dependency that contained a payload targeting a specific cryptocurrency wallet application, decrypting and activating only in that downstream context.
Why nothing caught it. The transfer was a normal open source event. The malicious code was in a transitive dependency, not in the package anyone chose. The payload was encrypted and targeted, so it did nothing observable in almost every environment that installed it.
What would have been required. Realistically, awareness of maintainer changes on critical dependencies and scrutiny of new transitive additions. This is the hardest of the four to defend against, because the mechanism was social rather than technical, and the underlying condition, unpaid maintainers of critical infrastructure being offered help, has not changed.
xz-utils: the source and release stage, over years
What happened. A contributor built credibility on the xz project over roughly two years, gained maintainer access, and introduced a backdoor into the release tarballs for versions 5.6.0 and 5.6.1. The backdoor targeted sshd through its indirect linkage to liblzma, allowing remote code execution for an attacker holding a specific private key. It was assigned CVE-2024-3094 with a CVSS score of 10.
The detail worth knowing. The malicious build machinery was present in the distributed release tarballs but not in an equivalent form in the source repository. Anyone reviewing the Git history was looking at a different artifact than the one distributions were actually building from.
How it was found. Andres Freund, while benchmarking PostgreSQL on Debian sid, noticed SSH logins taking around 500ms instead of the expected 100ms, investigated, found Valgrind errors pointing at liblzma, and traced it to the backdoor. Discovered on 29 March 2024, before the affected versions reached the stable branches of most major distributions.
What would have been required. Reproducible builds, or at minimum verifying that release tarballs correspond to the tagged source. The tarball-versus-repository discrepancy is exactly the kind of thing a build-from-source verification catches and a signature does not, because the tarball was authentically produced by the project.
The common threads
| Stage | Time to detection | Found by | Signature valid? | |
|---|---|---|---|---|
| SolarWinds | Build | Months | Third-party investigation | Yes |
| Codecov | Artifact distribution | 65 days | A customer checking a shasum | n/a |
| event-stream | Dependency | Weeks | Community investigation of a change | n/a |
| xz-utils | Source and release | ~2 years of preparation | Accidental performance investigation | Yes |
Three things repeat and are worth stating plainly:
Signatures were valid where they existed. In both cases with signing, the compromise happened before signing, so the signature certified the malicious artifact. This is the central point of Lesson 1.4.
Detection was accidental or manual in every case. Not one was caught by a scanner or an automated control. Two were found by an individual noticing something odd.
The victims were not careless. These are well-run projects and companies. The failure was in a trust model everyone shared, not in the diligence of any team.
The lesson people draw from xz-utils is usually "we got lucky," and that is true and incomplete. The more useful reading is that the attack was defeated by a discrepancy between two representations of the same thing: the repository and the tarball. Almost every technical control in this course exists to create discrepancies that a machine can notice, so that detection does not depend on an engineer being curious about half a second of latency.
In Production
- Use these incidents to scope, not to alarm. Each one maps to a stage and a control, which makes them useful for prioritising rather than just for slide decks.
- Ask which of these you would survive. Walking the four against your own pipeline is a faster gap analysis than any questionnaire.
- Note that two were caught by consumers, not producers. Being a good consumer, verifying digests and pinning, protects you even when upstream is compromised.
- Maintainer changes on critical dependencies are a signal. Few organisations watch for them; the tooling exists (Module 2).
- Assume a valid signature and a clean scan will both be present in your next incident, because they were in these.
- Time to detection is the number to reduce. Sixty-five days and two years are the benchmarks to beat, and neither is unusual.
Tradeoffs and Decision Framework
Preventing against detecting. You cannot prevent an upstream compromise you do not control. You can shorten detection and reduce exposure, and that is where consumer-side controls (pinning, verification, SBOMs) earn their cost.
Depth of review against volume. Reviewing every dependency is impossible. Reviewing the handful whose compromise would be catastrophic, and watching for maintainer changes on them, is achievable.
Trusting artifacts against building from source. Building critical dependencies yourself catches tarball-versus-repository discrepancies and is a substantial ongoing commitment. Reproducible builds are the middle path, and they are why that practice matters beyond determinism.
Speed against verification in consumption. Piping a remote script into a shell is fast and convenient, and it is the exact pattern Codecov exploited. Verifying a digest costs one line.
Common Mistakes
Treating these as vendor failures rather than model failures. The same trust assumptions exist in your pipeline.
Concluding that signing would have prevented SolarWinds. It was signed. Signing addresses a different threat.
Assuming scanners would catch a repeat. None of these were catalogued vulnerabilities at the time.
Ignoring transitive dependencies because you did not choose them. event-stream's payload was two levels down from anything anyone selected.
Reading xz-utils as pure luck. It was found because two representations disagreed, which is a property you can engineer for.
Skipping consumer-side verification because upstream signs things. Codecov was detected by a consumer checking, and by no one else, for 65 days.