Investigate build differences and improve reproducibility

A reproducible build produces identical specified outputs from the same source, build instructions and defined environment. Reproducible Builds' definition centers on independent recreation of identical artifacts. Hermeticity concerns controlling external inputs; isolation concerns interference between executions; a signature concerns authenticity. They reinforce one another but are not synonyms. An isolated build may embed the current time, and two reproducible builds may faithfully compile malicious source. The xz Utils backdoor (/domains/supply-chain/dependencies/case-study-xz-utils/) shows the limit: rebuilding its tampered release tarball reproduces the backdoor; only regenerating the tarball from the tagged source would expose the injected file.

For Cedar, the intended comparison is the unsigned release archive. Signing adds a timestamp and identity evidence that can intentionally vary, so comparing signed envelopes without defining the comparison boundary can create misleading failures. State whether the target is an executable, package payload or entire archive, and keep every difference outside that target explicit. Ignoring arbitrary differences until hashes match weakens the claim rather than achieving reproducibility.

A source commit and lockfile are important inputs, but not the whole environment. Compiler version, base image, build flags, locale, timezone, filesystem ordering, absolute source paths and timestamps can affect outputs. A dependency fetched by a moving tag is not pinned merely because the source repository was pinned. Capture effective resolved inputs, including downloaded toolchain artifacts, and record whether network access was permitted during the build.

The fixture supplies two builds from the same source and toolchain. Their extracted executables have the same hash, but archive metadata differs in modification time and owner ID. This points to packaging nondeterminism rather than changed executable content. It does not establish that all inputs were controlled: build B used an unrestricted network, so an equal result in this pair is weaker than a deliberately constrained process. SLSA provenance (/domains/supply-chain/provenance/read-and-verify-slsa/) can help record build inputs and producer identity, while reproducibility testing asks the distinct question of whether another execution yields the expected bytes.

Most nondeterminism comes from a few sources: the current time, file-system order, user and group IDs, the build path, locale and timezone, and unpinned inputs. SOURCE_DATE_EPOCH, a count of seconds since 1970-01-01 UTC that is usually the time of the release commit, is the convention tools use to replace the current time. Set TZ=UTC and a fixed locale such as LC_ALL=C as well.

The script below implements the Cedar packaging contract with GNU tar: sorted entry names, one timestamp taken from the release commit, numeric owner 0, no access or change times in pax headers, and a gzip header without a name or time. It keeps file modes, so bin/cedar stays 0755. Run in two directories with different file times, creation order and owners, it produced byte-identical archives whose entries all carry time 1700000000.

#!/usr/bin/env bash
# Package dist/ deterministically: sorted names, one timestamp, numeric owner 0,
# no atime/ctime headers and no gzip name or time. File modes are preserved.
set -euo pipefail
export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log -1 --pretty=%ct)}"
tar --sort=name --mtime="@${SOURCE_DATE_EPOCH}" \
    --owner=0 --group=0 --numeric-owner \
    --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
    -cf - dist | gzip -9 -n > cedar.tar.gz
sha256sum cedar.tar.gz

Compare hashes at the declared boundary, then inspect structure rather than immediately rebuilding repeatedly. For an archive, compare entry names, file payloads, modes, ordering, owners and times. If the extracted payload matches but container metadata differs, adjust the packaging process to normalize only fields whose meaning is intentionally fixed by policy. A stable release epoch, consistent ordering and fixed owner metadata can remove accidental variation; the packaging script above applies all three. Executable permission bits and symlink targets may be semantically significant and must not be discarded casually.

For the supplied data, fixing mtime and owner IDs is justified by the packaging contract. It is not justified to ignore the mode of bin/cedar, because 0755 versus 0644 changes whether it is executable after extraction. The later build C changes the executable payload hash as well as metadata, so the previous explanation is insufficient. Investigate source, compiler options and fetched inputs before accepting C. A reproducibility mismatch is a diagnostic signal; without more evidence it does not identify an attacker or the root cause.

diffoscope recursively unpacks archives, packages and binaries and reports the differences in a readable form. On two plain tar builds of Cedar it reports the changed modification time of bin/cedar and the changed owner of README, exactly the metadata the contract normalizes, and nothing in the payloads. It exits with status 0 when the inputs are identical and 1 when they differ, so it can gate a pipeline. reprotest automates the other half: it builds twice while varying time, locale, user, umask and build path, then compares the results.

# Compare two builds of the same release. Exit status 0 means identical, 1 means different.
diffoscope --text diffoscope.txt build-a/cedar.tar.gz build-b/cedar.tar.gz

Rebuilding twice on the same compromised worker can reproduce the same tampered output. Independent reconstruction is stronger when it varies the organization, infrastructure or verification path while keeping the declared build inputs fixed. Independence still has limits: shared compiler binaries, source hosting or dependency mirrors may be common points of failure. Record those shared dependencies rather than presenting the second build as universally independent. Two incidents show why: SolarWinds' build server compiled altered Orion source (/domains/supply-chain/builds/case-study-solarwinds/), and attackers who reached 3CX's build environments shipped signed, trojanized desktop apps (/domains/supply-chain/builds/case-study-3cx/); an independent rebuild from the reviewed source would not have matched either release.

Operationally, deterministic packaging makes release diffs and incident investigation easier, but strict bit-for-bit gates can delay releases while harmless nondeterminism is investigated. Define the desired target, a documented normalization policy and a triage path. Keep the original differing artifacts and comparison reports; destructive normalization would erase useful evidence. The desk exercise asks for a precise explanation and next test, not an actual compiler setup. Its output hashes are symbolic and its conclusions apply only to the supplied comparison boundary.

Sources & further reading

The standards, specifications and documentation behind the guidance above. Each is the evidence for a claim on this page; consult it for detail beyond the defaults and versions listed here.

1. Reproducible Builds: definitions
2. SLSA 1.2 specification. Approved, November 2025.
3. Reproducible Builds: SOURCE_DATE_EPOCH
4. Reproducible Builds: archives
5. diffoscope
6. Go blog: perfectly reproducible, verified Go toolchains. 28 August 2023.
7. Maven: configuring for reproducible builds
8. Gradle: working with files (reproducible archives). Gradle 9.8.1.
9. BuildKit: build reproducibility

Reference information

Tags

Build integrity and provenance

SDLC phase

Release

Role

Platform engineers and SREs

Download this guide as Markdown

Continue exploring