---
key: "reproducible-builds"
title: "Investigate build differences and improve reproducibility"
url: "https://softwaresecurity.com/domains/supply-chain/builds/reproducibility-and-hermeticity/"
api: "https://softwaresecurity.com/api/v1/articles/reproducible-builds.json"
domain: "supply-chain"
topic: "Software supply chain"
type: "Guide"
identifiers: ["A03:2025","A08:2025"]
tags: ["Build integrity and provenance"]
stages: ["release"]
roles: ["platform-sre"]
license: "CC-BY-4.0"
license_url: "https://creativecommons.org/licenses/by/4.0/"
code_license: "MIT"
---

# Investigate build differences and improve reproducibility

Control inputs and environment, compare artifact bytes, and explain the security properties a repeatable build can demonstrate.

Canonical URL: https://softwaresecurity.com/domains/supply-chain/builds/reproducibility-and-hermeticity/

## In short

Fix every input that reaches the bytes: pin source, toolchain and dependencies, set `SOURCE_DATE_EPOCH` from the release commit, and normalize archive order, timestamps and owners while keeping file modes. Rebuild on an independent clean builder and compare hashes at a declared boundary such as the unsigned archive. When the hashes differ, run diffoscope to find the first difference instead of normalizing fields until the hashes happen to match.

- Reproducibility, hermeticity, isolation and signing answer different questions; identical bytes can still come from malicious source.
- `SOURCE_DATE_EPOCH` is the standard variable that compilers, archivers and image builders read instead of the current time.
- Normalize archive order, times and owners on purpose, but keep file modes and symlink targets, which change behavior.
- Two builds on the same compromised worker can agree; an independent rebuild must vary the builder or the organization.
- diffoscope shows where two artifacts differ, down to archive members and metadata fields.

## Recommended defaults

| Setting | Value | Note | Source |
| --- | --- | --- | --- |
| `SOURCE_DATE_EPOCH` | `$(git log -1 --pretty=%ct)`, the release commit time | — | Reproducible Builds: SOURCE_DATE_EPOCH |
| GNU tar options | `--sort=name --mtime=@$SOURCE_DATE_EPOCH --owner=0 --group=0 --numeric-owner` | — | Reproducible Builds: archives |
| gzip | `gzip -n`, which omits the file name and time from the header | — | Reproducible Builds: archives |
| Go | `CGO_ENABLED=0 go build -trimpath` | — | Go blog: perfectly reproducible, verified Go toolchains |
| Maven | `project.build.outputTimestamp` in pom.xml | Check with `mvn clean verify artifact:compare`. | Maven: configuring for reproducible builds |
| Docker BuildKit image layers | `SOURCE_DATE_EPOCH` build argument plus `--output type=image,rewrite-timestamp=true` | — | BuildKit: build reproducibility |

## Verify it

1. Build twice with the same `SOURCE_DATE_EPOCH` on two clean builders and compare `sha256sum` of the unsigned archive.
2. Run `diffoscope build-a/cedar.tar.gz build-b/cedar.tar.gz` and expect exit status 0 at the declared boundary.
3. List the archive with `tar -tvzf cedar.tar.gz` and confirm sorted names, owner `0/0`, the release timestamp and the original modes.
4. Rebuild in a different directory, timezone and locale and confirm the hash does not change.
5. For Maven projects, run `mvn clean verify artifact:compare` after a release build.

## What you will learn

- Distinguish reproducibility from isolation and authenticity.
- Diagnose metadata-only and payload divergences at the correct boundary.
- Define justified normalization and independent reconstruction tests.

Audience: Build and release engineers who need independent rebuilds to match, and reviewers who verify them

Before you start:

- Distinguish deterministic output, hermetic inputs and independently trusted build infrastructure.

Time: About 28 minutes: 6 to read and 22 for the exercise

## Separate reproducibility, isolation and authenticity

A reproducible build produces identical specified outputs from the same source, build instructions and defined environment. Reproducible Builds' definition centers on independent recreation of identical artifacts. Hermeticity concerns controlling external inputs; isolation concerns interference between executions; a signature concerns authenticity. They reinforce one another but are not synonyms. An isolated build may embed the current time, and two reproducible builds may faithfully compile malicious source. The xz Utils backdoor (/domains/supply-chain/dependencies/case-study-xz-utils/) shows the limit: rebuilding its tampered release tarball reproduces the backdoor; only regenerating the tarball from the tagged source would expose the injected file.

For Cedar, the intended comparison is the unsigned release archive. Signing adds a timestamp and identity evidence that can intentionally vary, so comparing signed envelopes without defining the comparison boundary can create misleading failures. State whether the target is an executable, package payload or entire archive, and keep every difference outside that target explicit. Ignoring arbitrary differences until hashes match weakens the claim rather than achieving reproducibility.

## Record inputs that actually influence the bytes

A source commit and lockfile are important inputs, but not the whole environment. Compiler version, base image, build flags, locale, timezone, filesystem ordering, absolute source paths and timestamps can affect outputs. A dependency fetched by a moving tag is not pinned merely because the source repository was pinned. Capture effective resolved inputs, including downloaded toolchain artifacts, and record whether network access was permitted during the build.

The fixture supplies two builds from the same source and toolchain. Their extracted executables have the same hash, but archive metadata differs in modification time and owner ID. This points to packaging nondeterminism rather than changed executable content. It does not establish that all inputs were controlled: build B used an unrestricted network, so an equal result in this pair is weaker than a deliberately constrained process. SLSA provenance (/domains/supply-chain/provenance/read-and-verify-slsa/) can help record build inputs and producer identity, while reproducibility testing asks the distinct question of whether another execution yields the expected bytes.

## Use the standard levers for each toolchain

Most nondeterminism comes from a few sources: the current time, file-system order, user and group IDs, the build path, locale and timezone, and unpinned inputs. `SOURCE_DATE_EPOCH`, a count of seconds since 1970-01-01 UTC that is usually the time of the release commit, is the convention tools use to replace the current time. Set `TZ=UTC` and a fixed locale such as `LC_ALL=C` as well.

| Toolchain | Lever | Watch for |
| --- | --- | --- |
| GNU tar 1.28 or later | `--sort=name --mtime=@$SOURCE_DATE_EPOCH --owner=0 --group=0 --numeric-owner`, piped to `gzip -n` | gzip stores a name and time unless `-n` is set. |
| Zip and JAR | `zip -X` with fixed file times, or strip-nondeterminism afterwards | Unzip with `TZ=UTC`, because zip stores local times. |
| Go 1.21 or later | `CGO_ENABLED=0 go build -trimpath` | cgo makes the host C toolchain an input; pin it in a build image. |
| Maven | `project.build.outputTimestamp`, then `mvn clean verify artifact:compare` | Reproducible mode is on by default since Maven 4.0.0-beta-5. |
| Gradle 9 or later | Archive tasks are reproducible by default | Files default to mode 0644, so set `filePermissions` for executables. |
| Rust | `RUSTFLAGS="--remap-path-prefix=$PWD=/build"` and `cargo build --locked` | Build scripts and proc macros can read the environment. |
| Docker BuildKit 0.13 or later | `SOURCE_DATE_EPOCH` build argument and `--output type=image,rewrite-timestamp=true` | Pin base images by digest; Buildx passes `SOURCE_DATE_EPOCH` from the environment. |

## Package the release deterministically

The script below implements the Cedar packaging contract with GNU tar: sorted entry names, one timestamp taken from the release commit, numeric owner 0, no access or change times in pax headers, and a gzip header without a name or time. It keeps file modes, so `bin/cedar` stays `0755`. Run in two directories with different file times, creation order and owners, it produced byte-identical archives whose entries all carry time `1700000000`.

```bash
#!/usr/bin/env bash
# Package dist/ deterministically: sorted names, one timestamp, numeric owner 0,
# no atime/ctime headers and no gzip name or time. File modes are preserved.
set -euo pipefail
export SOURCE_DATE_EPOCH="${SOURCE_DATE_EPOCH:-$(git log -1 --pretty=%ct)}"
tar --sort=name --mtime="@${SOURCE_DATE_EPOCH}" \
    --owner=0 --group=0 --numeric-owner \
    --pax-option=exthdr.name=%d/PaxHeaders/%f,delete=atime,delete=ctime \
    -cf - dist | gzip -9 -n > cedar.tar.gz
sha256sum cedar.tar.gz
```

## Find and explain the first meaningful divergence

Compare hashes at the declared boundary, then inspect structure rather than immediately rebuilding repeatedly. For an archive, compare entry names, file payloads, modes, ordering, owners and times. If the extracted payload matches but container metadata differs, adjust the packaging process to normalize only fields whose meaning is intentionally fixed by policy. A stable release epoch, consistent ordering and fixed owner metadata can remove accidental variation; the packaging script above applies all three. Executable permission bits and symlink targets may be semantically significant and must not be discarded casually.

For the supplied data, fixing `mtime` and owner IDs is justified by the packaging contract. It is not justified to ignore the mode of `bin/cedar`, because `0755` versus `0644` changes whether it is executable after extraction. The later build C changes the executable payload hash as well as metadata, so the previous explanation is insufficient. Investigate source, compiler options and fetched inputs before accepting C. A reproducibility mismatch is a diagnostic signal; without more evidence it does not identify an attacker or the root cause.

## Let diffoscope find the divergence

diffoscope recursively unpacks archives, packages and binaries and reports the differences in a readable form. On two plain `tar` builds of Cedar it reports the changed modification time of `bin/cedar` and the changed owner of `README`, exactly the metadata the contract normalizes, and nothing in the payloads. It exits with status 0 when the inputs are identical and 1 when they differ, so it can gate a pipeline. reprotest automates the other half: it builds twice while varying time, locale, user, umask and build path, then compares the results.

```bash
# Compare two builds of the same release. Exit status 0 means identical, 1 means different.
diffoscope --text diffoscope.txt build-a/cedar.tar.gz build-b/cedar.tar.gz
```

## Make independence and the comparison claim explicit

Rebuilding twice on the same compromised worker can reproduce the same tampered output. Independent reconstruction is stronger when it varies the organization, infrastructure or verification path while keeping the declared build inputs fixed. Independence still has limits: shared compiler binaries, source hosting or dependency mirrors may be common points of failure. Record those shared dependencies rather than presenting the second build as universally independent. Two incidents show why: SolarWinds' build server compiled altered Orion source (/domains/supply-chain/builds/case-study-solarwinds/), and attackers who reached 3CX's build environments shipped signed, trojanized desktop apps (/domains/supply-chain/builds/case-study-3cx/); an independent rebuild from the reviewed source would not have matched either release.

Operationally, deterministic packaging makes release diffs and incident investigation easier, but strict bit-for-bit gates can delay releases while harmless nondeterminism is investigated. Define the desired target, a documented normalization policy and a triage path. Keep the original differing artifacts and comparison reports; destructive normalization would erase useful evidence. The desk exercise asks for a precise explanation and next test, not an actual compiler setup. Its output hashes are symbolic and its conclusions apply only to the supplied comparison boundary.

## Guided practice

Cedar compares unsigned tar archives built from the same source. The packaging contract requires sorted entries, fixed release timestamp `1700000000`, owner/group 0, and preserved file modes.

### Supplied material

```text
Inputs A/B: source C10; compiler T4; lockfile L7; target linux/amd64.
Build A archive hash AR1: bin/cedar payload EX1, mode 0755, mtime 1700000000, uid/gid 0/0; README payload DOC1, mode 0644.
Build B archive hash AR2: bin/cedar payload EX1, mode 0755, mtime 1700000300, uid/gid 501/20; README payload DOC1, mode 0644. Same entry order as A.
Build B network allowed; fetched inputs log incomplete.
Build C archive hash AR3: bin/cedar payload EX9, mode 0755, mtime 1700000000, uid/gid 0/0; README DOC1. Source claims C10; compiler flags missing.
Proposal P1: compare only extracted file hashes and call entire archives reproducible.
Proposal P2: normalize timestamps/owners per contract, preserve modes, rerun comparison while retaining originals.
```

### 1. Name the comparison target

Decide whether A/B meet the stated archive-level goal.

Worked answer: No. AR1 differs from AR2 at the unsigned archive boundary. Equal EX1 payloads support a narrower executable-content comparison, not equality of whole archives.

### 2. Explain the observed difference

Identify all supplied differences and avoid inventing another cause.

Worked answer: B changes mtime by 300 seconds and uid/gid from `0/0` to `501/20`. Payload hashes, modes and entry order match. These facts support packaging metadata as the observed source of divergence.

### 3. Choose normalization

Compare P1/P2 and state what must be preserved.

Worked answer: P2 matches the stated contract. P1 mislabels a narrower comparison. Preserve `0755`/`0644` modes and original artifacts; do not remove semantically relevant fields merely to obtain matching hashes.

### 4. Investigate C

Explain why the A/B remedy cannot authorize C.

Worked answer: EX9 differs from EX1, so timestamp/owner normalization cannot explain the executable content change. Obtain effective compiler flags and fetched inputs, then compare payload structure/source-to-output evidence. No malicious cause is established yet.

### 5. Bound the security claim

Write the result and a stronger next experiment.

Worked answer: A/B show identical payload content with packaging nondeterminism; archive reproducibility remains unproven until rerun. Rebuild with recorded, pinned inputs and controlled networking on a separate clean builder; compare whole unsigned archives under the same contract.

Deliverable: A divergence report separating archive metadata, payload bytes and environmental uncertainty.

Scope: Symbolic hashes are supplied observations. The exercise does not run a build, demonstrate reproducibility or establish compromise.

Articles and teaching material: CC BY 4.0. Original executable code: MIT. Attribution: Software Security.

## Sources

- [Reproducible Builds: definitions](https://reproducible-builds.org/docs/definition/)
- [SLSA 1.2 specification](https://slsa.dev/spec/v1.2/). Approved, November 2025.
- [Reproducible Builds: SOURCE_DATE_EPOCH](https://reproducible-builds.org/docs/source-date-epoch/)
- [Reproducible Builds: archives](https://reproducible-builds.org/docs/archives/)
- [diffoscope](https://diffoscope.org/)
- [Go blog: perfectly reproducible, verified Go toolchains](https://go.dev/blog/rebuild). 28 August 2023.
- [Maven: configuring for reproducible builds](https://maven.apache.org/guides/mini/guide-reproducible-builds.html)
- [Gradle: working with files (reproducible archives)](https://docs.gradle.org/current/userguide/working_with_files.html). Gradle 9.8.1.
- [BuildKit: build reproducibility](https://github.com/moby/buildkit/blob/master/docs/build-repro.md)