max / mountaineer
1 file changed,
+92 insertions,
-0 deletions
| @@ -279,6 +279,98 @@ reliable signal that mkimage actually re-assembled the image. Exit code | |||
| 279 | 279 | reusing a stale cached section. Always check `ls -la out/*.iso` against | |
| 280 | 280 | the wall-clock time the build started. | |
| 281 | 281 | ||
| 282 | + | ## Reproducibility audit (static, 2026-05-19) | |
| 283 | + | ||
| 284 | + | A read-through of `build.sh`, `tools/build-apkovl.sh`, | |
| 285 | + | `mkimg/genapkovl-mountaineer-server.sh`, and the Alpine `mkimg.iso.sh` | |
| 286 | + | contract, without actually rebuilding. The apkovl half is already | |
| 287 | + | defensive; the ISO half is where non-determinism likely lives. Findings | |
| 288 | + | below are sorted by suspected impact. | |
| 289 | + | ||
| 290 | + | ### Apkovl tarball — looks reproducible | |
| 291 | + | ||
| 292 | + | `tools/build-apkovl.sh` already does the right things: | |
| 293 | + | ||
| 294 | + | - staged tree is rebuilt from `apkovl-src/` every time | |
| 295 | + | - `find ! -type l -exec touch -d '@0'` zeroes mtimes (with a portable | |
| 296 | + | fallback to `-t 197001010000.00`) | |
| 297 | + | - `tar --sort=name --owner=0 --group=0 --numeric-owner --mtime='@0' | |
| 298 | + | --format=ustar` produces stable archive order/metadata | |
| 299 | + | - `gzip --no-name -9` strips the gzip timestamp/filename | |
| 300 | + | - `SOURCE_DATE_EPOCH=0` is exported into tar's environment | |
| 301 | + | ||
| 302 | + | Two builds of `mountaineer-server-target-0.0.1-mvp.apkovl.tar.gz` from | |
| 303 | + | the same source tree on the same builder *should* be byte-identical | |
| 304 | + | already. The live apkovl absorbs the target apkovl as a payload, so | |
| 305 | + | that should follow. | |
| 306 | + | ||
| 307 | + | Confidence: high pending an actual diff. Easy first verification: | |
| 308 | + | ||
| 309 | + | ./tools/build-apkovl.sh server-target /tmp/a.tar.gz | |
| 310 | + | ./tools/build-apkovl.sh server-target /tmp/b.tar.gz | |
| 311 | + | sha256sum /tmp/a.tar.gz /tmp/b.tar.gz | |
| 312 | + | ||
| 313 | + | ### ISO half — suspected non-determinism sources | |
| 314 | + | ||
| 315 | + | 1. **`apk` repository pinning**: `build.sh` passes | |
| 316 | + | `--repository https://dl-cdn.alpinelinux.org/alpine/v3.23/main` | |
| 317 | + | (and community). v3.23 *branch* drifts as upstream backports | |
| 318 | + | roll. Two builds a week apart will fetch different package | |
| 319 | + | versions even with no source change. Fix: snapshot mirror, or | |
| 320 | + | pre-fetched apk cache committed alongside the build, or pin | |
| 321 | + | to a tagged aports commit + that day's main snapshot. | |
| 322 | + | ||
| 323 | + | 2. **Aports HEAD**: `git clone --depth 1 https://gitlab.alpinelinux.org/alpine/aports.git` | |
| 324 | + | pulls current v3.23 head. Same drift as #1. Fix: clone with | |
| 325 | + | `--branch <tag>` once Alpine cuts a 3.23.x point release, or | |
| 326 | + | record the commit hash and re-checkout on every build. | |
| 327 | + | ||
| 328 | + | 3. **mkimage `SOURCE_DATE_EPOCH`**: build.sh does not export it into | |
| 329 | + | the mkimage invocation. Many of the section scripts (squashfs, | |
| 330 | + | xorriso, mkinitfs) honor it if set. Fix: `SOURCE_DATE_EPOCH=0 | |
| 331 | + | sh scripts/mkimage.sh ...`. Cheap, almost certainly helps. | |
| 332 | + | ||
| 333 | + | 4. **`mksquashfs` modloop**: Alpine's `mkimg.modloop.sh` calls | |
| 334 | + | `mksquashfs` without explicit determinism flags. Even with | |
| 335 | + | `SOURCE_DATE_EPOCH`, the filesystem inode order on the build host | |
| 336 | + | leaks into the image. Fix at upstream level: add `-no-exports | |
| 337 | + | -no-fragments -mkfs-time 0 -all-time 0`. We may need to patch | |
| 338 | + | the section script in the staged aports copy. | |
| 339 | + | ||
| 340 | + | 5. **`xorriso` ISO9660 timestamps**: `mkimg.iso.sh` issues an | |
| 341 | + | `xorrisofs` invocation that, by default, embeds the wall-clock | |
| 342 | + | time as the volume creation date and per-file dates. Need to | |
| 343 | + | verify whether `--set-all-file-dates @0` and the `volume_date` | |
| 344 | + | knobs are already applied. If not, patch the section script. | |
| 345 | + | ||
| 346 | + | 6. **`mformat` ESP volume serial**: the ESP image (FAT32 via | |
| 347 | + | `mformat`) is given a random volume serial on each invocation | |
| 348 | + | unless `-N <hex>` is passed. Cheap fix: `-N 0` everywhere it's | |
| 349 | + | called. | |
| 350 | + | ||
| 351 | + | 7. **mkinitfs/ramdisk**: cpio archive ordering and any embedded | |
| 352 | + | modaliases. Modern mkinitfs honors `SOURCE_DATE_EPOCH`, so #3 | |
| 353 | + | above probably covers it. Verify with `cmp` after a build pair. | |
| 354 | + | ||
| 355 | + | 8. **GRUB EFI image**: `grub-mkstandalone` / equivalent embeds a | |
| 356 | + | build timestamp inside the EFI binary by default. Pass | |
| 357 | + | `--modules=... -t 0` or rebuild from a deterministic source. | |
| 358 | + | ||
| 359 | + | Items 3–6 are the ones most worth attacking first — each is a | |
| 360 | + | single-line patch with a near-zero downside, and together they | |
| 361 | + | likely close the gap from "same source, different bytes" to | |
| 362 | + | "same source, same bytes". Items 1–2 are about reproducibility | |
| 363 | + | across machines/days and need package-pinning discipline. | |
| 364 | + | ||
| 365 | + | ### Recommended next concrete step | |
| 366 | + | ||
| 367 | + | Run the apkovl-only test (build twice, compare) without spinning up | |
| 368 | + | the builder VM. That validates the apkovl half is already | |
| 369 | + | deterministic, narrows the search space to the ISO assembly, and | |
| 370 | + | costs <30 seconds on the host. Then bring up the builder VM and | |
| 371 | + | attempt a full ISO diff with `SOURCE_DATE_EPOCH=0` exported, before | |
| 372 | + | attacking individual section scripts. | |
| 373 | + | ||
| 282 | 374 | ## Outstanding open issues (May 17 2026) | |
| 283 | 375 | ||
| 284 | 376 | - Reproducible-by-bytes builds — both builds-from-same-source and |