/`. Most sections (kernel, apks, grub) include a content hash in the directory name, so when their inputs change, mkimage rebuilds them. The apkovl section does not: its dir is just `.work/apkovl_/`, with no hash. mkimage finds the cached output and skips genapkovl entirely. The old "trick" of appending a source-tree hash to the staged genapkovl script content — which would invalidate other sections' content-hash caches — has no effect here. `build.sh` now does the right thing: `rm -rf .work/apkovl_* .work/image-*` before invoking mkimage. The kernel and apks sections still cache normally. ### `find ... -exec touch -d '@0' {} +` follows symlinks When zeroing mtimes on the staged apkovl tree for reproducibility, naïve `find -exec touch` will dereference symlinks. Some symlinks in the live profile (`etc/runlevels/default/sshd` → `/etc/init.d/sshd`) resolve to root-owned files *outside the stage*. The build user can't touch them, GNU find returns non-zero, `set -eu` in `build-apkovl.sh` aborts — *silently*, because mkimage doesn't check `build_apkovl()`'s exit code. Net effect: the target apkovl built fine (its sources had no such symlinks), the live apkovl was missing, mkimage reused the previous (stale) live apkovl. We thought the build "succeeded" because exit code 0 and a new ISO timestamp on the *target* apkovl. Fix: `find ! -type l`. Symlinks don't need their mtime zeroed — `tar --mtime='@0'` overrides archive mtime regardless. ### World-file parser keeps inline comments Mountaineer's `apkovl-src/common/etc/apk/world` allows trailing comments on package lines (`ifupdown-ng-wifi # harmless on Server; iwd is laptop-only`) for documentation. apk does not. The installer's `install_base()` originally used `grep -v '^#'`, which strips *comment-only* lines but leaves inline-comment lines intact. apk then sees `ifupdown-ng-wifi`, `#`, `harmless`, `on`, `Server;`, `iwd`, … as separate package names and reports `no such package` for every non-real token. Fix: `sed -e 's/#.*//' -e '/^[[:space:]]*$/d' "$WORLD" | xargs`. Strip inline comments first, then blank lines, then collapse to a single space-separated list. ## Live ISO quirks ### `networking` service requires `/etc/network/interfaces` Alpine's OpenRC `networking` service runs `ifquery -a` against `/etc/network/interfaces`. If the file is missing, `ifquery: could not parse` and networking fails to start. For VM tests this is silent — no DHCP, no IP, sshd reachable on the guest but not via the host port forward. Ship at minimum: ``` auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp ``` in `apkovl-src/live/etc/network/interfaces`. On real hardware with multiple interfaces, this brings up the first wired NIC; for Framework 13 wifi-only setups the operator runs `iwctl` from the live env before the installer. ### Alpine's OpenSSH build does not support `PrintLastLog` Despite being a stock OpenSSH option, Alpine's sshd reports `Unsupported option PrintLastLog` and warns on every connection. The default is "yes" anyway. Just don't set it. ### Live ISO has no `scp`/`sftp-server` `openssh-server` in the live profile doesn't pull `openssh-sftp-server`, so `scp` from the host to the live env fails with `scp: Connection closed`. Workaround: pipe over stdin: ``` ssh ... 'cat > /tmp/path' < local-file ``` This is what the iteration loop uses to live-patch the installer between builds. ## Installer step traps ### busybox `mount` can't auto-detect FAT32 on a fresh ESP Right after `mkfs.vfat -F32 ... $ESP`, `mount $ESP /mnt/boot/efi` fails with `Invalid argument`. busybox's filesystem detection can't read the boot sector of a freshly-formatted FAT partition fast enough or something similar. Pass `-t vfat` explicitly: ``` mount -t vfat "$ESP" "$TARGET/boot/efi" ``` ### apk on a new root needs `/etc/apk/keys/*.pub` seeded `apk --root /mnt --initdb --repositories-file ... add ` will report `UNTRUSTED signature` on every repository index and treat every package as `no such package`. Reason: the new root has no signing keys in `/etc/apk/keys/`. Seed from the live env: ``` mkdir -p "$TARGET/etc/apk/keys" cp /etc/apk/keys/*.pub "$TARGET/etc/apk/keys/" ``` ### chroot needs `/dev /proc /sys` bind mounts `chroot /mnt ssh-keygen -A` fails with `Couldn't open /dev/null` because the freshly-laid-down target has empty `/dev`. Same for `s6-rc-compile` and any post-install hook that reads `/proc`. Bind-mount before any chroot ops, unmount in finalize: ``` for d in dev proc sys; do mkdir -p "$TARGET/$d" mountpoint -q "$TARGET/$d" || mount --bind "/$d" "$TARGET/$d" done ``` ### `run "$@"` vs `run $@` — pick your poison `installer/install.sh` has a `run()` helper that prints in `--dry-run` mode and otherwise executes its args. The original implementation uses unquoted `$@` so that callers can pass word-splittable strings: ``` run zpool create $pool_opts rpool "$ZPART" ``` where `$pool_opts` is a multi-token string. Quoting `"$@"` breaks that. But unquoted `$@` then re-splits *every* argument on whitespace — including args that legitimately contain spaces, like a ZFS property value: ``` zfs set org.zfsbootmenu:commandline='ro quiet console=ttyS0,115200' rpool/ROOT/mountaineer ``` becomes six arguments to `zfs set`, which prints its usage. Don't try to "fix" `run`. Bypass it for the calls with embedded spaces: ``` if [ "$DRY_RUN" = 1 ]; then echo "[dry-run] zfs set ..." else zfs set org.zfsbootmenu:commandline='...' "$ROOT_DATASET" fi ``` ## ZFSBootMenu ### ZBM ships as a unified EFI binary The ZBM release contains a self-contained EFI binary (`zfsbootmenu-release-x86_64-vN.N.N-linuxX.Y.EFI`) that bundles a kernel, ZFS module, initramfs, and the menu UI. Drop it as `/EFI/BOOT/BOOTX64.EFI` on the ESP and the firmware finds it via the removable-media fallback path. The older approach — separate `vmlinuz-bootmenu` and `initramfs-bootmenu.img` — still works but adds two more files to update on every ZBM bump. INSTALL.md describes the three-file layout; the implementation uses the simpler unified binary. The spec should be updated when somebody cares. ### The `linuxX.Y` in the asset name is metadata `zfsbootmenu-release-x86_64-v3.1.0-linux6.18.EFI` tells you which kernel *ZBM itself was built with*, not which kernel ZBM will boot. ZBM chainloads whatever vmlinuz is in `/boot` of the selected root dataset. The `linux6.18` suffix is just hardware-support metadata — newer kernel = better support for recent NICs/storage controllers in the ZBM env itself. Pick the newest available; the installed system's kernel is independent. Verify the sha256 at install time. The signing key infrastructure is in place upstream (`sha256.sig`), but pinning the sha256 + serving the release URL over HTTPS is sufficient for MVP. ### `efibootmgr` fails in QEMU/EDK2 by default `-drive if=pflash,format=raw,file=$NVRAM` from the `edk2-i386-vars.fd` template gives you EFI vars storage, but the firmware refuses some efibootmgr operations. Make the boot-entry registration best-effort: ``` efibootmgr --create --disk $DISK --part 1 \ --label Mountaineer --loader '\EFI\BOOT\BOOTX64.EFI' || \ echo "warning: relying on \\EFI\\BOOT\\BOOTX64.EFI fallback path" ``` The removable-media fallback works whether or not efibootmgr did. ### Per-host kernel cmdline lives in `org.zfsbootmenu:commandline` ZBM reads the kernel command line for the dataset it boots from the `org.zfsbootmenu:commandline` ZFS property on that dataset. There is *also* a `KCL` baked into the ZBM EFI binary itself, but that controls ZBM's own kernel, not the chainloaded one. Per-install values: ``` zfs set org.zfsbootmenu:commandline='ro quiet console=tty0 console=ttyS0,115200' rpool/ROOT/mountaineer ``` The dual-console form means the kernel emits to both ttyS0 (VM serial, out-of-band management) and tty0 (physical display); init's controlling tty is whichever was listed last. Harmless on hardware without a serial port. ## Iteration mechanics ### Live-patch the installer instead of rebuilding When iterating on `installer/install.sh`, you don't need a full ISO rebuild — pipe the new script over ssh into the running live env: ``` ssh ... 'cat > /usr/local/sbin/mountaineer-install && chmod +x /usr/local/sbin/mountaineer-install' < installer/install.sh ``` The next `mountaineer-install` invocation picks it up. Only rebuild the ISO when you've reached a "real testable artifact" stopping point, or when the apkovl content itself (not the installer) changed. ### Standalone-boot the installed disk After the installer reboots, the install-target VM by default boots back into the live ISO (boot-order keeps the CD-ROM first). To exercise the installed system's UEFI → ZBM → Mountaineer path: ``` qemu-system-x86_64 \ -machine type=q35 -accel tcg,thread=multi -cpu max -smp 4 -m 4G \ -drive if=pflash,format=raw,readonly=on,file=/opt/homebrew/share/qemu/edk2-x86_64-code.fd \ -drive if=pflash,format=raw,file=.vms/nvram/installed_VARS.fd \ -drive file=.vms/disks/target.qcow2,if=virtio,format=qcow2 \ -serial file:.vms/logs/installed-boot-serial.log \ -netdev user,id=net0,hostfwd=tcp:127.0.0.1:2224-:22 \ -device virtio-net-pci,netdev=net0 \ -display none -monitor unix:.vms/sockets/installed-mon.sock,server,nowait \ -daemonize -vga virtio -device virtio-rng-pci ``` A fresh `installed_VARS.fd` (copy of `edk2-i386-vars.fd`) is needed because efivars from the install-target run carry no useful state. ### TCG timing on Apple Silicon x86_64-on-arm via TCG is the bottleneck. Rough numbers: - Builder VM first provision: ~5-10 min (cloud-init). - Kernel + modloop mksquashfs (cold cache): ~25-30 min. - apkovl + ISO assembly (kernel cached): ~3-5 min. - Boot the live ISO to sshd-ready: ~1-2 min. - Full install run end-to-end: ~5-8 min (mostly apk fetches + zfs pool/datasets). Don't `rm -rf .work/` on the builder unless you really mean it. The build script's auto-clean of `.work/apkovl_*` and `.work/image-*` is sufficient for normal iteration. ### macOS host helpers worth installing ``` brew install dosfstools mtools ``` Without these, `vm-test-install.sh` can't stage the FAT-formatted install-config image — it falls back to the "live-patch via ssh" workflow, which works fine but is one more step per iteration. ## Build numbering A build with timestamped-newer-than-expected ISO output is the only reliable signal that mkimage actually re-assembled the image. Exit code 0 is *not* sufficient: mkimage will happily return 0 after silently reusing a stale cached section. Always check `ls -la out/*.iso` against the wall-clock time the build started. ## Reproducibility audit (static, 2026-05-19) A read-through of `build.sh`, `tools/build-apkovl.sh`, `mkimg/genapkovl-mountaineer-server.sh`, and the Alpine `mkimg.iso.sh` contract, without actually rebuilding. The apkovl half is already defensive; the ISO half is where non-determinism likely lives. Findings below are sorted by suspected impact. ### Apkovl tarball — looks reproducible `tools/build-apkovl.sh` already does the right things: - staged tree is rebuilt from `apkovl-src/` every time - `find ! -type l -exec touch -d '@0'` zeroes mtimes (with a portable fallback to `-t 197001010000.00`) - `tar --sort=name --owner=0 --group=0 --numeric-owner --mtime='@0' --format=ustar` produces stable archive order/metadata - `gzip --no-name -9` strips the gzip timestamp/filename - `SOURCE_DATE_EPOCH=0` is exported into tar's environment Two builds of `mountaineer-server-target-0.0.1-mvp.apkovl.tar.gz` from the same source tree on the same builder *should* be byte-identical already. The live apkovl absorbs the target apkovl as a payload, so that should follow. Confidence: high pending an actual diff. Easy first verification: ./tools/build-apkovl.sh server-target /tmp/a.tar.gz ./tools/build-apkovl.sh server-target /tmp/b.tar.gz sha256sum /tmp/a.tar.gz /tmp/b.tar.gz ### ISO half — suspected non-determinism sources 1. **`apk` repository pinning**: `build.sh` passes `--repository https://dl-cdn.alpinelinux.org/alpine/v3.23/main` (and community). v3.23 *branch* drifts as upstream backports roll. Two builds a week apart will fetch different package versions even with no source change. Fix: snapshot mirror, or pre-fetched apk cache committed alongside the build, or pin to a tagged aports commit + that day's main snapshot. 2. **Aports HEAD**: `git clone --depth 1 https://gitlab.alpinelinux.org/alpine/aports.git` pulls current v3.23 head. Same drift as #1. Fix: clone with `--branch ` once Alpine cuts a 3.23.x point release, or record the commit hash and re-checkout on every build. 3. **mkimage `SOURCE_DATE_EPOCH`**: build.sh does not export it into the mkimage invocation. Many of the section scripts (squashfs, xorriso, mkinitfs) honor it if set. Fix: `SOURCE_DATE_EPOCH=0 sh scripts/mkimage.sh ...`. Cheap, almost certainly helps. 4. **`mksquashfs` modloop**: Alpine's `mkimg.modloop.sh` calls `mksquashfs` without explicit determinism flags. Even with `SOURCE_DATE_EPOCH`, the filesystem inode order on the build host leaks into the image. Fix at upstream level: add `-no-exports -no-fragments -mkfs-time 0 -all-time 0`. We may need to patch the section script in the staged aports copy. 5. **`xorriso` ISO9660 timestamps**: `mkimg.iso.sh` issues an `xorrisofs` invocation that, by default, embeds the wall-clock time as the volume creation date and per-file dates. Need to verify whether `--set-all-file-dates @0` and the `volume_date` knobs are already applied. If not, patch the section script. 6. **`mformat` ESP volume serial**: the ESP image (FAT32 via `mformat`) is given a random volume serial on each invocation unless `-N ` is passed. Cheap fix: `-N 0` everywhere it's called. 7. **mkinitfs/ramdisk**: cpio archive ordering and any embedded modaliases. Modern mkinitfs honors `SOURCE_DATE_EPOCH`, so #3 above probably covers it. Verify with `cmp` after a build pair. 8. **GRUB EFI image**: `grub-mkstandalone` / equivalent embeds a build timestamp inside the EFI binary by default. Pass `--modules=... -t 0` or rebuild from a deterministic source. Items 3–6 are the ones most worth attacking first — each is a single-line patch with a near-zero downside, and together they likely close the gap from "same source, different bytes" to "same source, same bytes". Items 1–2 are about reproducibility across machines/days and need package-pinning discipline. ### Recommended next concrete step Run the apkovl-only test (build twice, compare) without spinning up the builder VM. That validates the apkovl half is already deterministic, narrows the search space to the ISO assembly, and costs <30 seconds on the host. Then bring up the builder VM and attempt a full ISO diff with `SOURCE_DATE_EPOCH=0` exported, before attacking individual section scripts. ## Outstanding open issues (May 17 2026) - Reproducible-by-bytes builds — both builds-from-same-source and builds-from-different-checkouts. Best-effort today via `--sort=name`, `--mtime='@0'`, `--owner=0 --group=0`, `SOURCE_DATE_EPOCH=0`. Not byte-stable yet. - Vendor `vector` (the log shipper) from `vector.dev/releases` — STACK pick, not packaged in Alpine 3.23. - Hardware-test on Framework 13. The full installer path has been exercised in QEMU; the NVMe partition-naming branch (`installer/install.sh:171`) has been verified by code-reading only.