| 1 |
# Build Notes |
| 2 |
|
| 3 |
What we learned getting Mountaineer from "doesn't compile" to "ISO that |
| 4 |
boots, installs ZFS-on-root, and reboots into the installed system" over |
| 5 |
sixteen builds. Companion to `../mountaineer/todo.md` (which lists the |
| 6 |
fixes by number); this file explains the *why* and the gotchas that |
| 7 |
aren't obvious from the diff. Read this before you next touch the build |
| 8 |
or installer — most of these are silent-failure traps the first time |
| 9 |
around. |
| 10 |
|
| 11 |
## Build pipeline |
| 12 |
|
| 13 |
### mkimage's apkovl section caches by name only |
| 14 |
|
| 15 |
mkimage stores each "section" of an image under `.work/<section>/`. Most |
| 16 |
sections (kernel, apks, grub) include a content hash in the directory |
| 17 |
name, so when their inputs change, mkimage rebuilds them. The apkovl |
| 18 |
section does not: its dir is just `.work/apkovl_<profile>/`, with no |
| 19 |
hash. mkimage finds the cached output and skips genapkovl entirely. |
| 20 |
|
| 21 |
The old "trick" of appending a source-tree hash to the staged genapkovl |
| 22 |
script content — which would invalidate other sections' content-hash |
| 23 |
caches — has no effect here. `build.sh` now does the right thing: |
| 24 |
`rm -rf .work/apkovl_* .work/image-*` before invoking mkimage. The |
| 25 |
kernel and apks sections still cache normally. |
| 26 |
|
| 27 |
### `find ... -exec touch -d '@0' {} +` follows symlinks |
| 28 |
|
| 29 |
When zeroing mtimes on the staged apkovl tree for reproducibility, naïve |
| 30 |
`find -exec touch` will dereference symlinks. Some symlinks in the live |
| 31 |
profile (`etc/runlevels/default/sshd` → `/etc/init.d/sshd`) resolve to |
| 32 |
root-owned files *outside the stage*. The build user can't touch them, |
| 33 |
GNU find returns non-zero, `set -eu` in `build-apkovl.sh` aborts — |
| 34 |
*silently*, because mkimage doesn't check `build_apkovl()`'s exit code. |
| 35 |
|
| 36 |
Net effect: the target apkovl built fine (its sources had no such |
| 37 |
symlinks), the live apkovl was missing, mkimage reused the previous |
| 38 |
(stale) live apkovl. We thought the build "succeeded" because exit code |
| 39 |
0 and a new ISO timestamp on the *target* apkovl. |
| 40 |
|
| 41 |
Fix: `find ! -type l`. Symlinks don't need their mtime zeroed — |
| 42 |
`tar --mtime='@0'` overrides archive mtime regardless. |
| 43 |
|
| 44 |
### World-file parser keeps inline comments |
| 45 |
|
| 46 |
Mountaineer's `apkovl-src/common/etc/apk/world` allows trailing comments |
| 47 |
on package lines (`ifupdown-ng-wifi # harmless on Server; iwd is |
| 48 |
laptop-only`) for documentation. apk does not. The installer's |
| 49 |
`install_base()` originally used `grep -v '^#'`, which strips |
| 50 |
*comment-only* lines but leaves inline-comment lines intact. apk then |
| 51 |
sees `ifupdown-ng-wifi`, `#`, `harmless`, `on`, `Server;`, `iwd`, … |
| 52 |
as separate package names and reports `no such package` for every |
| 53 |
non-real token. |
| 54 |
|
| 55 |
Fix: `sed -e 's/#.*//' -e '/^[[:space:]]*$/d' "$WORLD" | xargs`. Strip |
| 56 |
inline comments first, then blank lines, then collapse to a single |
| 57 |
space-separated list. |
| 58 |
|
| 59 |
## Live ISO quirks |
| 60 |
|
| 61 |
### `networking` service requires `/etc/network/interfaces` |
| 62 |
|
| 63 |
Alpine's OpenRC `networking` service runs `ifquery -a` against |
| 64 |
`/etc/network/interfaces`. If the file is missing, `ifquery: could not |
| 65 |
parse` and networking fails to start. For VM tests this is silent — no |
| 66 |
DHCP, no IP, sshd reachable on the guest but not via the host port |
| 67 |
forward. |
| 68 |
|
| 69 |
Ship at minimum: |
| 70 |
``` |
| 71 |
auto lo |
| 72 |
iface lo inet loopback |
| 73 |
|
| 74 |
auto eth0 |
| 75 |
iface eth0 inet dhcp |
| 76 |
``` |
| 77 |
in `apkovl-src/live/etc/network/interfaces`. On real hardware with |
| 78 |
multiple interfaces, this brings up the first wired NIC; for |
| 79 |
Framework 13 wifi-only setups the operator runs `iwctl` from the live |
| 80 |
env before the installer. |
| 81 |
|
| 82 |
### Alpine's OpenSSH build does not support `PrintLastLog` |
| 83 |
|
| 84 |
Despite being a stock OpenSSH option, Alpine's sshd reports |
| 85 |
`Unsupported option PrintLastLog` and warns on every connection. The |
| 86 |
default is "yes" anyway. Just don't set it. |
| 87 |
|
| 88 |
### Live ISO has no `scp`/`sftp-server` |
| 89 |
|
| 90 |
`openssh-server` in the live profile doesn't pull `openssh-sftp-server`, |
| 91 |
so `scp` from the host to the live env fails with `scp: Connection |
| 92 |
closed`. Workaround: pipe over stdin: |
| 93 |
``` |
| 94 |
ssh ... 'cat > /tmp/path' < local-file |
| 95 |
``` |
| 96 |
This is what the iteration loop uses to live-patch the installer between |
| 97 |
builds. |
| 98 |
|
| 99 |
## Installer step traps |
| 100 |
|
| 101 |
### busybox `mount` can't auto-detect FAT32 on a fresh ESP |
| 102 |
|
| 103 |
Right after `mkfs.vfat -F32 ... $ESP`, `mount $ESP /mnt/boot/efi` fails |
| 104 |
with `Invalid argument`. busybox's filesystem detection can't read the |
| 105 |
boot sector of a freshly-formatted FAT partition fast enough or |
| 106 |
something similar. Pass `-t vfat` explicitly: |
| 107 |
``` |
| 108 |
mount -t vfat "$ESP" "$TARGET/boot/efi" |
| 109 |
``` |
| 110 |
|
| 111 |
### apk on a new root needs `/etc/apk/keys/*.pub` seeded |
| 112 |
|
| 113 |
`apk --root /mnt --initdb --repositories-file ... add <pkgs>` will |
| 114 |
report `UNTRUSTED signature` on every repository index and treat every |
| 115 |
package as `no such package`. Reason: the new root has no signing keys |
| 116 |
in `/etc/apk/keys/`. Seed from the live env: |
| 117 |
``` |
| 118 |
mkdir -p "$TARGET/etc/apk/keys" |
| 119 |
cp /etc/apk/keys/*.pub "$TARGET/etc/apk/keys/" |
| 120 |
``` |
| 121 |
|
| 122 |
### chroot needs `/dev /proc /sys` bind mounts |
| 123 |
|
| 124 |
`chroot /mnt ssh-keygen -A` fails with `Couldn't open /dev/null` because |
| 125 |
the freshly-laid-down target has empty `/dev`. Same for `s6-rc-compile` |
| 126 |
and any post-install hook that reads `/proc`. Bind-mount before any |
| 127 |
chroot ops, unmount in finalize: |
| 128 |
``` |
| 129 |
for d in dev proc sys; do |
| 130 |
mkdir -p "$TARGET/$d" |
| 131 |
mountpoint -q "$TARGET/$d" || mount --bind "/$d" "$TARGET/$d" |
| 132 |
done |
| 133 |
``` |
| 134 |
|
| 135 |
### `run "$@"` vs `run $@` — pick your poison |
| 136 |
|
| 137 |
`installer/install.sh` has a `run()` helper that prints in `--dry-run` |
| 138 |
mode and otherwise executes its args. The original implementation uses |
| 139 |
unquoted `$@` so that callers can pass word-splittable strings: |
| 140 |
``` |
| 141 |
run zpool create $pool_opts rpool "$ZPART" |
| 142 |
``` |
| 143 |
where `$pool_opts` is a multi-token string. Quoting `"$@"` breaks that. |
| 144 |
|
| 145 |
But unquoted `$@` then re-splits *every* argument on whitespace — |
| 146 |
including args that legitimately contain spaces, like a ZFS property |
| 147 |
value: |
| 148 |
``` |
| 149 |
zfs set org.zfsbootmenu:commandline='ro quiet console=ttyS0,115200' rpool/ROOT/mountaineer |
| 150 |
``` |
| 151 |
becomes six arguments to `zfs set`, which prints its usage. |
| 152 |
|
| 153 |
Don't try to "fix" `run`. Bypass it for the calls with embedded spaces: |
| 154 |
``` |
| 155 |
if [ "$DRY_RUN" = 1 ]; then |
| 156 |
echo "[dry-run] zfs set ..." |
| 157 |
else |
| 158 |
zfs set org.zfsbootmenu:commandline='...' "$ROOT_DATASET" |
| 159 |
fi |
| 160 |
``` |
| 161 |
|
| 162 |
## ZFSBootMenu |
| 163 |
|
| 164 |
### ZBM ships as a unified EFI binary |
| 165 |
|
| 166 |
The ZBM release contains a self-contained EFI binary |
| 167 |
(`zfsbootmenu-release-x86_64-vN.N.N-linuxX.Y.EFI`) that bundles a |
| 168 |
kernel, ZFS module, initramfs, and the menu UI. Drop it as |
| 169 |
`/EFI/BOOT/BOOTX64.EFI` on the ESP and the firmware finds it via the |
| 170 |
removable-media fallback path. |
| 171 |
|
| 172 |
The older approach — separate `vmlinuz-bootmenu` and |
| 173 |
`initramfs-bootmenu.img` — still works but adds two more files to |
| 174 |
update on every ZBM bump. INSTALL.md describes the three-file layout; |
| 175 |
the implementation uses the simpler unified binary. The spec should be |
| 176 |
updated when somebody cares. |
| 177 |
|
| 178 |
### The `linuxX.Y` in the asset name is metadata |
| 179 |
|
| 180 |
`zfsbootmenu-release-x86_64-v3.1.0-linux6.18.EFI` tells you which kernel |
| 181 |
*ZBM itself was built with*, not which kernel ZBM will boot. ZBM |
| 182 |
chainloads whatever vmlinuz is in `/boot` of the selected root dataset. |
| 183 |
The `linux6.18` suffix is just hardware-support metadata — newer kernel |
| 184 |
= better support for recent NICs/storage controllers in the ZBM env |
| 185 |
itself. Pick the newest available; the installed system's kernel is |
| 186 |
independent. |
| 187 |
|
| 188 |
Verify the sha256 at install time. The signing key infrastructure is in |
| 189 |
place upstream (`sha256.sig`), but pinning the sha256 + serving the |
| 190 |
release URL over HTTPS is sufficient for MVP. |
| 191 |
|
| 192 |
### `efibootmgr` fails in QEMU/EDK2 by default |
| 193 |
|
| 194 |
`-drive if=pflash,format=raw,file=$NVRAM` from the |
| 195 |
`edk2-i386-vars.fd` template gives you EFI vars storage, but the |
| 196 |
firmware refuses some efibootmgr operations. Make the boot-entry |
| 197 |
registration best-effort: |
| 198 |
``` |
| 199 |
efibootmgr --create --disk $DISK --part 1 \ |
| 200 |
--label Mountaineer --loader '\EFI\BOOT\BOOTX64.EFI' || \ |
| 201 |
echo "warning: relying on \\EFI\\BOOT\\BOOTX64.EFI fallback path" |
| 202 |
``` |
| 203 |
The removable-media fallback works whether or not efibootmgr did. |
| 204 |
|
| 205 |
### Per-host kernel cmdline lives in `org.zfsbootmenu:commandline` |
| 206 |
|
| 207 |
ZBM reads the kernel command line for the dataset it boots from the |
| 208 |
`org.zfsbootmenu:commandline` ZFS property on that dataset. There is |
| 209 |
*also* a `KCL` baked into the ZBM EFI binary itself, but that controls |
| 210 |
ZBM's own kernel, not the chainloaded one. Per-install values: |
| 211 |
``` |
| 212 |
zfs set org.zfsbootmenu:commandline='ro quiet console=tty0 console=ttyS0,115200' rpool/ROOT/mountaineer |
| 213 |
``` |
| 214 |
The dual-console form means the kernel emits to both ttyS0 (VM serial, |
| 215 |
out-of-band management) and tty0 (physical display); init's controlling |
| 216 |
tty is whichever was listed last. Harmless on hardware without a serial |
| 217 |
port. |
| 218 |
|
| 219 |
## Iteration mechanics |
| 220 |
|
| 221 |
### Live-patch the installer instead of rebuilding |
| 222 |
|
| 223 |
When iterating on `installer/install.sh`, you don't need a full ISO |
| 224 |
rebuild — pipe the new script over ssh into the running live env: |
| 225 |
``` |
| 226 |
ssh ... 'cat > /usr/local/sbin/mountaineer-install && chmod +x /usr/local/sbin/mountaineer-install' < installer/install.sh |
| 227 |
``` |
| 228 |
The next `mountaineer-install` invocation picks it up. Only rebuild the |
| 229 |
ISO when you've reached a "real testable artifact" stopping point, or |
| 230 |
when the apkovl content itself (not the installer) changed. |
| 231 |
|
| 232 |
### Standalone-boot the installed disk |
| 233 |
|
| 234 |
After the installer reboots, the install-target VM by default boots |
| 235 |
back into the live ISO (boot-order keeps the CD-ROM first). To exercise |
| 236 |
the installed system's UEFI → ZBM → Mountaineer path: |
| 237 |
``` |
| 238 |
qemu-system-x86_64 \ |
| 239 |
-machine type=q35 -accel tcg,thread=multi -cpu max -smp 4 -m 4G \ |
| 240 |
-drive if=pflash,format=raw,readonly=on,file=/opt/homebrew/share/qemu/edk2-x86_64-code.fd \ |
| 241 |
-drive if=pflash,format=raw,file=.vms/nvram/installed_VARS.fd \ |
| 242 |
-drive file=.vms/disks/target.qcow2,if=virtio,format=qcow2 \ |
| 243 |
-serial file:.vms/logs/installed-boot-serial.log \ |
| 244 |
-netdev user,id=net0,hostfwd=tcp:127.0.0.1:2224-:22 \ |
| 245 |
-device virtio-net-pci,netdev=net0 \ |
| 246 |
-display none -monitor unix:.vms/sockets/installed-mon.sock,server,nowait \ |
| 247 |
-daemonize -vga virtio -device virtio-rng-pci |
| 248 |
``` |
| 249 |
A fresh `installed_VARS.fd` (copy of `edk2-i386-vars.fd`) is needed |
| 250 |
because efivars from the install-target run carry no useful state. |
| 251 |
|
| 252 |
### TCG timing on Apple Silicon |
| 253 |
|
| 254 |
x86_64-on-arm via TCG is the bottleneck. Rough numbers: |
| 255 |
- Builder VM first provision: ~5-10 min (cloud-init). |
| 256 |
- Kernel + modloop mksquashfs (cold cache): ~25-30 min. |
| 257 |
- apkovl + ISO assembly (kernel cached): ~3-5 min. |
| 258 |
- Boot the live ISO to sshd-ready: ~1-2 min. |
| 259 |
- Full install run end-to-end: ~5-8 min (mostly apk fetches + zfs pool/datasets). |
| 260 |
|
| 261 |
Don't `rm -rf .work/` on the builder unless you really mean it. The |
| 262 |
build script's auto-clean of `.work/apkovl_*` and `.work/image-*` is |
| 263 |
sufficient for normal iteration. |
| 264 |
|
| 265 |
### macOS host helpers worth installing |
| 266 |
|
| 267 |
``` |
| 268 |
brew install dosfstools mtools |
| 269 |
``` |
| 270 |
Without these, `vm-test-install.sh` can't stage the FAT-formatted |
| 271 |
install-config image — it falls back to the "live-patch via ssh" |
| 272 |
workflow, which works fine but is one more step per iteration. |
| 273 |
|
| 274 |
## Build numbering |
| 275 |
|
| 276 |
A build with timestamped-newer-than-expected ISO output is the only |
| 277 |
reliable signal that mkimage actually re-assembled the image. Exit code |
| 278 |
0 is *not* sufficient: mkimage will happily return 0 after silently |
| 279 |
reusing a stale cached section. Always check `ls -la out/*.iso` against |
| 280 |
the wall-clock time the build started. |
| 281 |
|
| 282 |
## Reproducibility audit (static, 2026-05-19) |
| 283 |
|
| 284 |
A read-through of `build.sh`, `tools/build-apkovl.sh`, |
| 285 |
`mkimg/genapkovl-mountaineer-server.sh`, and the Alpine `mkimg.iso.sh` |
| 286 |
contract, without actually rebuilding. The apkovl half is already |
| 287 |
defensive; the ISO half is where non-determinism likely lives. Findings |
| 288 |
below are sorted by suspected impact. |
| 289 |
|
| 290 |
### Apkovl tarball — looks reproducible |
| 291 |
|
| 292 |
`tools/build-apkovl.sh` already does the right things: |
| 293 |
|
| 294 |
- staged tree is rebuilt from `apkovl-src/` every time |
| 295 |
- `find ! -type l -exec touch -d '@0'` zeroes mtimes (with a portable |
| 296 |
fallback to `-t 197001010000.00`) |
| 297 |
- `tar --sort=name --owner=0 --group=0 --numeric-owner --mtime='@0' |
| 298 |
--format=ustar` produces stable archive order/metadata |
| 299 |
- `gzip --no-name -9` strips the gzip timestamp/filename |
| 300 |
- `SOURCE_DATE_EPOCH=0` is exported into tar's environment |
| 301 |
|
| 302 |
Two builds of `mountaineer-server-target-0.0.1-mvp.apkovl.tar.gz` from |
| 303 |
the same source tree on the same builder *should* be byte-identical |
| 304 |
already. The live apkovl absorbs the target apkovl as a payload, so |
| 305 |
that should follow. |
| 306 |
|
| 307 |
Confidence: high pending an actual diff. Easy first verification: |
| 308 |
|
| 309 |
./tools/build-apkovl.sh server-target /tmp/a.tar.gz |
| 310 |
./tools/build-apkovl.sh server-target /tmp/b.tar.gz |
| 311 |
sha256sum /tmp/a.tar.gz /tmp/b.tar.gz |
| 312 |
|
| 313 |
### ISO half — suspected non-determinism sources |
| 314 |
|
| 315 |
1. **`apk` repository pinning**: `build.sh` passes |
| 316 |
`--repository https://dl-cdn.alpinelinux.org/alpine/v3.23/main` |
| 317 |
(and community). v3.23 *branch* drifts as upstream backports |
| 318 |
roll. Two builds a week apart will fetch different package |
| 319 |
versions even with no source change. Fix: snapshot mirror, or |
| 320 |
pre-fetched apk cache committed alongside the build, or pin |
| 321 |
to a tagged aports commit + that day's main snapshot. |
| 322 |
|
| 323 |
2. **Aports HEAD**: `git clone --depth 1 https://gitlab.alpinelinux.org/alpine/aports.git` |
| 324 |
pulls current v3.23 head. Same drift as #1. Fix: clone with |
| 325 |
`--branch <tag>` once Alpine cuts a 3.23.x point release, or |
| 326 |
record the commit hash and re-checkout on every build. |
| 327 |
|
| 328 |
3. **mkimage `SOURCE_DATE_EPOCH`**: build.sh does not export it into |
| 329 |
the mkimage invocation. Many of the section scripts (squashfs, |
| 330 |
xorriso, mkinitfs) honor it if set. Fix: `SOURCE_DATE_EPOCH=0 |
| 331 |
sh scripts/mkimage.sh ...`. Cheap, almost certainly helps. |
| 332 |
|
| 333 |
4. **`mksquashfs` modloop**: Alpine's `mkimg.modloop.sh` calls |
| 334 |
`mksquashfs` without explicit determinism flags. Even with |
| 335 |
`SOURCE_DATE_EPOCH`, the filesystem inode order on the build host |
| 336 |
leaks into the image. Fix at upstream level: add `-no-exports |
| 337 |
-no-fragments -mkfs-time 0 -all-time 0`. We may need to patch |
| 338 |
the section script in the staged aports copy. |
| 339 |
|
| 340 |
5. **`xorriso` ISO9660 timestamps**: `mkimg.iso.sh` issues an |
| 341 |
`xorrisofs` invocation that, by default, embeds the wall-clock |
| 342 |
time as the volume creation date and per-file dates. Need to |
| 343 |
verify whether `--set-all-file-dates @0` and the `volume_date` |
| 344 |
knobs are already applied. If not, patch the section script. |
| 345 |
|
| 346 |
6. **`mformat` ESP volume serial**: the ESP image (FAT32 via |
| 347 |
`mformat`) is given a random volume serial on each invocation |
| 348 |
unless `-N <hex>` is passed. Cheap fix: `-N 0` everywhere it's |
| 349 |
called. |
| 350 |
|
| 351 |
7. **mkinitfs/ramdisk**: cpio archive ordering and any embedded |
| 352 |
modaliases. Modern mkinitfs honors `SOURCE_DATE_EPOCH`, so #3 |
| 353 |
above probably covers it. Verify with `cmp` after a build pair. |
| 354 |
|
| 355 |
8. **GRUB EFI image**: `grub-mkstandalone` / equivalent embeds a |
| 356 |
build timestamp inside the EFI binary by default. Pass |
| 357 |
`--modules=... -t 0` or rebuild from a deterministic source. |
| 358 |
|
| 359 |
Items 3–6 are the ones most worth attacking first — each is a |
| 360 |
single-line patch with a near-zero downside, and together they |
| 361 |
likely close the gap from "same source, different bytes" to |
| 362 |
"same source, same bytes". Items 1–2 are about reproducibility |
| 363 |
across machines/days and need package-pinning discipline. |
| 364 |
|
| 365 |
### Recommended next concrete step |
| 366 |
|
| 367 |
Run the apkovl-only test (build twice, compare) without spinning up |
| 368 |
the builder VM. That validates the apkovl half is already |
| 369 |
deterministic, narrows the search space to the ISO assembly, and |
| 370 |
costs <30 seconds on the host. Then bring up the builder VM and |
| 371 |
attempt a full ISO diff with `SOURCE_DATE_EPOCH=0` exported, before |
| 372 |
attacking individual section scripts. |
| 373 |
|
| 374 |
## Outstanding open issues (May 17 2026) |
| 375 |
|
| 376 |
- Reproducible-by-bytes builds — both builds-from-same-source and |
| 377 |
builds-from-different-checkouts. Best-effort today via |
| 378 |
`--sort=name`, `--mtime='@0'`, `--owner=0 --group=0`, |
| 379 |
`SOURCE_DATE_EPOCH=0`. Not byte-stable yet. |
| 380 |
- Vendor `vector` (the log shipper) from `vector.dev/releases` — STACK |
| 381 |
pick, not packaged in Alpine 3.23. |
| 382 |
- Hardware-test on Framework 13. The full installer path has been |
| 383 |
exercised in QEMU; the NVMe partition-naming branch |
| 384 |
(`installer/install.sh:171`) has been verified by code-reading only. |
| 385 |
|