# Mountaineer Install The defended spec for how Mountaineer goes from a bootable artifact to an installed, first-boot-ready host. Covers: the two ISOs and how they're built, the disk layout, the ZFS pool and dataset hierarchy, the bootloader, the apkovl, and the installer flow. Defends every opinionated call. Not in scope for this document: the full UX of the installer's prompts (the *behavior* the operator sees is documented in the first-boot route); upgrade discipline (separate document when the upgrade story opens); CI and artifact publishing (open in todo). Tool-level orientation is in `BUILD-TOOLS.md`; this document is the design those tools execute. ## Two artifacts, one source tree Mountaineer ships exactly two ISOs: **Server** (foundation + spicy picks + the compatibility contract) and **Sysadmin** (Server plus the graphical layer). They are built from one source tree by two `mkimage` profile scripts that share the Server-layer function and differ only in whether they invoke the graphical-additions function on top. The strict-subset relationship from PROFILES.md is enforced *structurally* at build time, not by convention. The Server profile is a function the build calls; the Sysadmin profile is a function that *calls Server first* and then layers additions. There is no path for a package to land in Server-but-not-in-Sysadmin (the build doesn't know how to do that), and no path for a Sysadmin package to silently absent itself from a Server build (the graphical-additions function isn't invoked for Server). This is principle 13 ("the Sysadmin operator and the Server operator are the same person") as a build-shape constraint. Both ISOs are **UEFI-only**. Legacy BIOS is not supported. Mountaineer's target hardware is anything made in roughly the last decade; the BIOS-compat layer in modern bootloaders is an ongoing-work tax for a vanishing audience, and ZFS-on-root with zfsbootmenu wants UEFI anyway. Operators with pre-2012 hardware are not who Mountaineer is for. The installer refuses to proceed if it does not detect EFI variables; the message names the reason. ## Disk layout Mountaineer installs to a single disk. Multi-disk pools (mirrors, raidz) are supported by ZFS but configured *after* install — the first-boot dataset is on one disk, and `zpool attach` brings mirrors online without reinstalling. This keeps the installer's surface small and matches how operators actually decide: pick a disk, install, then decide about redundancy. The single-disk layout is GPT with exactly two partitions: | Partition | Type | Size | Purpose | |-----------|------|------|---------| | 1 | EFI System Partition | 512 MiB | FAT32; holds zfsbootmenu, kernel images, initramfs | | 2 | Solaris Root | remainder | The ZFS pool | 512 MiB on the ESP is more than needed (zfsbootmenu plus two or three kernel/initramfs pairs fits in <100 MiB) but the cost of a too-small ESP is a botched kernel upgrade; the cost of an oversized ESP is invisible. The size is defended on operational-margin grounds, not optimization. The ZFS partition takes the rest. No swap partition — Mountaineer does not configure swap by default. ZFS's ARC management does not interact well with swap, and modern systems with enough RAM for their workload should not be swapping; systems without enough RAM should get more RAM rather than slower disk-backed RAM. Operators who genuinely need swap (memory-pressured workloads where slower-but-alive beats OOM-killed) can create a zvol-backed swap or a swapfile post-install; the installer does not bake this in. ## The ZFS pool One pool, named **`rpool`**. Mountaineer-canonical, matches existing OpenZFS-on-root convention, short enough to type, distinct enough to not collide with operator-created pools later (data pools get descriptive names: `tank`, `archive`, the workload's name). Pool creation options that ship as defaults: ``` zpool create -f -o ashift=12 \ -o autotrim=on \ -O compression=lz4 \ -O atime=off \ -O xattr=sa \ -O acltype=posixacl \ -O dnodesize=auto \ -O normalization=formD \ -O mountpoint=none \ -O canmount=off \ rpool ``` Each option defended: - `ashift=12` — 4 KiB sector alignment. The right answer for every disk made in roughly the last fifteen years; the wrong answer (ashift=9) is unrecoverable without pool re-creation. Default conservatively, the cost on legacy 512n disks is negligible. - `autotrim=on` — TRIM commands sent to SSDs in the background. On SSDs this preserves performance over time; on rust disks it's a no-op the kernel ignores. Set unconditionally because the installer does not always know what the disk is. - `compression=lz4` — fast, near-free CPU cost, typically 1.5–2× space win on real-world data. Considered: `zstd-1` is now the OpenZFS default-leaning recommendation and gives slightly better ratios; lz4 remains slightly faster and has a longer track record. The case for picking lz4 over zstd is conservatism in a defending-defaults document; the case for switching is real and this decision is reopenable when there is operational evidence either way. - `atime=off` — access-time updates are an ongoing write-amp tax for a feature almost nothing actually uses. `relatime` is a half-measure; `off` is the principled answer. - `xattr=sa`, `acltype=posixacl`, `dnodesize=auto`, `normalization=formD` — the modern OpenZFS recommended set for Linux. xattr=sa stores extended attributes in the inode rather than a hidden directory entry, halving the IO cost; posixacl matches Linux's userland; dnodesize=auto allows variable-sized inodes; formD normalizes Unicode filenames so identical-looking paths actually match. - `mountpoint=none`, `canmount=off` on the pool root — the pool itself does not mount anywhere; only child datasets do. Forces explicit per-dataset mount-point declarations. Encryption is **optional at install time** and dataset-level when enabled. The installer asks once: encrypt the root, yes or no. If yes, `rpool/ROOT` is created with `-o encryption=on -o keyformat=passphrase` and the operator types the passphrase at every boot (entered via zfsbootmenu). If no, no encryption is configured and the operator can enable it per-dataset later for specific data. Reason for the per-install choice: full-disk encryption is the right answer for laptops, often wrong for servers in physically secured racks where the boot interaction is a liability and the threat model is different. ## Dataset hierarchy The canonical layout the installer creates: | Dataset | Mountpoint | Special options | Why separate | |---------|------------|-----------------|--------------| | `rpool/ROOT` | none | `canmount=off` | Container for boot environments | | `rpool/ROOT/mountaineer` | `/` | — | The bootable root | | `rpool/home` | `/home` | — | User data; survives root reinstall via boot-environment swap | | `rpool/var` | `/var` | `canmount=off` | Container | | `rpool/var/log` | `/var/log` | — | Log-volume blowups don't fill the rest of the system | | `rpool/var/lib` | `/var/lib` | — | Service state | | `rpool/var/lib/containers` | `/var/lib/containers` | `recordsize=128k` | Podman storage; large recordsize matches image-layer access patterns | | `rpool/var/cache` | `/var/cache` | `sync=disabled` | Cache data; loss on crash is acceptable, write throughput matters | Other paths (`/etc`, `/usr`, `/opt`, `/srv`, `/boot/efi`) are on the root dataset or on the ESP. `/tmp` is `tmpfs` (RAM-backed); the installer writes the relevant `fstab` line. Boot environments — the practice of having multiple `rpool/ROOT/` datasets and being able to roll back to a snapshot or an older root at the bootloader — is a feature zfsbootmenu provides naturally. The installer creates only `rpool/ROOT/mountaineer` initially; operators (or the upgrade tooling, eventually) create snapshots and additional boot environments as needed. Datasets that look conspicuously missing: - `/usr` separate — Mountaineer is small enough that `/usr` lives on root; the `/usr` merge era is over and there is nothing to gain by re-separating. - `/root` — on root dataset; nothing it holds is large enough to warrant separation. - `/opt` — Mountaineer does not encourage `/opt`-style installs; STACK picks live in package-managed paths. - Application-data datasets (e.g., for a Postgres or a podman volume root) — these are operator-decided per-workload, not part of the install. Workloads that want their own dataset get one created at deploy time, typically as `rpool/` with workload-appropriate `recordsize`. A `reservation` of ~1 GiB is set on the root dataset (`rpool/ROOT/mountaineer`) so that a pool-full situation in some other dataset cannot lock the operator out of the root. ZFS's "pool full means writes hang" failure mode is real and the reservation is the canonical defense. ## ARC sizing ZFS's ARC (Adaptive Replacement Cache) defaults to consuming up to 50% of system RAM. That is the right answer for a file server and the wrong answer for a Mountaineer host running defined workloads where most RAM should go to the workload, not to the filesystem cache. The installer picks `zfs_arc_max` based on total system RAM detected at install time: | Total RAM | `zfs_arc_max` | |-----------|---------------| | ≤ 2 GiB | 256 MiB | | 2–4 GiB | 512 MiB | | 4–8 GiB | 1 GiB | | 8–16 GiB | 2 GiB | | 16–32 GiB | 4 GiB | | > 32 GiB | 12.5% of RAM, capped at 16 GiB | Written to `/etc/modprobe.d/zfs.conf` as `options zfs zfs_arc_max=`. The value is conservative — better to leave RAM for workloads and let the OS page cache pick up the slack than to have ZFS evict workload memory under cache pressure. Operators with file-server-shaped workloads can raise this manually post-install; the installer's default favors the median Mountaineer host. The installer prints the value it picks and the reasoning, so the operator who disagrees can override before continuing. ## Bootloader: zfsbootmenu The ESP holds **zfsbootmenu** (ZBM) as the bootloader. ZBM is a small Linux kernel + initramfs whose only job is: import ZFS pools, present the available `rpool/ROOT/*` datasets and their snapshots, let the operator pick one, chainload into it. It replaces the GRUB-on-ZFS configuration dance — which involves keeping GRUB's ZFS support synchronized with the running kernel's ZFS module across upgrades — with a tool designed for ZFS from the start. The killer feature: at boot, ZBM can roll back to a snapshot. A bad upgrade boots into ZBM, the operator picks "boot from the snapshot taken before the upgrade," the system comes up on the older root, the bad upgrade is gone. This is what makes ZFS-on-root operationally honest rather than just a checkbox. What ships on the ESP: - `/EFI/BOOT/BOOTX64.EFI` — the zfsbootmenu EFI binary. - `/EFI/zfsbootmenu/vmlinuz-bootmenu` — the ZBM kernel. - `/EFI/zfsbootmenu/initramfs-bootmenu.img` — the ZBM initramfs (includes ZFS, the menu UI, encryption-passphrase handling if enabled). That's the entire ESP. No Mountaineer kernel on the ESP — kernels live in the root dataset at `/boot/`, and ZBM mounts the root dataset to find them. This is what enables the snapshot-boot feature: the snapshot contains its own kernel and initramfs, so booting an old snapshot boots the old kernel that goes with it. Mountaineer's installer uses upstream zfsbootmenu binaries (signed by the ZBM project) when available, or builds them locally as part of the installer environment. Either path is documented; the default is whichever the live ISO ships. Rejected: - **GRUB.** Functional with ZFS but only via a build of GRUB with ZFS support, and the support is fragile across feature-flag upgrades on the pool. The kind of working-most-of-the-time tool Mountaineer refuses to default to. - **systemd-boot.** Excluded by the no-systemd posture. - **rEFInd.** General-purpose UEFI boot manager; works fine but does not understand ZFS, so it cannot do snapshot-boot. The whole reason for picking a ZFS-aware bootloader. - **Direct EFI stub boot (kernel-as-EFI-binary).** Bypasses a bootloader entirely; loses the menu, loses snapshot-boot, gains nothing Mountaineer wants. ## The apkovl Mountaineer ships per-install configuration via Alpine's apkovl mechanism: a `.apkovl.tar.gz` whose contents are extracted onto the root filesystem at install time. The apkovl is the *only* way Mountaineer-canonical configuration reaches the installed system. The ISO itself is generic; the apkovl is what makes it Mountaineer-shaped. This is a deliberate split. The ISO is the kernel + apk + a small installer environment; it would be the same ISO whether or not Mountaineer has hardened sshd, picked nftables, defended unbound, etc. The apkovl carries all the spec material from STACK and the routes into the installed system. Rebuilding the apkovl with different configs (a fleet operator who wants their own SSH key baked in, a custom Caddyfile, a non-default Nebula lighthouse) does not require rebuilding the ISO. What the canonical Mountaineer apkovl contains, organized by path: ``` etc/ ├── apk/ │ ├── world # the declarative package list per profile │ └── repositories # Alpine community + Mountaineer overlay ├── ssh/ │ └── sshd_config # hardened (no passwords, no root, etc.) ├── nftables.d/ │ └── 00-baseline.nft # the first-boot baseline ruleset ├── unbound/ │ └── unbound.conf # local validating resolver, DoT upstreams ├── chrony/ │ └── chrony.conf # public NTP pool, iburst ├── caddy/ │ ├── Caddyfile # one-line import of Caddyfile.d/* │ └── Caddyfile.d/.keep # empty drop-in directory ├── s6-rc/ │ └── source/ # service tree skeleton (sshd, chrony, unbound, nftables, ...) ├── nebula/ │ └── (empty) # populated by `sysop mesh join` ├── msmtprc # fail-loud default; operator points it ├── hostname # placeholder; rewritten at first-boot └── modprobe.d/ └── zfs.conf # installer writes zfs_arc_max here usr/ └── local/ └── sbin/ # Mountaineer-authored install-time helpers ``` For the Sysadmin profile, the apkovl additionally carries `/etc/river/`, `/etc/foot/`, `/etc/yambar/`, `/etc/swaylock/`, etc. — the graphical layer's configs. What the apkovl deliberately does **not** carry: - SSH host keys. Generated per-host at install time; baking them into the apkovl would mean every install of that apkovl has the same identity, which is a real security failure. - Machine ID. Per-host, generated at install. - The operator's authorized_keys. Prompted for at first-boot, written then. - Nebula host certs. Issued by the operator's CA after install. - Any age-encrypted material. The host has no age key yet at apkovl-extraction time. - The host's network configuration. Operator-supplied or DHCP-derived at install. The split is principled: anything that should be **identical across every Mountaineer host of a given profile** is in the apkovl. Anything that is **per-host identity, generated freshly, or operator-supplied** is generated or prompted-for at install time or first-boot. ### apkovl source tree The apkovl tarball is built from a Mountaineer-authored source tree, not assembled by hand. The source tree: ``` apkovl-src/ ├── common/ # included in every Mountaineer apkovl │ └── ├── server/ # Server-profile additions │ └── ├── sysadmin/ # Sysadmin-profile additions (graphical) │ └── └── manifest.toml # per-profile world file, hooks, metadata ``` The builder takes `--profile {server,sysadmin}`, merges `common/` plus the chosen profile directory, applies any computed substitutions from `manifest.toml` (kernel version pinning, repository URL, signing key paths), and emits `mountaineer--.apkovl.tar.gz`. Same builder, two outputs, strict-subset relationship preserved by construction (a path in `server/` cannot fail to also be in the Sysadmin output because Sysadmin starts from Server, not from common). The builder is Mountaineer-authored — likely Rust given the existing `sysop` direction, but the language decision is not load-bearing for this document. What matters is that it is **reproducible** (same source tree, same builder version → bit-identical tarball) and **declarative** (the source tree is the spec; there is no "run the system, then capture the diff" step in the canonical build path). ## The installer The installer is a Mountaineer-authored program (shell + a small ratatui-shaped helper binary for the interactive prompts) that runs in the live ISO environment. It is *not* `setup-alpine` and does *not* call `setup-disk`; those tools assume a different disk layout and different bootloader and are easier to replace than to wrap. The installer's flow, in order: 1. **Refuse legacy BIOS.** Check for EFI variables; if absent, print the reason and exit. No prompt to override — Mountaineer is UEFI-only and the installer says so. 2. **Pick a disk.** List candidate disks via `lsblk`, exclude removable / readonly / the installer medium itself, present a ratatui picker with model, size, and current partition table summary. Confirm wipe. 3. **Optional encryption.** One question: encrypt the root, yes or no. If yes, prompt for passphrase twice; the installer holds it in memory only until the pool is created. 4. **Partition.** GPT, ESP (512 MiB FAT32 labeled `MTN-ESP`), Solaris-root partition (remainder). `parted` or `sgdisk` — implementation detail, output is the same. 5. **Create the pool.** `zpool create` with the option set above; on the partition created in step 4. If encryption was requested, `-o encryption=on -O encryption=on` on `rpool/ROOT` (so child datasets inherit). 6. **Create datasets.** The full hierarchy from the table above, with mountpoints and per-dataset options. Mount under `/mnt` in the installer environment. 7. **Format and mount the ESP.** `mkfs.vfat -F32 -n MTN-ESP `, mount at `/mnt/boot/efi`. 8. **Compute ARC size.** Read `/proc/meminfo`, apply the table, print the chosen value and reasoning. Operator can override now or accept. 9. **Install the base.** `apk --root /mnt --initdb add` the world-file contents (read from the apkovl's `etc/apk/world`). This populates `/mnt` with the package set. 10. **Unpack the apkovl.** Extract `mountaineer--.apkovl.tar.gz` over `/mnt`. After this step the installed root has the canonical Mountaineer configuration. 11. **Write ARC config.** The chosen `zfs_arc_max` lands in `/mnt/etc/modprobe.d/zfs.conf`. 12. **Install zfsbootmenu.** Copy ZBM EFI binary, kernel, and initramfs to `/mnt/boot/efi/EFI/`. Create the UEFI boot entry via `efibootmgr`. 13. **Generate per-host identity.** SSH host keys via `ssh-keygen -A`; machine-id via `systemd-machine-id-setup` equivalent (busybox: just `dd if=/dev/urandom bs=16 count=1 | hexdump`). The host's age recipient is *not* generated here yet — deferred along with the `sysop id` age slot. 14. **First-boot prompts.** Hostname, operator SSH public key (paste or fetch from URL), network configuration (DHCP / static / wifi if Sysadmin), root password (with the warning that password login is disabled in sshd; this is for console access only), and on Sysadmin: browser pick. Each answer is written to the relevant config file under `/mnt/etc/`. 15. **Compile the s6-rc tree.** `s6-rc-compile /mnt/etc/s6-rc/compiled /mnt/etc/s6-rc/source`. The first boot starts services from this compiled tree without further intervention. 16. **Confirm.** Print the install summary (disk, pool, datasets, hostname, network, kernel, ZBM version, apkovl version). Operator confirms or cancels. 17. **Unmount, export, reboot.** `umount -R /mnt`, `zpool export rpool`, `reboot`. The next boot lands in zfsbootmenu, then in the installed Mountaineer. Steps 2–4 and 8 and 14 are interactive. Everything else is non-interactive once the operator confirms in step 16. Non-interactive install (for CI, fleet provisioning) is supported by passing a `--config ` flag with all answers pre-populated; this is how Mountaineer hosts get built unattended. ## What the installer does *not* do Deliberate exclusions, each defended: - **Create users beyond root.** Operator user accounts are first-boot route work, not installer work. The installer leaves a root account with the password the operator set (used only for console) and the SSH host keys; everything else is post-install. - **Configure backups.** The restic restore-drill route is the canonical setup. The installer does not write backup configs or schedule cron jobs — that is an explicit operator decision per host. - **Join a mesh.** The Nebula mesh route is post-install. The apkovl ships the Nebula daemon and config skeleton but does not auto-join anything. - **Issue TLS certs.** Caddy ships ready to issue certs but does not request any until a Caddyfile site block exists, which is workload-time work. - **Apply per-host firewall changes beyond the baseline.** The baseline drops inbound, allows SSH, allows ICMP, rate-limits SSH. That is it. Workload-specific allow rules are first-boot or workload-deploy time. - **Generate a Nebula CA or any age keys.** Those are operator-scope concerns and belong to `sysop id` when it lands. The principle is: the installer produces a host that is **bootable, reachable, and safe** — and nothing else. Everything that distinguishes one Mountaineer host's *purpose* from another's is post-install work. The installer is not a configuration-management tool, and Mountaineer does not ship one. ## v0.1 install dogfood gate The install-mechanics work is done, for v0.1 purposes, when: 1. Both ISOs (`mountaineer-server-0.1.iso`, `mountaineer-sysadmin-0.1.iso`) build reproducibly from the source tree. 2. Either ISO boots in a UEFI VM and an x86_64 physical machine made in the last decade. 3. The installer completes end-to-end on a clean disk in under 15 minutes of real time. 4. The installed host boots, comes up on the network, accepts the operator's SSH key, and passes every step of the first-boot route without modification. 5. The Caddy + podman route works on a fresh Server install with no installer-side changes. 6. The Nebula mesh route works between a fresh Server install (lighthouse) and a fresh Sysadmin install (laptop). 7. The restic restore drill route works (any backend). 8. A reboot leaves the system in the same state — every service started, no manual intervention, no degradation. When all eight are true, the install-mechanics track is closed for v0.1 and Mountaineer can be installed by anyone with the ISO. Mountaineer's own infrastructure (build server, docs site, package mirror) being hosted on Mountaineer running on s6-rc is a separate dogfood gate (already in todo) and is what earns the *release* of v0.1 — but the install gate is the necessary precondition. ## Open install-mechanics decisions Recorded here so they don't get lost; resolved as the work progresses. - **Where the installer's helper binary lives.** Same Rust crate as `sysop`, separate crate, or shell-only with no helper binary? Affects build dependency surface for the installer ISO. - **The kernel/zfs-lts pairing on upgrade.** apk's transaction model handles this for installed systems; the question is whether the installer environment pins kernel-and-zfs versions in lock-step (yes, probably) and how it surfaces a pairing failure if one occurs. - **ESP redundancy.** A single ESP on a single disk is a single point of failure for boot. Mirror-ESP-to-second-disk on multi-disk hosts is operationally desirable; not in v0.1 install scope, but the layout should not paint us into a corner. - **Installer logs.** Where they write to during install (tmpfs only? piped to a serial console if available? saved to the new root after step 10?). Affects post-mortem debugging of failed installs. - **The non-interactive `--config` schema.** What the config file looks like, where defaults come from, how it interacts with future `sysop` flag conventions. - **Signing.** The ISO is signed by what key, the apkovl is signed by what key, the overlay-repository APKs are signed by what key. Probably three different keys; touches the `sysop id` work but the build side needs an answer first.