Skip to main content

max / mountaineer

Initialize Mountaineer monorepo: spec into docs/, build pipeline into build/ Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author: Max J. <87768334+MaxJMath@users.noreply.github.com> · 2026-05-18 03:10 UTC
Commit: 2b96943159debad8446549bff833e94c4f89003a
63 files changed, +4891 insertions, -0 deletions
@@ -0,0 +1,9 @@
1 + # Build outputs
2 + out/
3 + .work/
4 +
5 + # VM working state (disks, ISOs, NVRAM, snapshots)
6 + .vms/
7 +
8 + # Shallow aports clone for mkimage
9 + .aports/
@@ -0,0 +1,195 @@
1 + # mountaineer-build
2 +
3 + The implementation side of Mountaineer. Spec lives in `../mountaineer/`;
4 + this repo builds artifacts (apkovl, ISO) from that spec.
5 +
6 + **Status: MVP, installer end-to-end in QEMU.** 660 MB ISO at
7 + `out/mountaineer-server-v3.23-x86_64.iso` boots in UEFI QEMU. The
8 + installer runs all 17 steps against a clean disk — ZFS pool, dataset
9 + layout, base packages, apkovl unpack, ZBM (zfsbootmenu) install,
10 + per-host identity, finalize/reboot — and the installed disk boots
11 + standalone via ZBM. Ready for Framework 13 hardware verification.
12 +
13 + Build/install history and rediscovered traps live in
14 + [`build_notes.md`](build_notes.md). Open work tracked in
15 + `../mountaineer/todo.md`.
16 +
17 + Anything marked `TODO-verify` or `TODO` in the source needs real
18 + hardware testing or upstream-vendoring to commit to a final shape.
19 +
20 + ## Scope (MVP)
21 +
22 + What the MVP aims for, in plain terms: a Mountaineer Server ISO that boots
23 + in a UEFI VM, runs an installer, lays down ZFS-on-root, unpacks the
24 + Mountaineer apkovl, reboots, and passes every step of the first-boot route
25 + from the spec.
26 +
27 + What is deliberately **not** in MVP scope:
28 +
29 + - **Sysadmin profile / graphical layer.** Only Server is built. Sysadmin
30 + comes when Server is stable.
31 + - **The `sysop` CLI.** Routes use canonical tools directly (`nft -f`,
32 + `s6-rc -u change`, etc.) until `sysop` lands.
33 + - **Reproducible builds, bit-for-bit.** The apkovl builder is
34 + reproducible-best-effort (same source + same builder → same bytes).
35 + Bit-identical across builder versions is a v0.2 goal.
36 + - **Caddy / Nebula / podman / Vector running by default.** They're
37 + installed but not started; the operator configures and starts them per
38 + the routes in `../mountaineer/ROUTES.md`.
39 + - **Mountaineer overlay apk repository.** Until we have one, everything
40 + comes from Alpine repos. Mountaineer-authored packages (`sysop` when it
41 + ships, the installer eventually) are TBD.
42 +
43 + ## Layout
44 +
45 + ```
46 + mountaineer-build/
47 + ├── apkovl-src/ # declarative source for the apkovl
48 + │ ├── common/ # files shipped in every Mountaineer build
49 + │ ├── manifest.toml # per-profile metadata (sources, version)
50 + │ └── README.md # what lives where, how to verify reproducibility
51 + ├── tools/
52 + │ └── build-apkovl.sh # apkovl-src/ → reproducible tarball
53 + ├── mkimg/
54 + │ └── mkimg.mountaineer-server.sh # mkimage profile (drop into aports/scripts/)
55 + ├── installer/
56 + │ └── install.sh # 17-step shell+dialog installer
57 + ├── build.sh # top-level: apkovl + ISO
58 + └── README.md # this file
59 + ```
60 +
61 + ## Building
62 +
63 + ### Quickest path: Alpine host
64 +
65 + On any host running Alpine (physical, VM, container), in this directory:
66 +
67 + ```sh
68 + apk add alpine-sdk abuild xorriso squashfs-tools mkinitfs grub-efi git tar
69 + ./build.sh
70 + ```
71 +
72 + Output:
73 +
74 + - `out/mountaineer-server-<version>.apkovl.tar.gz` — the apkovl
75 + - `out/mountaineer-server-*.iso` — the bootable installer ISO
76 +
77 + The first run clones a shallow copy of Alpine's `aports` repo into
78 + `.aports/` (≈100 MiB). Subsequent runs reuse it.
79 +
80 + ### From a Mac (the dev loop)
81 +
82 + The canonical Mac iteration loop uses qemu to run a long-lived Alpine
83 + builder VM. See `tools/qemu/README.md` for the full walkthrough; the
84 + short version, once `brew install qemu` is done:
85 +
86 + ```sh
87 + ./tools/qemu/vm-up.sh --provision # first time only: download ISO + setup-alpine
88 + ./tools/qemu/vm-seed-builder.sh # install build deps in the builder VM
89 + ./tools/qemu/vm-build.sh # rsync repo + run ./build.sh inside builder
90 + ./tools/qemu/vm-test-install.sh # boot the new ISO in a disposable target VM
91 + ```
92 +
93 + Apple Silicon caveat: x86_64 guests run via TCG emulation (no HVF for
94 + cross-arch), so a full build cycle takes ~10-20 minutes. On Intel Macs
95 + HVF makes it ~2-3 minutes.
96 +
97 + ### From a non-Alpine Linux workstation
98 +
99 + Run the build inside an Alpine container via podman:
100 +
101 + ```sh
102 + podman run --rm -it --privileged \
103 + -v "$PWD":/build \
104 + -w /build \
105 + docker.io/alpine:edge \
106 + sh -c 'apk add alpine-sdk abuild xorriso squashfs-tools mkinitfs grub-efi git tar && ./build.sh'
107 + ```
108 +
109 + `--privileged` is required for `mkfs`/`mksquashfs`/loop-device work inside
110 + the container. Outputs land on the host via the bind mount.
111 +
112 + We do not use Nix, devbox, or any other ecosystem-replacing tooling.
113 + Mountaineer's build environment is apk-native (see
114 + `../mountaineer/BUILD-TOOLS.md`).
115 +
116 + ### Just the apkovl
117 +
118 + `./tools/build-apkovl.sh server` produces only the apkovl tarball (no ISO).
119 + Useful for iterating on `apkovl-src/` content without rebuilding the ISO
120 + every time. Requires GNU tar; the script auto-detects and prints the fix
121 + if it's missing.
122 +
123 + ## Testing the installer
124 +
125 + The installer's `--dry-run` flag walks every prompt and prints every
126 + destructive command instead of executing it. Run it inside a VM that has
127 + `dialog` installed to exercise the prompt flow without losing data:
128 +
129 + ```sh
130 + ./installer/install.sh --dry-run
131 + ```
132 +
133 + End-to-end testing requires a fresh UEFI VM with a blank disk. The
134 + canonical loop:
135 +
136 + 1. `./build.sh` to produce the ISO.
137 + 2. Boot the ISO in a UEFI VM (qemu, UTM, VMware, etc.) with at least
138 + 2 GiB RAM and a 20 GiB blank disk.
139 + 3. Log in to the live environment, run `mountaineer-install` (or the path
140 + to `install.sh` if not yet installed in PATH on the live image).
141 + 4. Walk through the prompts; complete the install; reboot.
142 + 5. Verify the installed system passes the first-boot route in
143 + `../mountaineer/ROUTES.md`.
144 +
145 + The v0.1 dogfood gate is eight criteria documented in
146 + `../mountaineer/INSTALL.md`.
147 +
148 + ## Known gaps (recorded here so we don't lose them)
149 +
150 + These are the load-bearing things still TODO:
151 +
152 + - **zfsbootmenu install path.** Installer step 12 has a placeholder. Need
153 + to decide: apk-package ZBM if Alpine ships it, or fetch upstream
154 + release signed by the ZBM project. Affects build reproducibility and
155 + the signing-trust story.
156 + - **mkimg profile bootloader.** The profile script declares
157 + `boot_efi="grub-efi"` for the live ISO; need to verify mkimage's
158 + expected token names against the current aports tree.
159 + - **OpenRC vs s6-rc on the live ISO.** The installed system uses s6-rc.
160 + The live ISO inherits Alpine's default (OpenRC) for now; the installer
161 + itself doesn't care which init it ran under. Migrating the live ISO to
162 + s6-rc is a v0.2 cleanup.
163 + - **Vector packaging on Alpine.** The world file lists `vector`;
164 + TODO-verify the package name and repo placement before the first build
165 + is attempted.
166 + - **Non-interactive install config schema.** `--config <file>` is in the
167 + spec but the schema doesn't exist yet.
168 + - **Installer log routing.** Right now the installer logs to stderr;
169 + decide whether to also tee to tmpfs and copy into the installed root
170 + before reboot for post-mortem debugging.
171 + - **Signing.** ISO and apkovl signing keys, lifecycle, where they live.
172 + Open in `../mountaineer/INSTALL.md` and `../mountaineer/todo.md`.
173 +
174 + When any of these get resolved, the corresponding `TODO-verify` or
175 + `TODO` comment in the source code gets removed in the same commit that
176 + resolves it.
177 +
178 + ## Relationship to the spec
179 +
180 + This repo implements `../mountaineer/`. When the two disagree:
181 +
182 + - If the spec is right and the implementation is wrong, fix the code.
183 + - If the implementation is right and the spec is wrong, fix the spec —
184 + but only after the implementation has actually been tested. A spec
185 + changed to match unverified code is worse than a spec at odds with it.
186 +
187 + The spec docs that matter most for implementation work:
188 +
189 + - `STACK.md` — what packages, what configs, what defaults.
190 + - `INSTALL.md` — disk layout, dataset hierarchy, ARC sizing, installer
191 + flow, dogfood gate.
192 + - `ROUTES.md` — the end-state UX the installed system must support.
193 + - `BUILD-TOOLS.md` — the toolchain orientation.
194 + - `MANIFESTO.md` — the principles to consult when a question doesn't
195 + have an obvious answer.
@@ -0,0 +1,44 @@
1 + # apkovl source tree
2 +
3 + Declarative source for Mountaineer's apkovl. The builder
4 + (`tools/build-apkovl.sh`) reads this directory plus `manifest.toml` and emits
5 + a reproducible `mountaineer-<profile>-<version>.apkovl.tar.gz`.
6 +
7 + Layout:
8 +
9 + - `common/` — files included in every Mountaineer apkovl. Paths under
10 + `common/` mirror the paths-as-they-appear-in-`/` on the installed system.
11 + - `server/` — Server-profile additions. (MVP: empty — Server config lives
12 + in `common/` because Server is the only MVP profile.)
13 + - `sysadmin/` — Sysadmin-profile additions (graphical layer). Not yet
14 + populated; landed when the graphical track opens.
15 + - `manifest.toml` — per-profile world file pointers, version metadata, any
16 + build-time substitutions.
17 +
18 + ## What lives here vs. what is generated
19 +
20 + In the apkovl (this directory): identical-across-hosts material. Hardened
21 + sshd config, nftables baseline, unbound and chrony configs, s6-rc service
22 + tree, Caddy and unbound drop-in directory shapes, the canonical package
23 + world.
24 +
25 + **Not** in the apkovl, generated at install or first-boot time: SSH host
26 + keys, machine-id, hostname, network configuration, operator
27 + `authorized_keys`, the `zfs_arc_max` value (a placeholder file lives here;
28 + the installer rewrites it), Nebula host certs, age host keys.
29 +
30 + See `INSTALL.md` in the spec repo for the full split.
31 +
32 + ## Verifying changes
33 +
34 + The builder is reproducible-best-effort: same source tree + same builder
35 + version → same tarball bytes. To verify after editing:
36 +
37 + ```sh
38 + ./tools/build-apkovl.sh server > /tmp/a.apkovl.tar.gz
39 + ./tools/build-apkovl.sh server > /tmp/b.apkovl.tar.gz
40 + sha256sum /tmp/[ab].apkovl.tar.gz
41 + ```
42 +
43 + The two checksums should match. If they don't, the builder is leaking
44 + non-determinism (timestamps, file ordering, uid/gid) and that is a bug.
@@ -0,0 +1,17 @@
1 + # Mountaineer apk repositories
2 + #
3 + # Pinned to Alpine v3.23 (current stable as of 2026-05; supported until
4 + # 2027-11). Both main and community are in support for v3.23 — community
5 + # is where zfs, caddy, nebula, helix, podman, and most spicy picks live.
6 + #
7 + # To track edge instead, replace the v3.23 lines with edge/main, edge/community,
8 + # edge/testing. Mountaineer will bump this when the next Alpine stable lands
9 + # (v3.24 expected ~May 2026) and we have verified the build passes against it.
10 + #
11 + # The Mountaineer overlay repository is commented out until the overlay
12 + # infrastructure ships — see todo.md "Mountaineer overlay repository".
13 +
14 + https://dl-cdn.alpinelinux.org/alpine/v3.23/main
15 + https://dl-cdn.alpinelinux.org/alpine/v3.23/community
16 +
17 + # https://pkg.mountaineer.example.org/v0/main
@@ -0,0 +1,120 @@
1 + # Mountaineer Server MVP — declarative package set
2 + #
3 + # One package per line. Comments document the STACK source for each group.
4 + # Verify-on-Alpine items (marked TODO-verify) are picks I expect to be in the
5 + # stated repository but have not confirmed against a live Alpine mirror.
6 + # When the first build runs on Alpine, any missing package surfaces here as
7 + # a hard error — which is the right failure mode.
8 +
9 + # Kernel and base
10 + linux-lts
11 + linux-firmware-none
12 + alpine-baselayout
13 + busybox
14 + busybox-suid
15 + busybox-openrc
16 + musl
17 + musl-utils
18 + ca-certificates
19 + ca-certificates-bundle
20 + tzdata
21 +
22 + # Package management
23 + apk-tools
24 +
25 + # Init: s6 + s6-rc (STACK spicy pick — replaces OpenRC)
26 + s6
27 + s6-rc
28 + s6-linux-init
29 + execline
30 +
31 + # Filesystem: ZFS on root
32 + # `zfs` provides userland (zpool, zfs commands); `zfs-lts` is the kernel
33 + # module for linux-lts. `zfs-libs` is auto-pulled. There is no separate
34 + # `zfs-utils` package in Alpine — the userland lives in `zfs`.
35 + zfs
36 + zfs-lts
37 + # zfsbootmenu is NOT packaged in Alpine 3.23 (main, community, or
38 + # edge/testing). The installer fetches the signed EFI binary from
39 + # zbm-dev.github.io/zfsbootmenu/releases and installs it to the ESP at
40 + # install time. Not present in the installed-system package set.
41 + efibootmgr
42 + e2fsprogs
43 + dosfstools
44 + mkinitfs
45 + sgdisk
46 +
47 + # Network configuration
48 + ifupdown-ng
49 + ifupdown-ng-wifi # harmless on Server; iwd is laptop-only
50 + dhcpcd
51 +
52 + # Time and DNS
53 + chrony
54 + unbound
55 + unbound-libs
56 +
57 + # SSH
58 + openssh-server
59 + openssh-client
60 + openssh-keygen
61 +
62 + # Firewall
63 + nftables
64 +
65 + # Web ingress
66 + caddy
67 +
68 + # Secrets and backup
69 + age
70 + sops
71 + restic
72 +
73 + # Outbound mail
74 + msmtp
75 + mailx
76 +
77 + # Logs
78 + # vector is NOT packaged in Alpine 3.23 (main, community, or edge/testing).
79 + # MVP ships without it — logs stay on disk via Alpine's default logger
80 + # (busybox syslogd) and Caddy/podman per-service log files. When the
81 + # Mountaineer overlay repo lands, vector ships as a vendored static
82 + # binary from vector.dev/releases. STACK pick stands; packaging path
83 + # changes from "apk add" to "ship in our overlay" for the time being.
84 +
85 + # Containers
86 + podman
87 + crun
88 + fuse-overlayfs
89 + slirp4netns
90 +
91 + # Overlay mesh
92 + nebula
93 +
94 + # Interactive userland (STACK)
95 + ripgrep
96 + fd
97 + sd
98 + bat
99 + eza
100 + dust
101 + procs
102 + bottom
103 + zoxide
104 + lnav
105 + starship
106 +
107 + # Spicy picks: editor, multiplexer, shell
108 + helix
109 + zellij
110 + nushell
111 +
112 + # Build/install conveniences kept on the system
113 + git
114 + curl
115 + wget
116 + jq
117 + yq
118 +
119 + # Installer UI (dialog-based prompts in mountaineer-install)
120 + dialog
@@ -0,0 +1,5 @@
1 + # Mountaineer Caddyfile — top-level imports the drop-in directory.
2 + # Per-site config lives as one .caddy file per site in /etc/caddy/Caddyfile.d/.
3 + # Adding a site is one file; removing it is `rm`.
4 +
5 + import /etc/caddy/Caddyfile.d/*.caddy
@@ -0,0 +1,33 @@
1 + # Mountaineer chrony: NTP client pointing at the public pool with iburst.
2 + # Operators with an organizational time source replace the pool line with
3 + # their NTP server(s) — that override goes in
4 + # /etc/chrony/chrony.conf.d/* if Alpine's chrony build supports a drop-in,
5 + # or by editing this file directly otherwise (TODO-verify on Alpine).
6 +
7 + pool 2.pool.ntp.org iburst maxsources 4
8 +
9 + # Initial step: allow large clock corrections at startup, only once.
10 + # After the initial sync, chrony slews the clock smoothly.
11 + initstepslew 30 2.pool.ntp.org
12 +
13 + # Drift file lives on persistent storage so re-sync after reboot is faster.
14 + driftfile /var/lib/chrony/drift
15 +
16 + # Allow the system clock to be stepped if the offset exceeds 1 second
17 + # during the first three updates only. Production-shaped: a clock that
18 + # slews continuously is more honest than one that jumps.
19 + makestep 1.0 3
20 +
21 + # Enable kernel synchronization of the real-time clock.
22 + rtcsync
23 +
24 + # Use only authenticated NTP if a key file exists. By default no NTS or
25 + # symmetric keys are configured; operators who need them add the relevant
26 + # server/key lines themselves.
27 +
28 + # Log level: terse. Sync events of interest land in /var/log/chrony/.
29 + logdir /var/log/chrony
30 +
31 + # Refuse to act as a server.
32 + port 0
33 + cmdport 0
@@ -0,0 +1 @@
1 + mountaineer
@@ -0,0 +1,2 @@
1 + Mountaineer \r \l
2 +
@@ -0,0 +1,8 @@
1 + # Mountaineer: zfs_arc_max placeholder.
2 + #
3 + # The installer overwrites this file at install time with a value computed
4 + # from total system RAM (see INSTALL.md ARC sizing table). This placeholder
5 + # only exists so the path is present in the apkovl; the real value is
6 + # determined per-host.
7 + #
8 + # options zfs zfs_arc_max=<bytes>
@@ -0,0 +1,5 @@
1 + Welcome to Mountaineer.
2 +
3 + Documentation: see /usr/share/doc/mountaineer/ (MANIFESTO, STACK, ROUTES).
4 + First-boot checklist: ROUTES.md "First boot" section.
5 +
@@ -0,0 +1,23 @@
1 + # Mountaineer msmtp: fail-loud default.
2 + #
3 + # msmtp is the outbound-only mail relay (STACK.md). Until the operator
4 + # points the `default` account at a real SMTP provider, every send fails
5 + # with a clear error rather than silently dropping mail.
6 + #
7 + # Replace this file (or use msmtprc.d if Alpine's msmtp build supports it
8 + # — TODO-verify) with real account credentials. Secrets should not live in
9 + # this file directly; reference them via `passwordeval` pointing at a
10 + # script that reads from age-encrypted storage.
11 +
12 + defaults
13 + auth on
14 + tls on
15 + tls_trust_file /etc/ssl/certs/ca-certificates.crt
16 + logfile /var/log/msmtp.log
17 +
18 + account default
19 + host unconfigured.invalid
20 + port 587
21 + from root@unconfigured.invalid
22 + user unconfigured
23 + passwordeval "echo 'msmtp is unconfigured — edit /etc/msmtprc' >&2; exit 1"
@@ -0,0 +1,6 @@
1 + # Mountaineer top-level nftables config: include every .nft file from
2 + # /etc/nftables.d/ in lexical order. The baseline (00-baseline.nft) flushes
3 + # the ruleset and installs the core policy; later files (10-*, 20-*, etc.)
4 + # add workload-specific allow rules without re-flushing.
5 +
6 + include "/etc/nftables.d/*.nft"
@@ -0,0 +1,74 @@
1 + # Mountaineer baseline firewall — IPv4 and IPv6
2 + #
3 + # Drop inbound by default, allow established/related, allow SSH with rate
4 + # limiting, allow ICMP/ICMPv6 sane filtering, log dropped.
5 + #
6 + # Per-workload allow rules belong in additional files in /etc/nftables.d/
7 + # (e.g. 10-web.nft for Caddy ports). The baseline is what every Mountaineer
8 + # host runs and is loaded by the s6-rc nftables service at boot.
9 +
10 + flush ruleset
11 +
12 + table inet filter {
13 + set ssh_offenders {
14 + type ipv4_addr
15 + flags timeout
16 + timeout 1h
17 + }
18 +
19 + chain input {
20 + type filter hook input priority filter; policy drop;
21 +
22 + # Loopback always allowed.
23 + iif lo accept
24 +
25 + # Reject invalid early.
26 + ct state invalid drop
27 +
28 + # Established and related conversations.
29 + ct state { established, related } accept
30 +
31 + # ICMPv4 — echo, destination-unreachable, time-exceeded, parameter-problem.
32 + ip protocol icmp icmp type {
33 + echo-request,
34 + destination-unreachable,
35 + time-exceeded,
36 + parameter-problem
37 + } limit rate 10/second accept
38 +
39 + # ICMPv6 — required for IPv6 to function (NDP, MLD, etc.) plus echo.
40 + ip6 nexthdr ipv6-icmp icmpv6 type {
41 + destination-unreachable,
42 + packet-too-big,
43 + time-exceeded,
44 + parameter-problem,
45 + echo-request,
46 + nd-router-advert,
47 + nd-neighbor-solicit,
48 + nd-neighbor-advert,
49 + mld-listener-query,
50 + mld-listener-report,
51 + mld-listener-done,
52 + mld2-listener-report
53 + } accept
54 +
55 + # SSH — soft rate limit; offenders into a timeout set for one hour.
56 + tcp dport 22 ct state new ip saddr @ssh_offenders drop
57 + tcp dport 22 ct state new limit rate 4/minute burst 6 packets accept
58 + tcp dport 22 ct state new add @ssh_offenders { ip saddr timeout 1h } drop
59 +
60 + # Log everything else dropped, rate-limited so the log doesn't flood.
61 + limit rate 5/minute log prefix "nft drop: " level info
62 + }
63 +
64 + chain forward {
65 + type filter hook forward priority filter; policy drop;
66 + # Mountaineer hosts do not forward by default. Workloads that need
67 + # forwarding (a podman network, a wireguard bridge) add a chain or
68 + # rule in their own .nft file.
69 + }
70 +
71 + chain output {
72 + type filter hook output priority filter; policy accept;
73 + }
74 + }
@@ -0,0 +1,7 @@
1 + # Mountaineer: unbound on loopback is the only resolver.
2 + # Do not edit this file. Stub zones and forwards go in
3 + # /etc/unbound/unbound.conf.d/.
4 +
5 + nameserver 127.0.0.1
6 + nameserver ::1
7 + options edns0 trust-ad
@@ -0,0 +1,4 @@
1 + nftables
2 + chrony
3 + unbound
4 + sshd
@@ -0,0 +1 @@
1 + bundle
@@ -0,0 +1,3 @@
1 + #!/bin/execlineb -P
2 + fdmove -c 2 1
3 + /usr/sbin/chronyd -d -f /etc/chrony/chrony.conf
@@ -0,0 +1 @@
1 + longrun
A docs/STACK.md +336
A docs/todo.md +237