Skip to main content

max / mountaineer

42.1 KB · 622 lines History Blame Raw
1 # Mountaineer Routes
2
3 Canonical recipes for common operator tasks. Each route is a sequence a competent operator can follow on a fresh Mountaineer install and reach a known-good end state. Routes are *defended* the same way STACK picks are: when a step exists, the next step over has a reason it doesn't.
4
5 Routes are not tutorials. They assume the operator has read MANIFESTO.md and STACK.md, knows which tool does what, and wants the canonical sequence rather than an explanation of each command. Where a step has a non-obvious failure mode, it gets a line. Where it doesn't, it doesn't.
6
7 A route earns its place here by being something Mountaineer wants to make boring. If a task is one-off, situational, or the operator should be making real choices, it belongs in prose elsewhere — not as a route.
8
9 ## First boot
10
11 The sequence between "machine just came up on Mountaineer for the first time" and "I trust this host enough to put work on it." Everything here is verification — the apkovl already did the install. If a step fails, the install is wrong, not the route.
12
13 Targets the Sysadmin and Server profiles equally; profile-specific divergences are called out inline.
14
15 ### 1. Get in
16
17 Console login as `root` with the apkovl-seeded password, or SSH in with the key baked into the overlay. If neither works, the apkovl is broken and the rest of this route is moot — reflash and start over rather than recovering in place. Mountaineer's first-boot story does not include "rescue an install that came up wrong."
18
19 ### 2. Hostname
20
21 ```sh
22 sysop hostname set <name>
23 ```
24
25 Writes `/etc/hostname`, updates the running kernel hostname, and regenerates the prompt without a reboot. Picking a hostname is the first decision the operator makes that the machine remembers; do it before anything else writes logs with the wrong name on them.
26
27 ### 3. Network
28
29 ```sh
30 sysop net status
31 ```
32
33 Confirms an interface is up, has an address, and has a default route. On Sysadmin builds with no ethernet, `sysop wifi connect <ssid>` first. The check is not "can I ping 8.8.8.8" — DNS and ICMP get verified separately below. Here we only want: link, address, route.
34
35 ### 4. Time
36
37 ```sh
38 chronyc tracking
39 chronyc sources
40 ```
41
42 `tracking` should show a non-zero `Reference ID`, a `Stratum` under 5, and a `System time` offset under a second. `sources` should show at least three reachable servers with `^*` or `^+` selection state. If time is wrong, every subsequent step that touches TLS — ACME, DoT to the resolver, package signature checks — will fail in confusing ways. Fix time before anything else.
43
44 ### 5. Resolver
45
46 ```sh
47 sysop resolver verify
48 ```
49
50 Wraps three checks: unbound is listening on `127.0.0.1:53`, a known-good name resolves, and a DNSSEC-signed zone validates (`dig +dnssec dnssec-failed.org` returns SERVFAIL; `dig sigok.verteiltesysteme.net` returns NOERROR with `ad` flag). The DoT upstream is in `/etc/unbound/unbound.conf` and the operator does not edit `/etc/resolv.conf` — that file points at `127.0.0.1` and stays that way.
51
52 ### 6. SSH keys
53
54 ```sh
55 sysop ssh keys list
56 sysop ssh keys add <path-or-url>
57 ```
58
59 The apkovl seeds at least one key; this step is for adding the operator's working keys and removing the bootstrap key if one was used. After this step, the bootstrap path is closed. `sshd_config` is already hardened per STACK.md (no passwords, no root, publickey only); the route does not re-defend that pick, it only verifies the keys list is what the operator intends.
60
61 ### 7. Firewall
62
63 ```sh
64 sysop fw status
65 sysop fw apply
66 ```
67
68 `status` shows the baseline nftables ruleset Mountaineer ships: drop inbound by default, allow established/related, allow SSH on 22, allow ICMP echo, log dropped on a rate limit. `apply` is idempotent and reloads from `/etc/nftables.d/`. Server profile: open whatever the workload needs here, explicitly, by editing a file in `/etc/nftables.d/` and re-applying. Sysadmin profile: usually nothing to add.
69
70 ### 8. ZFS sanity
71
72 ```sh
73 zpool status
74 zfs list
75 ```
76
77 `zpool status` should show `state: ONLINE` and no errors. `zfs list` should show the dataset layout the installer created (`rpool/ROOT/mountaineer`, `rpool/home`, `rpool/var`, etc. — exact layout defended in the install mechanics doc when it lands). If `zfs_arc_max` was sized at install, confirm it took: `cat /sys/module/zfs/parameters/zfs_arc_max`.
78
79 ### 9. Reboot
80
81 ```sh
82 reboot
83 ```
84
85 The only honest test that the apkovl + ZFS + s6-rc stack survives a real boot, not just the one the installer staged. After reboot, walk steps 3–5 again. If anything is different post-reboot, the install is non-idempotent and the operator should know now, not in three weeks under load.
86
87 ### 10. Record
88
89 Write down: hostname, primary interface MAC, ssh host key fingerprints (`sysop ssh hostkeys`), zpool GUID, install date. Mountaineer does not ship a fleet inventory tool; the operator owns this list. The list is what makes the next restic restore drill possible.
90
91 ---
92
93 End state: a host that resolves names, keeps time, accepts the operator's keys, drops uninvited traffic, and survives a power cycle. Everything else — Caddy, podman, WireGuard, mail — is a separate route on top of this one.
94
95 ## Bastion + ProxyJump (the non-mesh floor)
96
97 The path for operators whose servers have public IPs and who ssh from a known set of locations. One host gets the job of being internet-facing for ssh; every other host exposes ssh only to it. The operator's `~/.ssh/config` does the routing with `ProxyJump`, and the rest of the stack — sshd hardening, nftables, hostkey hygiene — does the work.
98
99 This is the floor, not the ceiling. If your laptop roams onto arbitrary networks, or you grow a second site, or anything inside the fleet stops having a path to the bastion, the mesh route is the right answer. But for the single-operator, fixed-location, public-IP-servers case — which is most homelabs and a lot of small production setups — a bastion is enough, and it is honest sysadmin work the operator should know how to do whether or not they ever run a mesh on top.
100
101 ### 1. Pick the bastion
102
103 One Server-profile host with a public IP, already through the first-boot route. Its only job is to be the ssh entry point — do not stack Caddy, podman workloads, or anything else with internet-facing ports on it. The bastion's blast radius is "everything reachable from the bastion," so it stays as small as possible. If the operator already has a host running Caddy with public 80/443 open, that's fine to also run sshd on (the attack surfaces are unrelated), but the *workloads* behind Caddy belong on an inside host that the bastion proxies into.
104
105 ### 2. Tighten the bastion's sshd
106
107 Mountaineer's default `sshd_config` is already hardened per STACK.md (no passwords, no root, modern KEX/cipher/MAC list, `AuthenticationMethods publickey`). For the bastion role, add:
108
109 - `AllowUsers <operator>` — explicit allow-list, not a deny-list. The bastion authenticates exactly one human; nothing else.
110 - `MaxAuthTries 3` and `LoginGraceTime 20` — short windows; bastions get probed constantly and there is no reason to be generous.
111 - `MaxStartups 10:30:60` — backpressure under flood; tighter than the default.
112 - `AllowAgentForwarding no` and `AllowTcpForwarding yes` — the bastion permits TCP forwarding (this is what `ProxyJump` needs) but refuses agent forwarding (this is the footgun `ProxyJump` makes unnecessary; see step 5).
113 - `PermitTunnel no`, `X11Forwarding no`, `GatewayPorts no` — features the bastion does not need are off.
114
115 Reload sshd. Re-test from a separate session before disconnecting the one you have open; a misconfigured sshd that won't accept new connections is the most preventable lockout in this route.
116
117 ### 3. Tighten the bastion's nftables
118
119 The first-boot baseline already drops inbound and allows ssh on 22 with a rate limit. For the bastion role, the rate limit should be more aggressive — public-internet ssh sees hundreds of attempts per hour from scanners, and the goal is to drop the noise before sshd has to deal with it. Mountaineer's bastion-mode ruleset (drop into `/etc/nftables.d/bastion-ssh.nft` and apply with `sysop fw apply`):
120
121 ```
122 table inet bastion {
123 set ssh_offenders { type ipv4_addr; flags timeout; timeout 1h; }
124 chain input {
125 type filter hook input priority -10; policy accept;
126 tcp dport 22 ct state new ip saddr @ssh_offenders drop
127 tcp dport 22 ct state new limit rate 4/minute burst 4 packets accept
128 tcp dport 22 ct state new add @ssh_offenders { ip saddr timeout 1h } drop
129 }
130 }
131 ```
132
133 Anyone who exceeds 4 new SSH connections per minute lands in the offenders set for an hour. This is the firewall doing what fail2ban does, without fail2ban — nftables already has the primitives, principle 15.
134
135 ### 4. Lock inside hosts to the bastion
136
137 On every Server-profile host *behind* the bastion, the sshd should be unreachable from anywhere except the bastion. Two layers, both required:
138
139 - **sshd listens on the right interfaces only.** If the host has a LAN address, `ListenAddress 10.x.x.x` binds sshd to LAN; if not, `ListenAddress 0.0.0.0` is fine because the firewall handles it. Do not rely on `ListenAddress` alone — operators who put their bastion and inside hosts on the same LAN often expose ssh more widely than they realize.
140 - **nftables only accepts ssh from the bastion's address.** Add `/etc/nftables.d/inside-ssh.nft`:
141
142 ```
143 table inet inside {
144 chain input {
145 type filter hook input priority -10; policy accept;
146 tcp dport 22 ip saddr <bastion-public-ip> accept
147 tcp dport 22 ip saddr <bastion-lan-ip> accept
148 tcp dport 22 drop
149 }
150 }
151 ```
152
153 Both addresses are listed because the bastion's connection to the inside host may take either path depending on routing. `sysop fw apply` to load it. Verify by trying to ssh to the inside host *directly* from your laptop — it should hang and time out, not refuse. A refusal means the firewall rule isn't matching the way you think it is.
154
155 ### 5. The operator's `~/.ssh/config`
156
157 This is the load-bearing piece. On the operator's laptop (or wherever they ssh from):
158
159 ```
160 Host bastion
161 HostName bastion.example.org
162 User <operator>
163 IdentityFile ~/.ssh/id_ed25519_personal
164 IdentitiesOnly yes
165
166 Host inside-*
167 User <operator>
168 IdentityFile ~/.ssh/id_ed25519_personal
169 IdentitiesOnly yes
170 ProxyJump bastion
171
172 Host inside-app1
173 HostName 10.0.0.16
174 Host inside-app2
175 HostName 10.0.0.17
176 ```
177
178 `ssh inside-app1` opens a connection to the bastion, asks the bastion to TCP-forward to `10.0.0.17:22`, and runs the ssh handshake with `inside-app1`'s sshd *through* that tunnel. The bastion sees an encrypted stream and the source/destination IPs — nothing more. It cannot read or sign anything in the inner handshake.
179
180 `IdentitiesOnly yes` is not optional. Without it, ssh offers every key in the agent to every host, which both leaks key fingerprints and causes `MaxAuthTries` lockouts on hosts that don't recognize the early offerings.
181
182 ### 6. Why not agent forwarding
183
184 The traditional "hop through a bastion" pattern was `ssh -A bastion` then `ssh inside-app1` from the bastion, with agent forwarding exposing the operator's ssh-agent socket on the bastion. **Don't do this.** Anyone with root on the bastion (including an attacker who has compromised it) can use the forwarded agent socket to authenticate as the operator anywhere the operator's keys are trusted — for as long as the forwarding session is open. `ProxyJump` makes this completely unnecessary: the inner handshake happens on the operator's laptop, the operator's keys never leave the laptop, the bastion is just a TCP relay. There is no scenario in 2026 where agent forwarding is the right answer for this pattern.
185
186 Mountaineer's bastion sshd_config sets `AllowAgentForwarding no` (step 2) to make the wrong path stop working. Operators who insist on agent forwarding for a specific case (some CI integration, etc.) can override it per-host, but the default is off and the route does not document the override.
187
188 ### 7. Hostkey hygiene
189
190 Every inside host has its own SSH host key (verified during first-boot step 6 of that host's install). The operator's `known_hosts` accumulates entries for the bastion *and* every inside host — `ProxyJump` does a real handshake with each, not a transitive trust through the bastion. This means a compromised bastion **cannot** silently MITM the operator's connection to an inside host; the inner handshake would present the wrong host key and ssh would refuse to connect.
191
192 Record every inside host's host key fingerprint in the inventory list from first-boot step 10. If a fingerprint changes unexpectedly, that is the signal that something is wrong — not a thing to click past.
193
194 For operators with many inside hosts, an SSH certificate authority that signs host certificates is the cleaner answer (`@cert-authority` line in `known_hosts` replaces N per-host entries). Mountaineer does not document this in v1 because the `sysop id` SSH CA model is deliberately deferred (see todo). When that ships, this section grows a "use an SSH CA" subsection; until then, per-host `known_hosts` entries are the route.
195
196 ### 8. Test the failure modes
197
198 Three things to deliberately try before declaring this route done:
199
200 - **Bastion sshd misconfig.** Reload sshd with a broken config (intentionally — `Port 22222` and don't open it in the firewall). Confirm you cannot log in *from a new session* and that your existing session is still alive. Roll back. The lesson: never disconnect the working session before the new config is verified.
201 - **Direct connection to an inside host.** From the operator's laptop, `ssh -o ProxyJump=none inside-app1`. It should hang. If it connects, the inside host's firewall is wrong and you have just learned this in the safe way.
202 - **Bastion down.** Power off the bastion (or just `sysop service disable sshd` on it). The operator now has no path to the inside hosts. Plan for this: a console (KVM, IPMI, cloud provider serial console, physical access) must exist as the emergency entrypoint to at least one inside host, or the operator is one bastion outage away from a fleet they cannot reach. The bastion is a single point of failure by design; the recovery story is "bring the bastion back," and the operator needs to know what that requires.
203
204 ### 9. Record
205
206 Add to the per-host inventory from first-boot: bastion FQDN, bastion's public IP, inside-host address scheme, the contents of `~/.ssh/config` (the operator's working copy is the canonical one, but a checked-in copy in the operator's own dotfiles repo is the right backup).
207
208 ---
209
210 End state: every inside host accepts ssh only from the bastion; the bastion accepts ssh only from the operator's keys with aggressive rate limiting; the operator types `ssh inside-app1` and reaches the inside host with a real end-to-end handshake; agent forwarding is off; the failure mode (bastion down) is known and the recovery path exists. When the operator's fleet outgrows this shape — roaming laptop, second site, NAT-trapped hosts — the mesh route is the next step, and nothing in this route blocks adopting it.
211
212
213
214 The canonical way to bring a Mountaineer fleet onto its overlay mesh. Nebula is the default mesh tool — defended in STACK.md — and ships installed on every build. The daemon does not start until a host has a signed certificate and a config; this route is the path from "nothing" to "every host I care about is on the mesh and ssh-over-mesh works."
215
216 Skip this route if your fleet is one box with a public IP and you ssh from a single fixed location. Hardened sshd plus the bastion route is the floor; the mesh is for laptops that roam, servers behind CGNAT, or multiple sites. The mesh handles transport only — SSH identity is unchanged, every host's `/etc/ssh/sshd_config` is still the hardened default from the first-boot route, and the operator's SSH keys are still the authentication that matters.
217
218 ### 1. Pick the CA host and the lighthouse host
219
220 Two roles, often the same host on a small fleet:
221
222 - **CA host.** Where `nebula-cert` lives and where signing happens. The CA private key is the keys-to-the-kingdom secret — anyone with it can mint a cert that joins your mesh. Mountaineer's convention: the CA lives on the operator's Sysadmin laptop (offline-ish, encrypted at rest with age — see the sops+age route when written), *not* on a server. Signing is a deliberate operator action, not a network service.
223 - **Lighthouse host.** A Server-profile box with a public IP that helps other nodes discover each other and punch through NAT. Lighthouses are stateless — they hold no authoritative data — so picking one is low-stakes. Pick a box you would already keep alive; the same box that runs Caddy and the bastion sshd is the classic answer. Two lighthouses is fine and the right answer for any fleet that cares about uptime; the route uses one for clarity.
224
225 ### 2. Create the CA
226
227 On the laptop:
228
229 ```sh
230 sysop mesh ca init --name "<operator-or-org>"
231 ```
232
233 Wraps `nebula-cert ca -name "<...>"`, writes `ca.crt` and `ca.key` to `~/.config/mountaineer/nebula-ca/`, and encrypts `ca.key` with age to a recipient list the operator chooses (their own age key at minimum). The CA cert is valid for one year by default — Mountaineer prefers shorter CA lifetimes than Nebula's default because a one-year forced rotation is a feature, not a bug. Re-keying is in a separate route.
234
235 The CA private key never leaves the laptop. Treat it like an SSH key for an account that owns the whole network — because that is what it is.
236
237 ### 3. Sign the lighthouse cert
238
239 ```sh
240 sysop mesh sign --name lighthouse-1 --ip 10.42.0.1/24 --groups lighthouse,server
241 ```
242
243 Wraps `nebula-cert sign`. Mountaineer's address-space convention: `10.42.0.0/24` for the first mesh, lighthouses in the first 16 addresses, servers in `10.42.0.16-127`, laptops in `10.42.0.128-255`. The operator can pick anything that doesn't collide with their LAN; document the choice alongside the first-boot record.
244
245 Groups are the ACL primitive — Nebula's per-host firewall stanza references them. The defaults Mountaineer signs with:
246
247 - `lighthouse` — runs in lighthouse mode
248 - `server` — Server-profile host
249 - `laptop` — Sysadmin-profile host
250 - `admin` — gets shell access to everything (the operator's own laptop)
251
252 Output is `lighthouse-1.crt` and `lighthouse-1.key`. Both need to reach the lighthouse host; the route below uses a single tarball generated by `sysop mesh bundle`.
253
254 ### 4. Bundle and transfer
255
256 ```sh
257 sysop mesh bundle --name lighthouse-1 --lighthouse-mode --output lighthouse-1-bundle.tar.age
258 ```
259
260 Produces an age-encrypted tarball containing: the host cert, the host key, the CA cert (public — needed by every node to validate other certs), and a generated `/etc/nebula/config.yml` pre-populated for lighthouse mode. The recipient is the host's age key (which the operator added during first-boot step 6, or generates now via `sysop ssh keys` — TODO: reconcile age-key-per-host with the ssh keys command when the sysop section lands).
261
262 Transfer the bundle by whatever path you already trust (scp over the bastion, USB stick, etc). The bundle is encrypted so the transport doesn't have to be.
263
264 ### 5. Bring up the lighthouse
265
266 On the lighthouse host:
267
268 ```sh
269 sysop mesh join lighthouse-1-bundle.tar.age
270 ```
271
272 Decrypts the bundle, installs `/etc/nebula/{ca.crt,host.crt,host.key,config.yml}` with `0600` on the key, opens UDP/4242 in nftables (the route adds this to `/etc/nftables.d/`, persistent and visible — not magic), and starts the `nebula` service. Verify:
273
274 ```sh
275 sysop mesh status
276 ```
277
278 Should show: daemon running, lighthouse mode on, zero connected peers (no other nodes exist yet), listening on `:4242`.
279
280 ### 6. Enroll a server
281
282 Back on the laptop, sign a cert for the first server, naming the lighthouse in the bundle's config:
283
284 ```sh
285 sysop mesh sign --name app-1 --ip 10.42.0.16/24 --groups server
286 sysop mesh bundle --name app-1 --lighthouse lighthouse-1.example.org:4242 --output app-1-bundle.tar.age
287 ```
288
289 On the server, `sysop mesh join app-1-bundle.tar.age` and then `sysop mesh status` should show one connected peer (the lighthouse). `ping 10.42.0.1` from the server confirms the tunnel. If it doesn't, the failure is almost always UDP/4242 blocked somewhere between the server and the lighthouse — fix the firewall, not the config.
290
291 The server's nftables baseline drops inbound by default. To allow ssh from the mesh, the route adds a rule allowing tcp/22 from `10.42.0.0/24` to `/etc/nftables.d/mesh-ssh.nft`. The operator can tighten this further (only from the laptop group, etc) but Nebula's own per-host firewall (configured in `config.yml`) is the more idiomatic place — see step 8.
292
293 ### 7. Enroll the laptop
294
295 ```sh
296 sysop mesh sign --name alice-laptop --ip 10.42.0.128/24 --groups laptop,admin
297 sysop mesh bundle --name alice-laptop --lighthouse lighthouse-1.example.org:4242 --output alice-laptop-bundle.tar.age
298 sysop mesh join alice-laptop-bundle.tar.age # on the laptop
299 ```
300
301 After this, `ssh operator@10.42.0.16` from the laptop reaches the server. Same ssh keys, same hardened sshd, mesh handling the transport. The address is the part that gets boring; everything else is what you already had.
302
303 ### 8. Per-host firewall (Nebula's, not nftables')
304
305 Nebula's `config.yml` has a `firewall` stanza that filters based on groups embedded in the *remote* peer's cert. This is the mesh's contribution on top of nftables — the host firewall blocks unknown CIDRs, the Nebula firewall blocks unknown *identities*.
306
307 The bundle ships with a default policy: `laptop` and `admin` groups can reach `server` group on tcp/22; `admin` can reach `server` on all ports; nothing else is allowed inbound on the mesh interface. Edit the file on the host (`/etc/nebula/config.yml`) and `sysop mesh reload` to apply. Validation happens before reload — a broken policy returns an error instead of leaving the mesh half-configured.
308
309 ### 9. DNS
310
311 Mountaineer does not add mesh DNS in this route. unbound owns resolution on every host (per the first-boot route), and the canonical pattern is to maintain a small zone file with mesh hostnames and forward it from unbound as a local stub. The zone file lives at `/etc/unbound/mesh-zone.conf` and is operator-maintained. Auto-generating it from `sysop mesh status` is a candidate `sysop` feature but not part of this route.
312
313 Operationally: ssh by IP works today; ssh by name works once the operator adds entries to the stub zone. Both are fine; one requires less remembering.
314
315 ### 10. Verify and record
316
317 ```sh
318 sysop mesh status --all
319 ```
320
321 Lists every node the lighthouse has seen, with last-seen time, group memberships, and cert expiry. Record alongside the first-boot inventory: mesh CIDR, lighthouse FQDN(s), CA cert expiry, group naming convention. The CA private key location is *not* written down here — that lives in the operator's head and in the age recipient list.
322
323 ### 11. Adding nodes later
324
325 `sysop mesh sign` on the laptop, `sysop mesh bundle`, transfer, `sysop mesh join` on the new host. Each cert is independent — there is no enrollment server to be online, no preauth key to rotate, no central database to update. The lighthouse learns about the new node the first time it phones home.
326
327 Revoking a node: `sysop mesh revoke <name>` adds the cert fingerprint to the CRL distributed via the lighthouse. The route does not cover the CRL distribution mechanic in detail yet — TODO when the cert-rotation route lands.
328
329 ---
330
331 End state: a self-hosted overlay mesh where every node is a Mountaineer host, SSH identity is unchanged from the first-boot route, unbound still owns DNS, the firewall is two layers (nftables blocks CIDRs, Nebula blocks identities), and the only piece of state the operator must protect is the CA private key on their laptop. The lighthouse can be rebuilt from the bundle the laptop can re-emit at any time.
332
333 ## Reverse-proxy a podman container (Caddy + auto-HTTPS)
334
335 The canonical "I have a containerized web service, put it on the internet with TLS" route. Caddy runs as a native package (not in a container) — it is stack-level infrastructure and STACK.md defends that posture. The *application* runs in podman, supervised by s6-rc so it survives reboots. Caddy reverse-proxies to the container's loopback-bound port and handles HTTPS via Let's Encrypt automatically.
336
337 This route assumes the first-boot route is complete on the target host and that the operator can ssh in (directly, through the bastion route, or over the mesh — doesn't matter to this route).
338
339 ### 1. Prereqs
340
341 Three things must be true before the route can succeed; verify all three before touching configs:
342
343 - **Ports 80 and 443 are reachable from the public internet.** Caddy needs both: 80 for the ACME HTTP-01 challenge, 443 for TLS. The first-boot nftables baseline drops inbound, so add `/etc/nftables.d/web.nft` allowing tcp/80 and tcp/443 and run `sysop fw apply`. If there's a NAT or upstream firewall, forward both ports there too.
344 - **DNS A (and AAAA, if you have IPv6) records resolve to this host.** Caddy's auto-HTTPS uses HTTP-01 by default, which requires the FQDN to point at the box answering on port 80. `dig +short app.example.org` from any internet-reachable resolver should return this host's public IP. If it doesn't, the cert issuance will fail in a way that is easy to misdiagnose as a Caddy bug — it isn't, it's DNS.
345 - **The host has the time right.** ACME breaks badly when system time is off by more than a few seconds. Step 4 of first-boot covers this; re-confirm with `chronyc tracking` if you haven't recently.
346
347 ### 2. Pick the host
348
349 A Server-profile host. Public IP, already through first-boot, ports 80/443 free (no other web server bound). Mountaineer's convention: web ingress concentrates on one host per site — the same host that runs the bastion sshd and the Nebula lighthouse is the classic choice for a small fleet. Co-tenancy is fine; the surfaces don't interact.
350
351 ### 3. Run the application container
352
353 Pull and run the image once, by hand, to confirm it works and to see what it binds to. Real example using a small static-site image:
354
355 ```sh
356 podman pull docker.io/library/caddy:2-alpine # any test image; pretend it's your app
357 podman run --rm --name app-test -p 127.0.0.1:8080:80 docker.io/library/caddy:2-alpine
358 curl -sS http://127.0.0.1:8080/
359 ```
360
361 The `127.0.0.1:8080:80` binding is load-bearing: the container's port must be exposed *only on loopback*, not on `0.0.0.0`. Caddy reaches it via loopback; the internet does not. If you bind to `0.0.0.0:8080` you are publishing the app without TLS on a non-standard port and discovering this on a security audit instead of in setup.
362
363 Kill the test container (`podman stop app-test`, or just Ctrl-C with `--rm`) once you've confirmed it serves. The supervised long-running version comes next.
364
365 ### 4. Supervise the container with s6-rc
366
367 Mountaineer's init is s6-rc (STACK.md). For v1, the route uses a hand-written s6-rc service definition; when the podman-s6 service-template library lands (see todo), the template generates this from a single declaration. The hand-written version below is what the template will produce.
368
369 Create `/etc/s6-rc/source/app/`:
370
371 ```
372 /etc/s6-rc/source/app/type # contains: longrun
373 /etc/s6-rc/source/app/run # the script below
374 /etc/s6-rc/source/app/dependencies.d/podman # empty file
375 ```
376
377 The `run` script:
378
379 ```sh
380 #!/bin/execlineb -P
381 podman run --rm --name app
382 -p 127.0.0.1:8080:80
383 --log-driver=k8s-file
384 --log-opt path=/var/log/podman/app.log
385 docker.io/library/caddy:2-alpine
386 ```
387
388 Compile and bring it up:
389
390 ```sh
391 s6-rc-compile /etc/s6-rc/compiled-new /etc/s6-rc/source
392 mv /etc/s6-rc/compiled /etc/s6-rc/compiled-prev && mv /etc/s6-rc/compiled-new /etc/s6-rc/compiled
393 s6-rc -u change app
394 ```
395
396 Verify with `s6-rc -a list` (should show `app` running) and `curl http://127.0.0.1:8080/`. The container is now supervised: if it crashes, s6 restarts it; if the host reboots, it comes back. `--rm` matters — it means the container doesn't accumulate a stopped-container record on every restart.
397
398 The `dependencies.d/podman` empty file declares this service depends on the podman bundle (network ready, podman storage initialized). Mountaineer's default service tree includes that bundle; the file just opts this service into it.
399
400 ### 5. Configure Caddy
401
402 Drop a site block into `/etc/caddy/Caddyfile.d/app.caddy`:
403
404 ```
405 app.example.org {
406 reverse_proxy 127.0.0.1:8080
407 log {
408 output file /var/log/caddy/app.log
409 format json
410 }
411 encode zstd gzip
412 }
413 ```
414
415 That is the whole file. Caddy handles ACME, TLS, HSTS, HTTP→HTTPS redirect, OCSP stapling, and modern cipher selection without configuration. The `encode` line opts into compression. The `log` block writes structured JSON that Vector picks up in step 7.
416
417 Mountaineer's `/etc/caddy/Caddyfile` is a one-liner that imports `/etc/caddy/Caddyfile.d/*.caddy`; per-site config goes in the drop-in directory so adding a site is one file, removing it is `rm`, and the operator never edits a giant top-level Caddyfile.
418
419 Reload Caddy:
420
421 ```sh
422 caddy validate --config /etc/caddy/Caddyfile
423 sysop service reload caddy # SIGUSR1 under the hood; no connection drop
424 ```
425
426 `validate` first; a syntactically broken Caddyfile that survives a reload will refuse to start on the next restart and the operator will discover this at the worst possible time. The route does not skip this step.
427
428 ### 6. Verify HTTPS issuance
429
430 Watch Caddy's log for the ACME handshake:
431
432 ```sh
433 tail -F /var/log/caddy/caddy.log
434 ```
435
436 Expect to see Caddy fetch a directory from the ACME endpoint, complete an HTTP-01 challenge, and issue a certificate — typically within 10 seconds of the reload on a clean DNS setup. Then from any internet host:
437
438 ```sh
439 curl -sS -I https://app.example.org/
440 ```
441
442 `HTTP/2 200`, valid cert chain, HSTS header present. If it returns a self-signed or fallback cert, ACME is rate-limited or upstream-blocked; the log will name which. If the connection refuses, ports 80/443 aren't reachable from the public internet (step 1).
443
444 ### 7. Logs
445
446 Vector (STACK.md) picks up two streams for this route: the Caddy access log at `/var/log/caddy/app.log` (JSON, structured per-request) and the container's stdout/stderr at `/var/log/podman/app.log` (whatever the app emits). The default Vector config tails both and forwards to the operator's chosen sink. If no sink is configured yet, the logs stay on disk and rotate via the system's logrotate config — fine for development, not fine for production. The "configure a Vector sink" route is a separate piece of work.
447
448 One thing worth verifying once: tail the access log while making a request, and confirm the request lands in the log within a second or so. A reverse proxy whose logs you never look at is a reverse proxy you cannot debug.
449
450 ### 8. Reboot test
451
452 ```sh
453 reboot
454 ```
455
456 After reboot, the container should be running (`podman ps`), Caddy should be serving (`curl https://app.example.org/`), and the operator should not have touched anything. If any piece doesn't come back on its own, the supervision or service tree is wrong and the route is not done. The fix is in the s6-rc setup, not in a `crontab @reboot` workaround.
457
458 This is the same idempotence test the first-boot route ends with, applied to a deployed service. The same lesson applies: discover non-idempotence in setup, not under load.
459
460 ### 9. Record
461
462 Add to the host inventory from first-boot: the FQDN, the container image (with tag — `:latest` is not a tag), the s6-rc service name, the Caddyfile drop-in path, the loopback port, the public ports opened. The container's image digest is worth recording separately if reproducibility matters; `podman inspect --format '{{.Image}}' app` prints it.
463
464 ---
465
466 End state: a containerized application reachable at `https://app.example.org/` with a valid certificate, supervised by s6-rc, reverse-proxied by Caddy, logging to disk for Vector to pick up, surviving reboots without operator intervention. Adding a second app is the same route from step 3 onward with a different image and a different drop-in Caddyfile; the route itself doesn't need to be re-run from the top.
467
468 ## Restic restore drill
469
470 The point of this route is not to set up backups. The point is to **deliberately restore from them, before you need to**, on a host that pretends the original is gone. A backup tool you have never restored from is not a backup tool. The drill exists to make every failure mode (forgotten password, password-file lives on the dead host, environment variable not set in the recovery shell, repository unreachable from the network you're restoring from, rest-server append-only token expired) surface during a practice run instead of during an incident.
471
472 The route covers the full loop: initialize a repository, take a real backup, restore a file on the same host (smoke test), and then the drill proper — restore the same data on a *different* Mountaineer host using only what you would have in a real recovery scenario. The full loop is the route; skipping any step is what people do when they want to feel safe rather than be safe.
473
474 ### 1. Pick the repository backend
475
476 Two reasonable answers for Mountaineer:
477
478 - **An S3-compatible object store** (Backblaze B2, Hetzner Object Storage, Wasabi, MinIO on a host you control). The standard answer for small operators. Off-host by default, cheap, durable, restic speaks S3 natively.
479 - **A `rest-server` on a Mountaineer host you control.** A small Go binary that exposes restic's repository format over HTTPS, supports append-only mode (a compromised host cannot delete its own backups), and lives behind Caddy via the previous route. The right answer when the operator already runs a server with disk for this and wants to keep backup data on infrastructure they own.
480
481 Local disk is **not** the drill's target. "The host is gone" includes the local disk. Backing up a Mountaineer host to its own ZFS pool is fine as a *first* tier (and ZFS snapshots cover the bad-upgrade case from a separate route), but it does not satisfy this route's premise. Pick an off-host backend before continuing.
482
483 The route uses B2 as the example because it is the most common case. The other backends substitute cleanly — only the URL form and the credential variables change.
484
485 ### 2. Provision the credentials
486
487 Three secrets the restoring host will need:
488
489 - **The repository URL** (`B2:bucket-name:/path/to/repo` or `rest:https://restic.example.org/repo` or `s3:s3.eu-central-003.backblazeb2.com/bucket-name`).
490 - **The repository password.** Used to derive the encryption key for the repo. Lose this and the backups are unreadable, full stop. This is the most important piece of identity material introduced by this route.
491 - **The backend credentials.** For B2: an application key with read+write to the bucket (operator's account creates this in B2's web UI). For rest-server: an HTTP basic-auth token. For S3: an access key + secret.
492
493 Where these live on the host:
494
495 ```
496 /etc/restic/repo # the URL, plain text, mode 0644
497 /etc/restic/password # the repo password, mode 0600
498 /etc/restic/env # backend creds as env vars, mode 0600
499 ```
500
501 The `env` file for B2 looks like:
502
503 ```sh
504 export B2_ACCOUNT_ID=<keyID>
505 export B2_ACCOUNT_KEY=<applicationKey>
506 ```
507
508 These three files together let any process on the host that can read them perform any operation on the repository. Mode them tight (`0600`), own them root, and treat the password file with the same care as an SSH private key.
509
510 **Critical for the drill: the repo password must also exist somewhere other than this host.** When the deferred `sysop id` age slot ships, the canonical answer is "encrypted to the operator's scope, in the scope's storage." For v1, the operator manages this themselves — a printed copy in a safe, an entry in the operator's password manager, an age-encrypted file on a different machine, whatever. The drill will not work if the password lives only on the host being restored from.
511
512 ### 3. Initialize the repository
513
514 ```sh
515 source /etc/restic/env
516 restic -r "$(cat /etc/restic/repo)" --password-file /etc/restic/password init
517 ```
518
519 Output should end with "created restic repository". From here on, every restic command needs `-r <url> --password-file ...` or the equivalent env vars (`RESTIC_REPOSITORY`, `RESTIC_PASSWORD_FILE`). The `sysop backup` wrapper (see step 7) makes this implicit; for now, type it out so the moving parts are visible.
520
521 ### 4. First real backup
522
523 ```sh
524 restic -r "$(cat /etc/restic/repo)" --password-file /etc/restic/password \
525 backup /etc /home /var/lib --tag firstboot --tag $(hostname)
526 ```
527
528 Mountaineer's convention for what gets backed up:
529
530 - `/etc` always — system configuration, the apkovl-shaped parts that are not in the apkovl.
531 - `/home` always (Sysadmin profile) or selectively (Server profile, often empty).
532 - `/var/lib` always — service state, databases, podman volumes if not separated.
533 - `/var/log` deliberately *not*. Logs are Vector's job; backing them up to restic is paying twice for the same data.
534 - `/var/cache`, `/tmp`, `/run` deliberately not. Reproducible from elsewhere.
535 - ZFS-mounted application data datasets are backed up by their own backup tier (ZFS send-receive to a second pool) and surface here only if the operator has decided restic is also their off-site copy.
536
537 Tag every backup with the hostname (so multi-host repositories stay sortable) and any relevant phase tag (`firstboot`, `pre-upgrade`, etc.).
538
539 Verify it landed:
540
541 ```sh
542 restic -r "$(cat /etc/restic/repo)" --password-file /etc/restic/password snapshots
543 ```
544
545 One snapshot with the tags you set. If the snapshot is missing or the bytes-sent number was implausibly small, the backup didn't actually copy what you thought. Look at `restic backup`'s output before declaring this step done.
546
547 ### 5. Smoke restore (same host)
548
549 Restore a single file to a scratch location and diff it against the original:
550
551 ```sh
552 restic -r "$(cat /etc/restic/repo)" --password-file /etc/restic/password \
553 restore latest --target /tmp/restore-test --include /etc/hostname
554 diff /etc/hostname /tmp/restore-test/etc/hostname && echo OK
555 rm -rf /tmp/restore-test
556 ```
557
558 This proves the round-trip works on the host that took the backup. It does *not* prove anything about disaster recovery — the host still exists, the credentials are still on disk, the network path is the one the backup used. That comes next.
559
560 ### 6. The drill (different host)
561
562 Pick a second Mountaineer host. Sysadmin laptop is fine; a freshly-installed Server VM is better because it more closely simulates "the original host is gone and you're rebuilding." This host must be different enough from the original that you can't accidentally rely on something only the original had.
563
564 On the drill host:
565
566 ```sh
567 apk add restic
568 ```
569
570 Now write the three credential files from scratch using *only the copies that don't live on the original host* — the password from your password manager / safe / age-encrypted backup, the backend creds from the B2 console (or your records), the repo URL from the inventory list. If you cannot reconstruct any of the three from off-host sources, the drill has already found a problem and the route is paused here until it's resolved.
571
572 Then:
573
574 ```sh
575 source /etc/restic/env
576 restic -r "$(cat /etc/restic/repo)" --password-file /etc/restic/password snapshots
577 ```
578
579 You should see the snapshots from the original host. If you don't, the network path or credentials are wrong; debug here before going further.
580
581 Restore the original host's `/etc` to a scratch location:
582
583 ```sh
584 restic -r "$(cat /etc/restic/repo)" --password-file /etc/restic/password \
585 restore latest --target /tmp/drill --host <original-hostname> --path /etc
586 ls -la /tmp/drill/etc/
587 ```
588
589 Verify a handful of files are present and look right. The `--host` filter matters when the repo holds backups from multiple hosts; without it you'd get whatever snapshot was most recent across the whole repo.
590
591 If you got here, the drill has succeeded: a second machine, with only the off-host secrets, can read the backups of the first. **Write down the date.** This is the only thing that earns the "we tested this" claim.
592
593 Clean up: `rm -rf /tmp/drill`. The drill host's job is done; nothing of the drill itself should persist on it.
594
595 ### 7. Schedule periodic backups
596
597 Mountaineer uses busybox `crond` for time-driven tasks. The Mountaineer-canonical wrapper for restic lives at `/usr/local/sbin/sysop-backup` (until `sysop backup` lands properly), sources `/etc/restic/env`, and exposes a single verb: take a backup with sensible flags. Add a crontab entry:
598
599 ```
600 # /etc/crontabs/root
601 17 3 * * * /usr/local/sbin/sysop-backup
602 30 4 * * 0 /usr/local/sbin/sysop-backup forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune
603 ```
604
605 Daily backup at 03:17 (offset minute so the operator's fleet doesn't all hit the backend on the same second), weekly forget+prune at 04:30 Sunday. Adjust the retention to taste; the example is a reasonable starting point that keeps a year of monthly recovery points without unbounded growth.
606
607 The backup invocation should `set -e`, capture failures, and emit them via msmtp (STACK.md) so the operator's "cron emails me when something fails" expectation works without ceremony. A backup script that fails silently is worse than no backup script — at least with no script the operator knows where they stand.
608
609 ### 8. Schedule periodic drills
610
611 This is the part most operators skip and it is the part that actually matters. Put a calendar event on a recurring date (quarterly is reasonable; semi-annual is the absolute floor) titled something like "Restic restore drill — Mountaineer fleet." When it fires, do step 6 again on a fresh host or VM. **Each drill regenerates the credential files from off-host sources** — that is the whole point. If you find yourself copying `/etc/restic/password` from the original host to the drill host, you have stopped doing the drill and started lying to yourself.
612
613 Mountaineer does not ship anything to schedule this for you; calendar discipline is the operator's. The route documents the cadence as a defended convention.
614
615 ### 9. Record
616
617 Add to the inventory from first-boot: repository URL, repository backend type, last successful backup timestamp (cron writes this), last *successful drill* date and the host the drill ran on. The drill date is the load-bearing number — if it is more than a year old, the route is overdue and the operator should treat the backups as untested until they re-drill.
618
619 ---
620
621 End state: a working restic backup against an off-host repository, with credentials persisted both on the host and in a place that survives the host's death; a smoke restore proves the round-trip; a real drill on a different machine proves disaster recovery; a daily cron keeps the backups current; a calendar entry keeps the drill honest. The "we tested this" claim from STACK.md is now redeemable, with a date attached.
622