max / alloy

16.9 KB · 174 lines History Blame Raw

1	# Cutover runbook: fw13, Pop!_OS to Alloy
2
3	This file answers one question. What is the order of operations that moves fw13 from Pop!_OS 24.04 to Alloy, and what makes each step safe to take.
4
5	Companion to [HARDWARE-FW13.md](HARDWARE-FW13.md), which records what the machine is, and to [IMAGE.md](IMAGE.md), which records how the image is built. The track this belongs to is wiki `alloy-fleet-cutover`; the hardware measurements are wiki `alloy-fw13-migration`. The cut itself is GoingsOn infra `19a82995`.
6
7	## Read this first
8
9	This is a destructive swap on the machine's only drive. Ruled by Max 2026-09-03: standard swap, current drive, no replacement NVMe, no shrink, no dual boot, no partition kept bootable. Pop!_OS does not survive it and there is no reboot that brings it back. Shrinking `nvme0n1p3` or booting an external root are not options to weigh; they are mistakes.
10
11	What replaces the rollback is the data hot potato. Machines are cut one at a time and each is restored to usable before the next is touched, so the machine that is up holds what the machine being cut cannot lose. For fw13 that machine is astra, still on Pop. Phase 1 is the act of putting things there, and it is the only safety net this operation has.
12
13	fw13 is not a laptop being reinstalled. It is the tree's only x86_64 build host, and that role is four jobs:
14
15	1. Bento's local x86_64 builder.
16	2. Bento's controller for every other host, which reaches astra, mbp and windows-x86 over rsync.
17	3. The crates.io publish host for eleven registered crates.
18	4. Sando's controller and the MNW build host, with a local PostgreSQL cluster the deploy gates restore into.
19
20	A machine that boots Alloy and cannot do those four is not a migration. It is an outage that lasts until somebody notices. So each of the four is proven inside the built image, as a container on the running Pop install, before the disk is touched. That is Phase 0, and it is what makes the cut cheap: four passes or fails rather than four opinions.
21
22	## What is measured and what is not
23
24	Hardware figures are in HARDWARE-FW13.md and were measured from the machine's own running Pop!_OS. Alloy has never run on this machine. The display seed, amdgpu behaviour under Alloy's kernel, glyph crispness at 1.75x, s2idle drain on AMD and fingerprint enrollment are all unobserved.
25
26	Disk, measured 2026-09-03: `nvme0n1` is 1.8T, `/` is 94% full with 107G free, and `~/Code` alone is 916G, nearly all `target/` directories and caches.
27
28	Nothing in this file has been executed. It is a plan.
29
30	## Phases at a glance
31
32	\| Phase \| Where \| Reversible \|
33	\|---\|---\|---\|
34	\| 0. Prove the four roles in containers \| Ahead, no downtime \| Yes, nothing touched \|
35	\| 1. Empty the machine \| Ahead, no downtime \| Yes, nothing touched \|
36	\| 2. Mint the medium \| Ahead, one long build \| Yes \|
37	\| 3. No-write boot test \| One reboot \| Yes, nothing written \|
38	\| 4. The cut \| At the machine \| No \|
39	\| 5. Restore, then prove a release \| At the machine, then daily \| n/a \|
40
41	Phases 0 through 3 leave the working machine exactly as it was. Everything irreversible is in phase 4, and phase 4 is one step.
42
43	## Phase 0: prove the four roles, in containers, on today's Pop
44
45	Build the image with `build/build-image.sh --host fw13`, which carries `LANGS=rust,js` and `DB=postgres16` (see IMAGE.md, "Per-machine mint recipes"). Then run each role inside it with the tree bind-mounted. `distrobox` 1.8 is unconditional in both profiles and mounts the host home, so `~/Code` paths and `target/` directories are identical inside and out.
46
47	Role 1, compile. `cargo test` for one crate under `~/Code` completes inside the image, and `rustup toolchain list` shows the pinned 1.97.1 active. The image declines rustup and 28 repos pin a toolchain only rustup honours; Fedora 43's rust RPM happens to be 1.97.1 today, so nothing breaks now and nothing would warn later. rustup goes into `$HOME`, which is a machine step rather than an image change (infra `6faa22a6`).
48
49	Role 2, the Tauri release. An AppImage built through a Debian-family distrobox has a highest referenced `GLIBC_` symbol no higher than the AppImage released today, measured at the time rather than assumed (infra `91b1b311`). The reason it cannot be built natively is the bundled libraries, not the compiled executable: cargo-tauri copies the WebKit libraries into the bundle and linuxdeploy excludes libc from what it bundles, so those copies meet the user's glibc. Fedora 43's `libwebkit2gtk-4.1` (2.52.5-1.fc43) requires GLIBC_2.42; Ubuntu 24.04 LTS is 2.39. Measured separately: a Rust executable's own floor is set by which symbols the code uses, not by the build host, and comes out at GLIBC_2.39 on both hosts from `pidfd_getpid` and `pidfd_spawnp`.
50
51	Role 3, the Sando gate. A production dump restores into a local postgres 16 cluster over a socket at `/var/run/postgresql`, and `code_smoke`'s two npm builds complete. The socket detail decides the size of this: the deployed `/etc/sando/sando-daemon.toml:26` uses `postgres:///sando_scratch?host=/var/run/postgresql&user=sando` while the in-repo template uses a TCP URL, and the deployed file is what the gate runs against. Reading the template instead is how this gets mis-sized as client-only. The node half is unverified: Fedora 43 carries node 22.22.2 against fw13's current 24.18.0, and neither frontend declares an `engines` field. `code_smoke` is the one place a frontend failure is fatal, because the server's `build.rs` deliberately is not; with no node it warns and ships whatever `static/dist/` already holds.
52
53	Role 4, the publish. Bento's publish preflight for one makeover crate runs to the point of upload and stops. It exercises `cargo fmt --all --check`, which is where a toolchain that is not the pinned one first shows up; the annotated `v<version>` tag the preflight builds from rather than the branch; and minisign on PATH, since both Linux release recipes call `dist/sign-artifacts.sh` and it hard-fails without it.
54
55	### What the image already carries, and what it does not
56
57	Probed against the built `alloy:clip-client` and `alloy:clip-server`. Read these as requirements rather than as work: `rsync` and `minisign` are both in the base package block, `xdg-mime` and `xdg-open` are in the unconditional base rather than the client branch, and `fprintd` with `fprintd-pam` are in the client branch with `pam_fprintd` asserted `sufficient` in `system-auth`. The database and the language toolchains are the mint-time dials Phase 0 uses.
58
59	Three that stay open and turn into confusing failures rather than clean ones:
60
61	- `libfuse.so.2` is absent, so the vendored appimagetool cannot self-mount for the audiofiles build. Measured that `APPIMAGE_EXTRACT_AND_RUN=1` on that exact tool succeeds and writes a valid AppImage, so the fix is one environment variable in `dist/build-appimage.sh`, not a package.
62	- `patchelf` is needed in neither the image nor a container. linuxdeploy ships its own inside its AppImage and resolves it from the extracted AppDir. What does need doing is unrelated: that path fetches linuxdeploy and its plugins from GitHub on every run, so an offline or rate-limited fw13 cannot release, on Alloy or on Pop (infra `83ee4a38`).
63	- `sccache` is a `cargo install` in `~/.cargo/bin`, not a system package, so restoring it is a `$HOME` step. Worth restoring: 18273 cached objects over 13G and a 71% hit rate on a controlled cross-project test.
64
65	## Phase 1: empty the machine
66
67	Nothing here needs downtime, and after it there is nothing on fw13 that exists only on fw13.
68
69	- Every repo pushed to the remotes its policy allows. An unpushed commit is the single most losable thing in this operation. `wright` goes to astra and srht and never to mnw.
70	- `clone-tree.sh` completes into a scratch root on a machine that is not fw13. A dry run on the machine being wiped proves less than it looks like.
71	- The `$HOME` credential material confirmed on two other machines. Release signing keys, updater keys and registry credentials live in `$HOME` and in no repo; astra holds byte-identical copies today and fw12 takes a third under the fleet plan. `settings.local.json` sits outside `_private` because it carries live tokens (infra `7f373187`).
72	- `~/.bashrc` translated or kept. It carries the daemon tokens and the PATH the build-host role depends on, and Alloy's default shell is nushell. bash is in the image, so keeping a bash login shell makes this a non-event and taking nushell makes it one file (infra `e4ddabbf`).
73	- The cargo-installed tool set recorded in `_private/infra/bootstrap`, which lists repos and the cargo config and stops there. From `~/.cargo/.crates2.json`, eleven entries: `cargo-audit`, `cargo-deny`, `cargo-fuzz`, `cargo-mutants`, `cargo-sweep`, `samply`, `sccache`, `sqlx-cli` (built `--no-default-features --features postgres`, so reinstalling from the bare name silently produces a different binary), `tauri-cli`, and the two path installs `spag-mcp` and `spaghetti`, which follow the clone rather than precede it.
74
75	## Phase 2: mint the medium
76
77	The image build takes over an hour and can only run on an x86_64 host, which today means fw13 itself. Build and mint before the machine goes down, not during.
78
79	- `build/build-iso.sh --host fw13 --update-target fw13:5000/alloy:local`, on `main`, and record the commit it was minted from. A boot test against a stale image answers a question nobody asked. The update target is not optional here; see below.
80	- Write it with `build/write-device.sh`. Standing drive rule: the SanDisk is the install-media and scratch drive, the Samsung T9 is never a write target.
81	- Treat whatever is already on the SanDisk as unknown until it is plugged in and read.
82	- The `rust-stage` witchbroom cell is live on astra and guards the failure class that bites here: a `SHOP_REV` bump silently adding a system build dependency and taking the image build with it.
83
84	### Bake an update target, or the machine cannot be fixed without a second install
85
86	Without `--update-target` the installer writes `quay.io/alloy/alloy:43` into the deployment origin, and `crates/alloy/src/install/image.rs` calls that address a deliberate dead end rather than a plan: nothing is published there and nothing is going to be. It is written anyway because omitting `--target-imgref` makes bootc record the live medium's own mount path, which fails at reference parsing instead of degrading to "no update found".
87
88	So a machine minted without the flag has no update path on the far side of Phase 4, at the point where it is newest and most likely to need one, and recovering costs a `bootc switch` typed at the machine.
89
90	- The tag is `local` and not `43` because `local` is what `build/dev-push.sh` publishes. `43` names a channel that does not exist. An untagged reference is refused at build time and again at install time, because the containers stack reads a missing tag as `latest` and IMAGE.md makes a Fedora major bump a deliberate act.
91	- The host name rather than `127.0.0.1`, so one medium serves fw12 pulling from fw13 as well as fw13 pulling from itself.
92
93	The same flag enables `bootc-fetch-apply-updates.timer`. `updates_scheduled()` reads the same kernel parameter, on the reasoning that a medium naming a target is exactly what a development build is. bootc's cadence is an hour after boot and every eight after that, and it stages what it finds for the next boot. On a machine that four other things depend on, an image arriving because a timer fired is not something anyone asked for, and the registry it would reach is a podman container that is usually down. Mask it after the install and drive updates by hand:
94
95	sudo systemctl mask --now bootc-fetch-apply-updates.timer
96
97	## Phase 3: the no-write boot test
98
99	One reboot, nothing written. Boot the medium, stop at the disk selection screen, answer three questions, power off.
100
101	1. Does amdgpu bring up the 2880x1920 panel under Alloy's kernel?
102	2. What does the display generator propose? HARDWARE-FW13.md predicts 1.75x for the panel and 1.0x for the BenQ RD280U on DP-3, from EDID arithmetic. Neither has been observed, and `PPI_PER_SCALE` in `crates/alloy/src/display.rs` has exactly one data point behind it.
103	3. Does 1.75x look crisp, and does 1.0x on a 163 PPI 28 inch panel look right at desk distance?
104
105	Way back: power off and remove the medium. This is the last phase where a way back exists.
106
107	## Phase 4: the cut
108
109	Wipe `nvme0n1`, install, unlock, log in. The encrypted path is proven on fw12 hardware (alloy `6ef745fd`), not on this machine.
110
111	After this step Pop!_OS is gone. Everything that made it recoverable was Phase 1.
112
113	## Phase 5: restore, then prove a release
114
115	1. Clone `_private` from astra, run `clone-tree.sh`.
116	2. Restore the `$HOME` material and the cargo tool set. sccache first: nothing under `~/Code` compiles pleasantly without it.
117	3. Build one crate and confirm the toolchain answers.
118	4. Run one real release end to end: a makeover publish is the smallest, and a goingson AppImage is the one that exercises the whole gap list.
119	5. Only then point bentod and sandod at this machine.
120
121	Until step 5, fw13 is a machine that boots Alloy. After it, fw13 is the x86_64 build host again (infra `28cd7d0c`).
122
123	### Fixing Alloy itself, without minting a second medium
124
125	This is what the Phase 2 update target buys, and it is the loop for every "did that config land" question. An ISO is the wrong artifact for that: measured on fw13, a config-only change is fifteen seconds of `podman build` followed by an ISO assembly costing sixteen times as much, producing installer media nobody is going to install from.
126
127	Once, because the dev registry speaks plain HTTP:
128
129	printf '[[registry]]\nlocation = "fw13:5000"\ninsecure = true\n' \
130	\| sudo tee /etc/containers/registries.conf.d/99-alloy-dev.conf
131
132	`build/dev-push.sh` also prints a `bootc switch`. It is not needed when the medium baked the same reference, so read `bootc status` before typing it. Then every fix is:
133
134	podman build ... # or build/build-iso.sh --skip-source
135	build/dev-push.sh
136	sudo bootc upgrade && sudo systemctl reboot
137
138	and a bad one is `sudo bootc rollback && sudo systemctl reboot`. Same A/B staged switch and same rollback the installer uses, with the image coming off a registry on this box rather than one on the internet.
139
140	Two things about this are unproven on fw13 specifically, and the second is a trap. dev-push has only ever been used with the registry on a different machine from the target, and here it is both; that is alloy `eca827e8`. And the registry is a podman container that does not come back on its own after a reboot, so it has to be running at the moment `bootc upgrade` does, which is why the timer is masked in Phase 2 rather than left to fire into a closed port.
141
142	Nothing here needs astra, mbp or any other machine. fw13 builds its own updates and serves them to itself.
143
144	Then the machine gets daily-driven, which is where the rest of the reading happens: s2idle drain on AMD, multi-output reconcile with the laptop panel plus the BenQ, fprintd enroll on the Goodix part, and the dark-mode L stops on this panel as a second reading against the FW12's. The migration also pays Alloy back here: fw13 on Pop runs systemd 255 and `run0` needs 256, which is why alloy `3587c247` sat on needing a host.
145
146	## What is not recoverable
147
148	All of it is lost at Phase 4. Check each one in Phase 1, not after.
149
150	- Unpushed commits in any repo. The bootstrap manifest rebuilds the tree from the remotes. It cannot rebuild what was never pushed.
151	- Anything in `$HOME` that is not in a repo, not mirrored to astra, and not a Syncthing folder. Max, 2026-09-03: the machines hold no personal information worth keeping, so the exceptions are the credential material and nothing else. Shell history, browser profiles and local scratch go.
152	- The Pop-side build environment itself. The webkit, gtk, libsoup and FUSE packages that make releases work today are apt packages on that install. Once it is gone the fallback is gone, which is why Phase 0 proves the Alloy side first.
153	- Local caches. The podman image cache, `target/` directories and the sccache cache all regenerate from source. Not backed up on purpose.
154
155	## Ahead of time, versus at the machine
156
157	Machine time is the scarce half, so the split is deliberate.
158
159	Ahead, with fw13 fully in service: all of Phase 0, 1 and 2.
160
161	At the machine: the Phase 3 boot test and its three judgments, the Phase 4 install and unlock, and the display and suspend readings that only exist under a running session.
162
163	Over time, and not compressible: the daily driving in Phase 5.
164
165	## astra is not covered here
166
167	This runbook covers fw13 and only fw13. astra is a service host with state, it is cut first and fixed after, and its ordering is wiki `alloy-fleet-cutover` and wiki `alloy-astra-migration`. Do not try to answer astra's questions inside this file.
168
169	## Open questions this runbook does not answer
170
171	- Whether bentod grows a distrobox transport, or whether the build deps go in the image (`d77339f0`, `c4eb6d5e`).
172	- Whether the server profile is a product at all (`d1fed0d7`), which the build-host role argues for.
173	- Whether the two MNW frontends build under node 22, which Phase 0 role 3 answers by running them.
174