Skip to main content

max / alloy

Rewrite the fw13 runbook as a destructive swap gated on four role rehearsals
Author: Max Johnson <me@maxj.phd> · 2026-09-03 17:19 UTC
Signed with PGP, not checked
Commit: aeead281161c6355b083c609606b1aff2f870a3b
Parent: 50bff9b
1 file changed, +67 insertions, -174 deletions
M docs/MIGRATION-FW13.md +67 -174
@@ -1,37 +1,29 @@
1 1 # Cutover runbook: fw13, Pop!_OS to Alloy
2 2
3 - > **Phases 2 and 3 below are void, 2026-09-03.** They describe a shrink of `nvme0n1p3` or an
4 - > external root, with Pop kept bootable as the way back. Max ruled out dual boot on 2026-08-27
5 - > and ruled out buying a drive on 2026-09-03: fw13 is a standard destructive swap on its current
6 - > drive, and the way back is the data hot potato, not a partition. Do not shrink anything on
7 - > this machine. The plan of record is wiki `alloy-fleet-cutover`; this file is rewritten to it
8 - > under GoingsOn alloy `5892fc62`.
9 - >
10 - > Two figures below are also stale, measured 2026-09-03: `/` is 94% full with 107G free, not
11 - > 798G, and fprintd plus fprintd-pam are in the image (`Containerfile:1637`) rather than absent.
3 + This file answers one question. What is the order of operations that moves fw13 from Pop!_OS 24.04 to Alloy, and what makes each step safe to take.
12 4
13 - This file answers one question. What is the order of operations that moves fw13 from Pop!_OS 24.04 to Alloy without stopping releases, and what is the way back at each step if it goes wrong.
14 -
15 - Companion to [HARDWARE-FW13.md](HARDWARE-FW13.md), which records what the machine is, and to [IMAGE.md](IMAGE.md), which records how the image is built. The migration ordering and the two genuinely unknown hardware facts come from wiki `alloy-fw13-migration`. The build-host gap list is GoingsOn alloy task `c4eb6d5e`.
5 + Companion to [HARDWARE-FW13.md](HARDWARE-FW13.md), which records what the machine is, and to [IMAGE.md](IMAGE.md), which records how the image is built. The track this belongs to is wiki `alloy-fleet-cutover`; the hardware measurements are wiki `alloy-fw13-migration`. The cut itself is GoingsOn infra `19a82995`.
16 6
17 7 ## Read this first
18 8
19 - fw13 is not only a laptop being reinstalled. It is the tree's only x86_64 build host, and that role is four jobs:
9 + **This is a destructive swap on the machine's only drive.** Ruled by Max 2026-09-03: standard swap, current drive, no replacement NVMe, no shrink, no dual boot, no partition kept bootable. Pop!_OS does not survive it and there is no reboot that brings it back. Shrinking `nvme0n1p3` or booting an external root are not options to weigh; they are mistakes.
10 +
11 + **What replaces the rollback is the data hot potato.** Machines are cut one at a time and each is restored to usable before the next is touched, so the machine that is up holds what the machine being cut cannot lose. For fw13 that machine is astra, still on Pop. Phase 1 is the act of putting things there, and it is the only safety net this operation has.
12 +
13 + **fw13 is not a laptop being reinstalled.** It is the tree's only x86_64 build host, and that role is four jobs:
20 14
21 15 1. Bento's local x86_64 builder.
22 16 2. Bento's controller for every other host, which reaches astra, mbp and windows-x86 over rsync.
23 17 3. The crates.io publish host for eleven registered crates.
24 18 4. Sando's controller and the MNW build host, with a local PostgreSQL cluster the deploy gates restore into.
25 19
26 - A machine that boots Alloy and cannot do those four is not a migration. It is an outage that lasts until somebody notices. So the runbook is ordered so that the build-host role is proven on the Alloy side before the machine stops being the build host, and so that nothing irreversible happens until a working Alloy install has been daily-driven.
27 -
28 - The one Alloy artifact fw13 alone can produce is the x86_64 ISO. Mint it before the machine goes offline.
20 + A machine that boots Alloy and cannot do those four is not a migration. It is an outage that lasts until somebody notices. So each of the four is proven inside the built image, as a container on the running Pop install, before the disk is touched. That is Phase 0, and it is what makes the cut cheap: four passes or fails rather than four opinions.
29 21
30 22 ## What is measured and what is not
31 23
32 24 Hardware figures are in HARDWARE-FW13.md and were measured from the machine's own running Pop!_OS. **Alloy has never run on this machine.** The display seed, amdgpu behaviour under Alloy's kernel, glyph crispness at 1.75x, s2idle drain on AMD and fingerprint enrollment are all unobserved.
33 25
34 - The build-host gaps in Phase 0 were measured by probing the built `alloy:clip-client` and `alloy:clip-server` images and by reading the recipes and daemon configs that drive releases. Items marked as a reading were inferred from binary strings or from how a tool works, not from a run.
26 + Disk, measured 2026-09-03: `nvme0n1` is 1.8T, `/` is 94% full with 107G free, and `~/Code` alone is 916G, nearly all `target/` directories and caches.
35 27
36 28 Nothing in this file has been executed. It is a plan.
37 29
@@ -39,208 +31,109 @@
39 31
40 32 | Phase | Where | Reversible |
41 33 |---|---|---|
42 - | 0. Close the build-host gaps, mint the media | Ahead, no machine time | Yes, nothing touched |
43 - | 1. No-write boot test | At the machine, one reboot | Yes, nothing written |
44 - | 2. Disk decision | Ahead, then at the machine | The decision, yes. The shrink, no. |
45 - | 3. Install alongside Pop | At the machine | Yes, Pop still boots |
46 - | 4. Two-week verdict window | Daily use | Yes, boot back to Pop |
47 - | 5. Reclaim `nvme0n1p3` | At the machine | **No** |
34 + | 0. Prove the four roles in containers | Ahead, no downtime | Yes, nothing touched |
35 + | 1. Empty the machine | Ahead, no downtime | Yes, nothing touched |
36 + | 2. Mint the medium | Ahead, one long build | Yes |
37 + | 3. No-write boot test | One reboot | Yes, nothing written |
38 + | 4. The cut | At the machine | **No** |
39 + | 5. Restore, then prove a release | At the machine, then daily | n/a |
48 40
49 - ## Phase 0: everything that can be done ahead
41 + Phases 0 through 3 leave the working machine exactly as it was. Everything irreversible is in phase 4, and phase 4 is one step.
50 42
51 - None of this needs the machine to be down. All of it should be finished before Phase 3.
43 + ## Phase 0: prove the four roles, in containers, on today's Pop
52 44
53 - ### 0.1 Close the six blockers
45 + Build the image with `build/build-image.sh --host fw13`, which carries `LANGS=rust,js` and `DB=postgres16` (see IMAGE.md, "Per-machine mint recipes"). Then run each role inside it with the tree bind-mounted. `distrobox` 1.8 is unconditional in both profiles and mounts the host home, so `~/Code` paths and `target/` directories are identical inside and out.
54 46
55 - Probed against the built image. Each of these stops the build-host role dead rather than merely degrading it.
47 + **Role 1, compile.** `cargo test` for one crate under `~/Code` completes inside the image, and `rustup toolchain list` shows the pinned 1.97.1 active. The image declines rustup and 28 repos pin a toolchain only rustup honours; Fedora 43's rust RPM happens to be 1.97.1 today, so nothing breaks now and nothing would warn later. rustup goes into `$HOME`, which is a machine step rather than an image change (infra `6faa22a6`).
56 48
57 - **Three of the six are closed.** rsync and minisign are both in the base package block, and the database ships as a build-time dial. Their entries stay below because the reasoning is worth keeping; read them as requirements, not as work.
49 + **Role 2, the Tauri release.** An AppImage built through a Debian-family distrobox has a highest referenced `GLIBC_` symbol no higher than the AppImage released today, measured at the time rather than assumed (infra `91b1b311`). The reason it cannot be built natively is the bundled libraries, not the compiled executable: cargo-tauri copies the WebKit libraries into the bundle and linuxdeploy excludes libc from what it bundles, so those copies meet the user's glibc. Fedora 43's `libwebkit2gtk-4.1` (2.52.5-1.fc43) requires GLIBC_2.42; Ubuntu 24.04 LTS is 2.39. Measured separately: a Rust executable's own floor is set by which symbols the code uses, not by the build host, and comes out at GLIBC_2.39 on both hosts from `pidfd_getpid` and `pidfd_spawnp`.
58 50
59 - **rsync must be present on both profiles.** `ops-exec` spawns the `rsync` binary directly for every push and pull, and bentod, the bento driver and sandod all link it. Without it, every Bento collect from astra, mbp and windows-x86 fails and Sando's release handoff fails. This is the control plane, not a build tool: fw13 stops being able to act as a controller at all, whether or not it can compile anything. Mechanical fix, one word in the base package block. Fedora 43 carries 3.4.4.
51 + **Role 3, the Sando gate.** A production dump restores into a local postgres 16 cluster over a socket at `/var/run/postgresql`, and `code_smoke`'s two npm builds complete. The socket detail decides the size of this: the deployed `/etc/sando/sando-daemon.toml:26` uses `postgres:///sando_scratch?host=/var/run/postgresql&user=sando` while the in-repo template uses a TCP URL, and the deployed file is what the gate runs against. Reading the template instead is how this gets mis-sized as client-only. The node half is unverified: Fedora 43 carries node 22.22.2 against fw13's current 24.18.0, and neither frontend declares an `engines` field. `code_smoke` is the one place a frontend failure is fatal, because the server's `build.rs` deliberately is not; with no node it warns and ships whatever `static/dist/` already holds.
60 52
61 - **minisign must be present.** Both Linux release recipes call `dist/sign-artifacts.sh`, which hard-fails when minisign is not on PATH, and the recipe step aborts the run. Mechanical fix, one word in the base package block. Fedora 43 carries 0.12. Base rather than client: signing is a build-host act.
53 + **Role 4, the publish.** Bento's publish preflight for one makeover crate runs to the point of upload and stops. It exercises `cargo fmt --all --check`, which is where a toolchain that is not the pinned one first shows up; the annotated `v<version>` tag the preflight builds from rather than the branch; and minisign on PATH, since both Linux release recipes call `dist/sign-artifacts.sh` and it hard-fails without it.
62 54
63 - **sccache is absent, and that is not fatal.** `rustc-wrapper` is out of the tracked cargo config and `RUSTC_WRAPPER` is exported per machine from the shell profile, which is the only conditional cargo offers; setting `rustc-wrapper` in the tracked file is fatal at the version probe before any compilation starts, so a machine without sccache would fail every cargo command under `~/Code`. A machine without sccache builds cold. sccache is a cargo-installed binary in `~/.cargo/bin` rather than a system package, so restoring it is a `$HOME` step rather than an image change, and it is worth restoring: measured on fw13, 18273 cached objects over 13G and a 71% hit rate on a controlled cross-project test.
55 + ### What the image already carries, and what it does not
64 56
65 - **node and npm are absent from a default mint, and `LANGS=rust,js` is what supplies them.** Sando's `code_smoke` gate builds two frontends with `npm run build` before it creates a database or boots anything, and the config text says outright that this is the one place a frontend failure is fatal. It is fatal there because the MNW server's `build.rs` deliberately is not: with no node it emits a warning and lets the Rust build succeed against whatever `static/dist/` already holds, which would silently ship the previous build's JavaScript. `LANGS` defaults to empty, so a stock mint builds no language layer at all and a machine in the build-host role names `LANGS=rust,js` explicitly. What LANGS a given machine starts from is the builder's choice, not a default this file sets. The arm installs with `install_weak_deps=False`, which leaves `nodejs-full-i18n` behind; Fedora 43's nodejs is 22.22.2 against fw13's current 24.18.0, and neither frontend declares an `engines` field, so it is still unverified that the two bundles build under 22.
57 + Probed against the built `alloy:clip-client` and `alloy:clip-server`. Read these as requirements rather than as work: `rsync` and `minisign` are both in the base package block, `xdg-mime` and `xdg-open` are in the unconditional base rather than the client branch, and `fprintd` with `fprintd-pam` are in the client branch with `pam_fprintd` asserted `sufficient` in `system-auth`. The database and the language toolchains are the mint-time dials Phase 0 uses.
66 58
67 - **PostgreSQL is a build-time dial, `ARG DB`, and a build host asks for it.** Sando's `migration_dry_run` and `cargo_test` gates restore prod dumps into a local scratch cluster over the unix socket. `DB=none` is the default and asserts that neither `psql` nor `postgres` is in the image; `DB=postgres16` installs `postgresql16` and `postgresql16-server` and asserts both binaries are on PATH at major 16. The dial is validated alongside `PROFILE`, `LANGS` and `TRIM`, so a typo fails the build rather than producing an image quietly missing a database.
59 + Three that stay open and turn into confusing failures rather than clean ones:
68 60
69 - There is no version problem to solve. Fedora 43 carries versioned packages, `postgresql16` and `postgresql16-server`, both `16.14-1.fc43` in `updates` (`dnf -q list --available 'postgresql16*'` inside `localhost/alloy:clip-client`); production is 16.14, fw13 is 16.14 and astra is 16.13, so the image can carry production's exact version to the patch release. Name `postgresql16-server` rather than the unversioned `postgresql-server`, which pulls 18.
61 + - **`libfuse.so.2` is absent**, so the vendored appimagetool cannot self-mount for the audiofiles build. Measured that `APPIMAGE_EXTRACT_AND_RUN=1` on that exact tool succeeds and writes a valid AppImage, so the fix is one environment variable in `dist/build-appimage.sh`, not a package.
62 + - **`patchelf` is needed in neither the image nor a container.** linuxdeploy ships its own inside its AppImage and resolves it from the extracted AppDir. What does need doing is unrelated: that path fetches linuxdeploy and its plugins from GitHub on every run, so an offline or rate-limited fw13 cannot release, on Alloy or on Pop (infra `83ee4a38`).
63 + - **`sccache` is a `cargo install` in `~/.cargo/bin`**, not a system package, so restoring it is a `$HOME` step. Worth restoring: 18273 cached objects over 13G and a 71% hit rate on a controlled cross-project test.
70 64
71 - The dial is orthogonal to `PROFILE`: it does not hang off the server profile, so whichever way the server-profile question lands, a build host mints with `DB=postgres16` and every other mint pays nothing. Creating the cluster and its socket is the machine's business, not the image's.
65 + ## Phase 1: empty the machine
72 66
73 - **One thing to get right when that lands.** The restore really does go over a unix socket, and the two Sando configs disagree about it. The deployed `/etc/sando/sando-daemon.toml:26` is `postgres:///sando_scratch?host=/var/run/postgresql&user=sando`, while the in-repo template at `MNW/sando/daemon/sando-daemon.toml:22` is `postgres://sando@127.0.0.1/sando_scratch`. The deployed file is what the gate runs against, so the image needs a local server with a socket at `/var/run/postgresql`, not merely a client that can reach a port. Reading the template instead of the deployed file is how this gets sized as a client-only problem.
67 + Nothing here needs downtime, and after it there is nothing on fw13 that exists only on fw13.
74 68
75 - **The Tauri Linux build environment is absent.** goingson and balanced_breakfast release through `cargo tauri build`, which needs webkit2gtk-4.1, gtk3, libsoup3 and javascriptcoregtk at build time. fw13 carries the whole set as host packages today. The image carries none of them and its pkgconfig directory has no matching files, so the link step cannot resolve. Largest single gap by weight, and the one most obviously suited to a container rather than to the image.
69 + - **Every repo pushed** to the remotes its policy allows. An unpushed commit is the single most losable thing in this operation. `wright` goes to astra and srht and never to mnw.
70 + - **`clone-tree.sh` completes into a scratch root on a machine that is not fw13.** A dry run on the machine being wiped proves less than it looks like.
71 + - **The `$HOME` credential material confirmed on two other machines.** Release signing keys, updater keys and registry credentials live in `$HOME` and in no repo; astra holds byte-identical copies today and fw12 takes a third under the fleet plan. `settings.local.json` sits outside `_private` because it carries live tokens (infra `7f373187`).
72 + - **`~/.bashrc` translated or kept.** It carries the daemon tokens and the PATH the build-host role depends on, and Alloy's default shell is nushell. bash is in the image, so keeping a bash login shell makes this a non-event and taking nushell makes it one file (infra `e4ddabbf`).
73 + - **The cargo-installed tool set recorded** in `_private/infra/bootstrap`, which lists repos and the cargo config and stops there. From `~/.cargo/.crates2.json`, eleven entries: `cargo-audit`, `cargo-deny`, `cargo-fuzz`, `cargo-mutants`, `cargo-sweep`, `samply`, `sccache`, `sqlx-cli` (built `--no-default-features --features postgres`, so reinstalling from the bare name silently produces a different binary), `tauri-cli`, and the two path installs `spag-mcp` and `spaghetti`, which follow the clone rather than precede it.
76 74
77 - ### 0.2 Decide, do not reflexively add
75 + ## Phase 2: mint the medium
78 76
79 - Three of the six are selection decisions the role census (alloy task `d77339f0`) has not been asked yet. Each item has three acceptable outcomes, and the third is real:
77 + The image build takes over an hour and can only run on an x86_64 host, which today means fw13 itself. Build and mint before the machine goes down, not during.
80 78
81 - 1. It belongs in the image, because the role is one Alloy claims to serve.
82 - 2. It belongs in a distrobox container with an exported wrapper.
83 - 3. It is a workflow change that gets absorbed, as the absence of tmux already is.
79 + - `build/build-iso.sh --host fw13`, on `main`, and record the commit it was minted from. A boot test against a stale image answers a question nobody asked.
80 + - Write it with `build/write-device.sh`. **Standing drive rule: the SanDisk is the install-media and scratch drive, the Samsung T9 is never a write target.**
81 + - Treat whatever is already on the SanDisk as unknown until it is plugged in and read.
82 + - The `rust-stage` witchbroom cell is live on astra and guards the failure class that bites here: a `SHOP_REV` bump silently adding a system build dependency and taking the image build with it.
84 83
85 - Two notes on the container answer. distrobox 1.8 is unconditional in both profiles and mounts the host home, so `~/Code` paths and `target/` directories are identical inside and out, and Bento's collect finds artifacts where it expects them. That makes it a genuine answer for node, postgres and the webkit deps. Against it: bentod's topology declares fw13 with `ssh = "local"`, so it runs cargo on the host itself, and routing those into a box needs a Bento change rather than a config edit. Pointing the other way, an AppImage links its build host's glibc, and Fedora 43's is newer than Ubuntu 24.04's, so building natively on Alloy would raise the glibc floor for everyone who downloads a release while building in a Debian-family box keeps it where it is.
84 + ## Phase 3: the no-write boot test
86 85
87 - **The executable's glibc floor is not set by the build host.** Measured by compiling the same source on both hosts and reading `objdump -T`. The Alloy image is glibc 2.42 (`glibc-2.42-13.fc43`) against fw13's 2.39. A Rust binary exercising fs, threads, time, net and collections floors at GLIBC_2.34 built either place, identically. Add process spawning, which is what pulls the newest symbols the tree actually uses, and both hosts produce GLIBC_2.39 from the same two symbols, `pidfd_getpid` and `pidfd_spawnp`. The deployed `makenotwork` binary built on fw13 already requires GLIBC_2.39 for exactly those two, plus `hypotf` at 2.35. The floor is set by which symbols the code uses and by what rustc's feature detection reaches for.
88 -
89 - **The AppImage's floor is not the executable's floor, and that is what decides the item.** cargo-tauri copies the WebKit libraries into the bundle and linuxdeploy deliberately excludes libc from what it bundles, so those copied libraries meet whatever glibc the *user* has. Measured by installing `webkit2gtk4.1` (2.52.5-1.fc43) into a throwaway container off the client image and reading `objdump -T`: **`libwebkit2gtk-4.1` requires GLIBC_2.42**, which is Fedora 43's own. `libjavascriptcoregtk-4.1` requires 2.38 and is not the constraint. fw13's WebKit is 2.52.3 from Ubuntu 24.04 and carries no such requirement.
90 -
91 - An AppImage built natively on Alloy would carry a WebKit that will not load on Ubuntu 24.04 LTS, which is glibc 2.39 and is the current LTS. Not a weight preference and not a matter of taste: the release would fail to start for the people most likely to download it. **The Tauri build belongs in a Debian-family distrobox**, and the argument is the bundled libraries rather than the compiled executable.
92 -
93 - Two limits, stated so nobody over-reads the Rust half. It measures Rust std only, so a C dependency compiled by `cc` against Fedora 43 headers could still pick up a newer symbol, and none of `openssl-sys` or `libsqlite3-sys` was tested. And the WebKit measurement is of the library Fedora ships, not of a bundle produced by an actual `cargo tauri build` on Alloy, which has not been run.
94 -
95 - ### 0.3 The six that degrade rather than block
96 -
97 - Worth closing in the same pass, since each one turns into a confusing failure during a release rather than a clean one.
98 -
99 - - **libfuse.so.2 is absent**, so the vendored appimagetool cannot self-mount for the audiofiles build. Measured on fw13 that setting `APPIMAGE_EXTRACT_AND_RUN=1` on that exact tool succeeds and writes a valid AppImage, so the fix is one environment variable in `dist/build-appimage.sh`, not a package. Tauri's own AppImage path already sets it.
100 - - **xdg-mime and xdg-open are present on both profiles.** tauri-bundler references them at hard-coded `/usr/bin` paths, so a build host needs them whatever profile it runs; `xdg-utils` sits in the unconditional base package block rather than in the client block, and headless is the build-host profile. Whether the server profile exists as a product is task `d1fed0d7`'s call, and the bundler does not bear on it. What stays client-side is the mime database the desktop half resolves against, which the bundler does not need.
101 - - **patchelf is needed in neither the image nor a container.** linuxdeploy ships its own patchelf inside its AppImage and resolves it from the extracted AppDir, never from the host. Extracting `~/.cache/tauri/linuxdeploy-x86_64.AppImage` puts `linuxdeploy`, `linuxdeploy-plugin-appimage`, `patchelf` and `strip` in `squashfs-root/usr/bin/`, and running it under a `/usr/bin` symlink farm with patchelf excluded prints `DEBUG: Using patchelf: /tmp/appimage_extracted_.../usr/bin/patchelf` and completes. The gtk plugin never calls it at all; `grep -n patchelf ~/.cache/tauri/linuxdeploy-plugin-gtk.sh` exits 1 over 326 lines.
102 -
103 - What does need doing is unrelated to patchelf: that build path fetches linuxdeploy and its plugins from GitHub on every run, so an offline or rate-limited fw13 cannot release, on Alloy or on Pop. That moved to infra `83ee4a38` and is tracked there.
104 - - **The image declines rustup, and 28 repos pin a toolchain only rustup honours.** Fedora 43's rust RPM happens to be the pinned 1.97.1 today, so nothing breaks now. The RPM has no mechanism to read the pin, so the agreement is coincidence rather than a control: the day Fedora moves, every build in the tree silently runs on a toolchain the repos did not ask for, and the first thing to notice is `cargo fmt --all --check`, a gate in every publish recipe.
105 - - **The signing material is `$HOME`-only.** Release signing keys, the updater keys and the registry credentials live in `$HOME` and in no repo. Verified that astra holds byte-identical copies, so a wipe loses nothing, and verified that the bootstrap README covers repos and the cargo config and stops there. The material survives and nobody has written down where it survives. Fix that before Phase 3, in `_private`, not here.
106 - - **`~/.bashrc` carries the daemon tokens and the PATH the role depends on**, and Alloy's default shell is nushell. bash is in the image, so keeping a bash login shell makes this a non-event and taking nushell makes it one file to translate. Worth naming because `~/.bashrc` is load-bearing here and unbacked.
107 -
108 - ### 0.4 Mint the media, and prove the image builds
109 -
110 - The image build takes over an hour and can only be done on an x86_64 host, which today means fw13 itself. Build and mint before the machine goes down, not during.
111 -
112 - - The `rust-stage` witchbroom cell is live on astra, and it guards the failure class that bites here: a `SHOP_REV` bump silently adding a system build dependency and taking the image build with it. It stops that failure rather than working around it, which is why it stays written down.
113 - - Build the image with whatever Phase 0.1 decided, on `main`. For the build-host role that is `LANGS=rust,js` and `DB=postgres16`, both named explicitly, since `LANGS` defaults to empty and `DB` to `none`.
114 - - Write the ISO to the SanDisk. **Standing drive rule: the SanDisk is the install-media and scratch drive, the Samsung T9 is never a write target.**
115 - - Record which commit the medium was minted from. A boot test against a stale image answers a question nobody asked.
116 -
117 - **Do not assume the SanDisk already carries a usable installer image.** There is no `output/*.iso` on fw13; `output/` holds `disk.raw` and `manifest-raw.json`, and the only ISO in the tree is `output.prev/install.iso`, a previous run's artifact and not the mint of record for anything. The drive's contents cannot be checked while it is unplugged.
118 -
119 - Practical effect on Phase 1: plan on an image build and a fresh mint, and treat any content already on the SanDisk as unknown until the drive is plugged in and read. The drive rule is unchanged by this. The SanDisk is the install-media and scratch drive and the one that gets written; the T9 is never a write target, whatever is or is not on the SanDisk.
120 -
121 - ### 0.5 Make the rebuild real
122 -
123 - `_private/infra/bootstrap/` already holds `repos.toml`, `clone-tree.sh` and `generate-manifest.sh`, and the cargo config is a tracked file with a symlink at the live path. Rebuilding the tree is cloning `_private` from astra and running one script. Before Phase 3:
124 -
125 - - Regenerate the manifest and commit it, so it describes the tree as it is on the day of the cutover.
126 - - Push every repo to the remotes its policy allows. An unpushed commit on fw13 is the single most losable thing in this whole operation.
127 - - Extend the bootstrap README to cover the `$HOME` material from 0.3, or at minimum to say where it lives.
128 - - Dry-run `clone-tree.sh` into a scratch root and confirm it still completes.
129 - - Write down the cargo-installed tool set, which nothing in this file or in
130 - `_private/infra/bootstrap/` currently lists. It is `$HOME` material of the same class as
131 - the signing keys in 0.3: it survives a wipe nowhere, and it is not one of the things
132 - `clone-tree.sh` rebuilds. Read from `~/.cargo/.crates2.json`, eleven entries:
133 - `cargo-audit`, `cargo-deny`, `cargo-fuzz`, `cargo-mutants`, `cargo-sweep`, `samply`,
134 - `sccache`, `sqlx-cli` (built `--no-default-features --features postgres`), `tauri-cli`,
135 - and the two path installs `spag-mcp` and `spaghetti`, which come from the tree rather than
136 - from crates.io and so have to follow the clone rather than precede it.
137 -
138 - Three of these are load-bearing in a way worth naming. `sccache` is blocker 0.1: nothing
139 - under `~/Code` compiles until it is back, so it is the first install after the toolchain
140 - and before the first build. `tauri-cli` is what 0.2's Tauri question is actually about on
141 - the cargo side, and it is already a `cargo install` rather than a system package, so
142 - whatever 0.2 decides about the webkit libraries does not move this one. `sqlx-cli` carries
143 - a non-default feature set, so reinstalling it from the bare name silently produces a
144 - different binary.
145 -
146 - ## Phase 1: the no-write boot test
147 -
148 - One reboot, at the machine, nothing written.
149 -
150 - Boot the SanDisk medium and stop at the disk selection screen. Answer three questions and then power off:
86 + One reboot, nothing written. Boot the medium, stop at the disk selection screen, answer three questions, power off.
151 87
152 88 1. Does amdgpu bring up the 2880x1920 panel under Alloy's kernel?
153 - 2. What does the display generator actually propose? HARDWARE-FW13.md predicts 1.75x for the panel and 1.0x for the BenQ RD280U on DP-3, from EDID arithmetic. Neither has been observed.
154 - 3. Does 1.75x look crisp, and does 1.0x on a 163 PPI 28 inch panel look right at desk distance? The FW12's 1.25x got that judgment and fw13's rungs have not.
89 + 2. What does the display generator propose? HARDWARE-FW13.md predicts 1.75x for the panel and 1.0x for the BenQ RD280U on DP-3, from EDID arithmetic. Neither has been observed, and `PPI_PER_SCALE` in `crates/alloy/src/display.rs` has exactly one data point behind it.
90 + 3. Does 1.75x look crisp, and does 1.0x on a 163 PPI 28 inch panel look right at desk distance?
155 91
156 - **Way back:** power off and remove the medium. Nothing was written.
92 + **Way back:** power off and remove the medium. This is the last phase where a way back exists.
157 93
158 - This phase is free and can happen any evening. Separating "does Alloy run on this hardware" from "where does it live" is the whole trick, and it is why the migration felt expensive for as long as those were one step.
94 + ## Phase 4: the cut
159 95
160 - ## Phase 2: the disk decision
96 + Wipe `nvme0n1`, install, unlock, log in. The encrypted path is proven on fw12 hardware (alloy `6ef745fd`), not on this machine.
161 97
162 - One NVMe device, `nvme0n1`, WD BLACK SN7100 2TB in the M.2 2280 slot. `nvme0n1p3` is a single 1.8T ext4 spanning everything after `/recovery`, with 798 G free and zero unpartitioned space. **There is no free second M.2 slot.** The 2230 socket exists and carries the MediaTek MT7925 Wi-Fi module. The only PCIe root ports with nothing behind them are the USB4 ones.
98 + After this step Pop!_OS is gone. Everything that made it recoverable was Phase 1.
163 99
164 - So there are two options.
100 + ## Phase 5: restore, then prove a release
165 101
166 - **A. Offline shrink of `nvme0n1p3`.** Boot live media, shrink the ext4 filesystem and its partition, install Alloy into the freed space, dual-boot. Keeps Pop!_OS bootable for the entire verdict window, which is what makes Phase 4 reversible. Costs one long offline window and carries the ordinary shrink risk: an interrupted resize on the partition holding everything.
102 + 1. Clone `_private` from astra, run `clone-tree.sh`.
103 + 2. Restore the `$HOME` material and the cargo tool set. sccache first: nothing under `~/Code` compiles pleasantly without it.
104 + 3. Build one crate and confirm the toolchain answers.
105 + 4. **Run one real release end to end**: a makeover publish is the smallest, and a goingson AppImage is the one that exercises the whole gap list.
106 + 5. Only then point bentod and sandod at this machine.
167 107
168 - **B. External root over USB4.** Install Alloy to an external SSD and boot from it. No shrink, no risk to the internal disk, and it leaves Pop untouched. Costs a drive the tree does not currently have free, since the SanDisk is install media and the T9 is never written. External-root boot and daily performance on this machine are unmeasured.
108 + Until step 5, fw13 is a machine that boots Alloy. After it, fw13 is the x86_64 build host again (infra `28cd7d0c`).
169 109
170 - Whichever is chosen, before the shrink: re-measure free space (798 G is an August figure), take a fresh backup of anything not in git or on astra, and confirm the recovery partition still boots.
171 -
172 - **Way back:** if the shrink fails, Pop is restored from backup onto the same disk. This is the first phase where "way back" means restore rather than reboot.
173 -
174 - ## Phase 3: install, without giving up the build host
175 -
176 - Install Alloy into whatever Phase 2 produced. Pop!_OS remains installed and bootable.
177 -
178 - Order inside the phase:
179 -
180 - 1. Install, unlock, log in. The encrypted path is proven on fw12 hardware, not on this machine.
181 - 2. Clone `_private` from astra, run `clone-tree.sh`, restore the `$HOME` material from astra.
182 - 3. Restore or drop sccache, per the 0.1 decision.
183 - 4. Build one crate under `~/Code` and confirm the toolchain answers.
184 - 5. Run one full release end to end on the Alloy side before trusting it: a makeover publish is the smallest, and a goingson AppImage is the one that exercises the whole gap list.
185 - 6. Only then point the daemons at the Alloy install.
186 -
187 - Until step 6, **releases still run from Pop!_OS.** That is the property this whole ordering exists to preserve. Booting back into Pop is a reboot, and everything the build host needs is still installed there.
188 -
189 - **Way back:** reboot into Pop. Cost is one reboot and whatever work was done on the Alloy side since the last push.
190 -
191 - ## Phase 4: the verdict window
192 -
193 - Daily-drive Alloy for two weeks, the same window `wright` Phase 1 uses. What this window is actually testing, beyond preference:
194 -
195 - - The build-host role, under real releases rather than one rehearsal.
196 - - s2idle drain on AMD, baseline against tuned.
197 - - Multi-output reconcile with the laptop panel plus the BenQ, which no machine has exercised.
198 - - fprintd enroll on the Goodix part, then unlock at swaylock and at a `run0` polkit prompt.
199 - - The dark-mode L stops on this panel, as a second reading against the FW12's.
200 -
201 - The migration also pays Alloy back here. fw13 on Pop runs systemd 255 and `run0` needs 256, which is why alloy task `3587c247` has been parked on needing a host. Fedora 43 clears it.
202 -
203 - **Way back:** reboot into Pop, and either keep the dual boot or reclaim the Alloy partition. Nothing is lost that was pushed.
204 -
205 - ## Phase 5: reclaim `nvme0n1p3`
206 -
207 - Only after Phase 4 returns a verdict, and only after the bootstrap rebuild has been exercised for real rather than dry-run.
208 -
209 - This is the irreversible step. After it, Pop!_OS is gone and there is no reboot that brings the old build host back.
110 + Then the machine gets daily-driven, which is where the rest of the reading happens: s2idle drain on AMD, multi-output reconcile with the laptop panel plus the BenQ, fprintd enroll on the Goodix part, and the dark-mode L stops on this panel as a second reading against the FW12's. The migration also pays Alloy back here: fw13 on Pop runs systemd 255 and `run0` needs 256, which is why alloy `3587c247` sat on needing a host.
210 111
211 112 ## What is not recoverable
212 113
213 - Everything below is lost at Phase 5, and some of it is lost earlier if the Phase 2 shrink goes wrong. Check each one before, not after.
114 + All of it is lost at Phase 4. Check each one in Phase 1, not after.
214 115
215 116 - **Unpushed commits in any repo.** The bootstrap manifest rebuilds the tree from the remotes. It cannot rebuild what was never pushed.
216 - - **Anything in `$HOME` that is not in a repo and not mirrored to astra.** Signing keys and updater keys are mirrored, verified. Shell history, browser profiles, local scratch and anything else is not.
217 - - **The Pop-side build environment itself.** The webkit, gtk, libsoup, patchelf and FUSE packages that make releases work today are apt packages on that install. Once p3 is gone, the fallback is gone with it, so the Alloy side has to be proven first rather than proven afterwards.
218 - - **Local caches.** The podman image cache, `target/` directories and the sccache cache all regenerate from source. They are not backed up on purpose, and the cost of losing them is an hour or more of rebuild, once.
117 + - **Anything in `$HOME` that is not in a repo, not mirrored to astra, and not a Syncthing folder.** Max, 2026-09-03: the machines hold no personal information worth keeping, so the exceptions are the credential material and nothing else. Shell history, browser profiles and local scratch go.
118 + - **The Pop-side build environment itself.** The webkit, gtk, libsoup and FUSE packages that make releases work today are apt packages on that install. Once it is gone the fallback is gone, which is why Phase 0 proves the Alloy side first.
119 + - **Local caches.** The podman image cache, `target/` directories and the sccache cache all regenerate from source. Not backed up on purpose.
219 120
220 121 ## Ahead of time, versus at the machine
221 122
222 - Split deliberately, because machine time is the scarce half.
123 + Machine time is the scarce half, so the split is deliberate.
223 124
224 - **Can be done ahead, with fw13 fully in service:** every item in Phase 0. Closing the blockers, the selection decisions, the image build, minting the medium, regenerating the manifest, pushing every repo, writing down where the `$HOME` material lives, and the dry-run rebuild. The sweep cell on astra was on this list and is done.
125 + **Ahead, with fw13 fully in service:** all of Phase 0, 1 and 2.
225 126
226 - **Has to happen at the machine:** the Phase 1 boot test and its three judgments, the Phase 2 shrink, the Phase 3 install and unlock, fingerprint enrollment, and the display and suspend readings that only exist under a running session.
127 + **At the machine:** the Phase 3 boot test and its three judgments, the Phase 4 install and unlock, and the display and suspend readings that only exist under a running session.
227 128
228 - **Happens over time and cannot be compressed:** Phase 4.
129 + **Over time, and not compressible:** the daily driving in Phase 5.
229 130
230 131 ## astra is not covered here
231 132
232 - This runbook covers fw13 and only fw13. There is no equivalent for astra, and the gap is worth naming so nobody reads a converted fw13 as a converted tree.
233 -
234 - Where an astra runbook starts is the astra role census: the same exercise Phase 0.2 runs for fw13, listing what the machine is actually asked to do before anyone decides what an image has to carry. Without it, the selection discipline has nothing to select against and every missing package becomes an argument for adding it.
235 -
236 - Converting astra is a different problem from converting fw13, and a larger one. fw13 is a build host: its state is caches, checkouts and `$HOME` material, and the caches regenerate. astra is a service host with state. It runs the git mirrors, the sweep, aarch64 builds and CI, and it is the machine the fw13 plan itself leans on for restore. Downtime there is not one person's laptop being unavailable for an evening, and the way-back at each step cannot be "reboot into the old install" the way it can here.
237 -
238 - Detail and the sequencing live in wiki `alloy-astra-migration`. Do not try to answer astra's questions inside this file.
133 + This runbook covers fw13 and only fw13. astra is a service host with state, it is cut first and fixed after, and its ordering is wiki `alloy-fleet-cutover` and wiki `alloy-astra-migration`. Do not try to answer astra's questions inside this file.
239 134
240 135 ## Open questions this runbook does not answer
241 136
242 - - What prod runs for PostgreSQL, which decides how much of the version mismatch matters.
243 137 - Whether bentod grows a distrobox transport, or whether the build deps go in the image (`d77339f0`, `c4eb6d5e`).
244 138 - Whether the server profile is a product at all (`d1fed0d7`), which the build-host role argues for.
245 - - Whether raising the AppImage glibc floor is acceptable, or whether Debian-family builds stay the rule.
246 - - Whether the external-root option in Phase 2 is worth buying a drive for. Nobody has measured USB4 boot on this machine.
139 + - Whether the two MNW frontends build under node 22, which Phase 0 role 3 answers by running them.