max / alloy

docs: fw13 cutover runbook Phase order from the no-write boot test through reclaiming nvme0n1p3, the build-host role fw13 has to keep playing across the cutover, the disk decision now that the 2230 socket is known to carry the Wi-Fi module, and the way back at each phase.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Max Johnson <me@maxj.phd> · 2026-08-20 22:46 UTC

Signed with PGP, not checked

Commit: 4d3120a16e8937f798a632420ddf5ebe2a5fcce6

Parent: 963449e

3 files changed, +190 insertions, -1 deletion

M README.md +1

			@@ -43,5 +43,6 @@
43	43		- [`docs/SHELL.md`](docs/SHELL.md) — Nushell as the login shell; bash unchanged as `/bin/sh` and `/bin/bash`. Architecture, interop rules, accepted costs.
44	44		- [`docs/HARDWARE-FW12.md`](docs/HARDWARE-FW12.md) — Framework Laptop 12 as the first hardware target: 2-in-1 stance, touch/tablet/rotation/OSK/fingerprint decisions.
45	45		- [`docs/HARDWARE-FW13.md`](docs/HARDWARE-FW13.md) — Framework Laptop 13 (AMD Ryzen AI 300) as the second target: measured panel, the 1.75x seed, AMD graphics, what is still unverified.
	46	+	- [`docs/MIGRATION-FW13.md`](docs/MIGRATION-FW13.md) — cutover runbook for moving fw13 off Pop!_OS: phase order, the disk decision, the build-host role it has to keep playing, and the way back.
46	47		- [`docs/IMAGE.md`](docs/IMAGE.md) — Image composition: bootc + Containerfile, source on `makenot.work`, built locally and natively. No image is published anywhere; you build your own.
47	48		- [`docs/todo.md`](docs/todo.md) — open questions and next steps.

M docs/HARDWARE-FW13.md +1 -1

			@@ -1,6 +1,6 @@
1	1		# Hardware target: Framework Laptop 13 (AMD Ryzen AI 300)
2	2
3		-	The second device Alloy is being written against, and the first that is not Intel. This document records what the machine is, what it changes about assumptions the FW12 doc could make, and which of those changes are measured rather than assumed. Companion to [STACK.md](STACK.md), [HARDWARE-FW12.md](HARDWARE-FW12.md), and wiki `alloy-fw13-migration`.
	3	+	The second device Alloy is being written against, and the first that is not Intel. This document records what the machine is, what it changes about assumptions the FW12 doc could make, and which of those changes are measured rather than assumed. Companion to [STACK.md](STACK.md), [HARDWARE-FW12.md](HARDWARE-FW12.md), [MIGRATION-FW13.md](MIGRATION-FW13.md), and wiki `alloy-fw13-migration`.
4	4
5	5		Per-device docs are the pattern the FW12 doc set, and this is the file it predicted.
6	6

A docs/MIGRATION-FW13.md +188

		@@ -1,0 +1,188 @@
1	+	# Cutover runbook: fw13, Pop!_OS to Alloy
2	+
3	+	This file answers one question. What is the order of operations that moves fw13 from Pop!_OS 24.04 to Alloy without stopping releases, and what is the way back at each step if it goes wrong.
4	+
5	+	Companion to [HARDWARE-FW13.md](HARDWARE-FW13.md), which records what the machine is, and to [IMAGE.md](IMAGE.md), which records how the image is built. The migration ordering and the two genuinely unknown hardware facts come from wiki `alloy-fw13-migration`. The build-host gap list is GoingsOn alloy task `c4eb6d5e`.
6	+
7	+	## Read this first
8	+
9	+	fw13 is not only a laptop being reinstalled. It is the tree's only x86_64 build host, and that role is four jobs:
10	+
11	+	1. Bento's local x86_64 builder.
12	+	2. Bento's controller for every other host, which reaches astra, mbp and windows-x86 over rsync.
13	+	3. The crates.io publish host for eleven registered crates.
14	+	4. Sando's controller and the MNW build host, with a local PostgreSQL cluster the deploy gates restore into.
15	+
16	+	A machine that boots Alloy and cannot do those four is not a migration. It is an outage that lasts until somebody notices. So the runbook is ordered so that the build-host role is proven on the Alloy side before the machine stops being the build host, and so that nothing irreversible happens until a working Alloy install has been daily-driven.
17	+
18	+	The one Alloy artifact fw13 alone can produce is the x86_64 ISO. Mint it before the machine goes offline.
19	+
20	+	## What is measured and what is not
21	+
22	+	Hardware figures are in HARDWARE-FW13.md and were measured on 2026-08-11 from the machine's own running Pop!_OS. Alloy has never run on this machine. The display seed, amdgpu behaviour under Alloy's kernel, glyph crispness at 1.75x, s2idle drain on AMD and fingerprint enrollment are all unobserved.
23	+
24	+	The build-host gaps in Phase 0 were measured on 2026-08-20 by probing the already-built `alloy:clip-client` and `alloy:clip-server` images and by reading the recipes and daemon configs that drive releases. Items marked as a reading were inferred from binary strings or from how a tool works, not from a run.
25	+
26	+	Nothing in this file has been executed. It is a plan.
27	+
28	+	## Phases at a glance
29	+
30	+	\| Phase \| Where \| Reversible \|
31	+	\|---\|---\|---\|
32	+	\| 0. Close the build-host gaps, mint the media \| Ahead, no machine time \| Yes, nothing touched \|
33	+	\| 1. No-write boot test \| At the machine, one reboot \| Yes, nothing written \|
34	+	\| 2. Disk decision \| Ahead, then at the machine \| The decision, yes. The shrink, no. \|
35	+	\| 3. Install alongside Pop \| At the machine \| Yes, Pop still boots \|
36	+	\| 4. Two-week verdict window \| Daily use \| Yes, boot back to Pop \|
37	+	\| 5. Reclaim `nvme0n1p3` \| At the machine \| No \|
38	+
39	+	## Phase 0: everything that can be done ahead
40	+
41	+	None of this needs the machine to be down. All of it should be finished before Phase 3.
42	+
43	+	### 0.1 Close the six blockers
44	+
45	+	Probed against the built image on 2026-08-20. Each of these stops the build-host role dead, not merely degrades it.
46	+
47	+	rsync is absent from both profiles. `ops-exec` spawns the `rsync` binary directly for every push and pull, and bentod, the bento driver and sandod all link it. Without it, every Bento collect from astra, mbp and windows-x86 fails and Sando's release handoff fails. This is the control plane, not a build tool: fw13 stops being able to act as a controller at all, whether or not it can compile anything. Mechanical fix, one word in the base package block. Fedora 43 carries 3.4.4.
48	+
49	+	minisign is absent. Both Linux release recipes call `dist/sign-artifacts.sh`, which hard-fails when minisign is not on PATH, and the recipe step aborts the run. Mechanical fix, one word in the base package block. Fedora 43 carries 0.12. Base rather than client: signing is a build-host act.
50	+
51	+	sccache is absent, and the tree-wide cargo config makes it mandatory. `~/Code/.cargo/config.toml` sets `rustc-wrapper = "sccache"` unconditionally. A missing wrapper is fatal at the version probe, before any compilation starts, so on a fresh install every cargo command under `~/Code` dies instantly. Measured by pointing the wrapper at a nonexistent binary. sccache is a cargo-installed binary in `~/.cargo/bin`, not a system package, so this is a `$HOME` restore rather than an image change. The config's own comment says the two lines are removable, so dropping sccache is a real option rather than a forced dependency.
52	+
53	+	node and npm are absent, and no `LANGS` value supplies them. Sando's `code_smoke` gate builds two frontends with `npm run build` before it creates a database or boots anything, and the config text says outright that this is the one place a frontend failure is fatal. It is fatal there because the MNW server's `build.rs` deliberately is not: with no node it emits a warning and lets the Rust build succeed against whatever `static/dist/` already holds, which would silently ship the previous build's JavaScript. The `LANGS` validator accepts only rust, go, python and zig, so this is not a mint-time flag away.
54	+
55	+	No PostgreSQL server or client, and the versions do not match. Sando's `migration_dry_run` and `cargo_test` gates restore prod dumps into a local scratch cluster over the unix socket. Alloy ships neither the server nor `psql`. Two problems, not one: the package is absent, and Fedora 43 offers 18.4 while fw13 and astra both run 16. A PG18 scratch cluster stops testing MNW against the version production runs, which is most of what the gate is for. Confirm what prod runs before sizing this.
56	+
57	+	The Tauri Linux build environment is absent. goingson and balanced_breakfast release through `cargo tauri build`, which needs webkit2gtk-4.1, gtk3, libsoup3 and javascriptcoregtk at build time. fw13 carries the whole set as host packages today. The image carries none of them and its pkgconfig directory has no matching files, so the link step cannot resolve. Largest single gap by weight, and the one most obviously suited to a container rather than to the image.
58	+
59	+	### 0.2 Decide, do not reflexively add
60	+
61	+	Three of the six are selection decisions the role census (alloy task `d77339f0`) has not been asked yet. Each item has three acceptable outcomes, and the third is real:
62	+
63	+	1. It belongs in the image, because the role is one Alloy claims to serve.
64	+	2. It belongs in a distrobox container with an exported wrapper.
65	+	3. It is a workflow change that gets absorbed, as the absence of tmux already is.
66	+
67	+	Two notes on the container answer. distrobox 1.8 is unconditional in both profiles and mounts the host home, so `~/Code` paths and `target/` directories are identical inside and out, and Bento's collect finds artifacts where it expects them. That makes it a genuine answer for node, postgres and the webkit deps. Against it: bentod's topology declares fw13 with `ssh = "local"`, so it runs cargo on the host itself, and routing those into a box needs a Bento change rather than a config edit. Pointing the other way, an AppImage links its build host's glibc, and Fedora 43's is newer than Ubuntu 24.04's, so building natively on Alloy would raise the glibc floor for everyone who downloads a release while building in a Debian-family box keeps it where it is. That last point is a reading about how AppImage works, not a measurement.
68	+
69	+	### 0.3 The six that degrade rather than block
70	+
71	+	Worth closing in the same pass, since each one turns into a confusing failure during a release rather than a clean one.
72	+
73	+	- libfuse.so.2 is absent, so the vendored appimagetool cannot self-mount for the audiofiles build. Measured on fw13 that setting `APPIMAGE_EXTRACT_AND_RUN=1` on that exact tool succeeds and writes a valid AppImage, so the fix is one environment variable in `dist/build-appimage.sh`, not a package. Tauri's own AppImage path already sets it.
74	+	- `PROFILE=server` lacks xdg-mime and xdg-open, which tauri-bundler references at hard-coded `/usr/bin` paths. So the build-host role cannot run on the server profile as it stands, which matters while task `d1fed0d7` is still deciding whether that profile exists as a product. A headless build host is the strongest argument for the server profile, and today it is the profile that cannot build the apps.
75	+	- patchelf is absent, and Tauri's AppImage path runs linuxdeploy's gtk plugin over the bundled libraries, which rewrites RPATHs. Read from the downloader strings in the cargo-tauri binary; the plugin scripts were not read and the path was not run, so the patchelf requirement is a reading. The same finding carries a second consequence: that build path fetches linuxdeploy and its plugins from GitHub every time, so an offline or rate-limited fw13 cannot release, on Alloy or on Pop.
76	+	- The image declines rustup, and 28 repos pin a toolchain only rustup honours. Fedora 43's rust RPM happens to be the pinned 1.97.1 today, so nothing breaks now. The RPM has no mechanism to read the pin, so the agreement is coincidence rather than a control: the day Fedora moves, every build in the tree silently runs on a toolchain the repos did not ask for, and the first thing to notice is `cargo fmt --all --check`, a gate in every publish recipe.
77	+	- The signing material is `$HOME`-only. Release signing keys, the updater keys and the registry credentials live in `$HOME` and in no repo. Verified that astra holds byte-identical copies, so a wipe loses nothing, and verified that the bootstrap README covers repos and the cargo config and stops there. The material survives and nobody has written down where it survives. Fix that before Phase 3, in `_private`, not here.
78	+	- `~/.bashrc` carries the daemon tokens and the PATH the role depends on, and Alloy's default shell is nushell. bash is in the image, so keeping a bash login shell makes this a non-event and taking nushell makes it one file to translate. Worth naming because moving those exports out of the harness config in August quietly made `~/.bashrc` load-bearing and unbacked.
79	+
80	+	### 0.4 Mint the media, and prove the image builds
81	+
82	+	The image build takes over an hour and can only be done on an x86_64 host, which today means fw13 itself. Build and mint before the machine goes down, not during.
83	+
84	+	- Get the `rust-stage` witchbroom cell live on astra (infra `d3c88c6c`). It guards the failure class that has actually been biting: a `SHOP_REV` bump silently adding a system build dependency and taking the image build with it. This is the one item that stops the recurring failure rather than working around it.
85	+	- Build the image with whatever Phase 0.1 decided, on `main`.
86	+	- Write the ISO to the SanDisk. Standing drive rule: the SanDisk is the install-media and scratch drive, the Samsung T9 is never a write target.
87	+	- Record which commit the medium was minted from. A boot test against a stale image answers a question nobody asked.
88	+
89	+	### 0.5 Make the rebuild real
90	+
91	+	`_private/infra/bootstrap/` already holds `repos.toml`, `clone-tree.sh` and `generate-manifest.sh`, and the cargo config is a tracked file with a symlink at the live path. Rebuilding the tree is cloning `_private` from astra and running one script. Before Phase 3:
92	+
93	+	- Regenerate the manifest and commit it, so it describes the tree as it is on the day of the cutover rather than as it was in August.
94	+	- Push every repo to the remotes its policy allows. An unpushed commit on fw13 is the single most losable thing in this whole operation.
95	+	- Extend the bootstrap README to cover the `$HOME` material from 0.3, or at minimum to say where it lives.
96	+	- Dry-run `clone-tree.sh` into a scratch root and confirm it still completes.
97	+
98	+	## Phase 1: the no-write boot test
99	+
100	+	One reboot, at the machine, nothing written.
101	+
102	+	Boot the SanDisk medium and stop at the disk selection screen. Answer three questions and then power off:
103	+
104	+	1. Does amdgpu bring up the 2880x1920 panel under Alloy's kernel?
105	+	2. What does the display generator actually propose? HARDWARE-FW13.md predicts 1.75x for the panel and 1.0x for the BenQ RD280U on DP-3, from EDID arithmetic. Neither has been observed.
106	+	3. Does 1.75x look crisp, and does 1.0x on a 163 PPI 28 inch panel look right at desk distance? The FW12's 1.25x got that judgment and fw13's rungs have not.
107	+
108	+	Way back: power off and remove the medium. Nothing was written.
109	+
110	+	This phase is free and can happen any evening. Separating "does Alloy run on this hardware" from "where does it live" is the whole trick, and it is why the migration felt expensive for as long as those were one step.
111	+
112	+	## Phase 2: the disk decision
113	+
114	+	Measured 2026-08-11: one NVMe device, `nvme0n1`, WD BLACK SN7100 2TB in the M.2 2280 slot. `nvme0n1p3` is a single 1.8T ext4 spanning everything after `/recovery`, with 798 G free and zero unpartitioned space. There is no free second M.2 slot. The 2230 socket exists and carries the MediaTek MT7925 Wi-Fi module. The only PCIe root ports with nothing behind them are the USB4 ones.
115	+
116	+	So there are two options.
117	+
118	+	A. Offline shrink of `nvme0n1p3`. Boot live media, shrink the ext4 filesystem and its partition, install Alloy into the freed space, dual-boot. Keeps Pop!_OS bootable for the entire verdict window, which is what makes Phase 4 reversible. Costs one long offline window and carries the ordinary shrink risk: an interrupted resize on the partition holding everything.
119	+
120	+	B. External root over USB4. Install Alloy to an external SSD and boot from it. No shrink, no risk to the internal disk, and it leaves Pop untouched. Costs a drive the tree does not currently have free, since the SanDisk is install media and the T9 is never written. External-root boot and daily performance on this machine are unmeasured.
121	+
122	+	Whichever is chosen, before the shrink: re-measure free space (798 G is an August figure), take a fresh backup of anything not in git or on astra, and confirm the recovery partition still boots.
123	+
124	+	Way back: if the shrink fails, Pop is restored from backup onto the same disk. This is the first phase where "way back" means restore rather than reboot.
125	+
126	+	## Phase 3: install, without giving up the build host
127	+
128	+	Install Alloy into whatever Phase 2 produced. Pop!_OS remains installed and bootable.
129	+
130	+	Order inside the phase:
131	+
132	+	1. Install, unlock, log in. The encrypted path is proven on fw12 hardware, not on this machine.
133	+	2. Clone `_private` from astra, run `clone-tree.sh`, restore the `$HOME` material from astra.
134	+	3. Restore or drop sccache, per the 0.1 decision.
135	+	4. Build one crate under `~/Code` and confirm the toolchain answers.
136	+	5. Run one full release end to end on the Alloy side before trusting it: a makeover publish is the smallest, and a goingson AppImage is the one that exercises the whole gap list.
137	+	6. Only then point the daemons at the Alloy install.
138	+
139	+	Until step 6, releases still run from Pop!_OS. That is the property this whole ordering exists to preserve. Booting back into Pop is a reboot, and everything the build host needs is still installed there.
140	+
141	+	Way back: reboot into Pop. Cost is one reboot and whatever work was done on the Alloy side since the last push.
142	+
143	+	## Phase 4: the verdict window
144	+
145	+	Daily-drive Alloy for two weeks, the same window `wright` Phase 1 uses. What this window is actually testing, beyond preference:
146	+
147	+	- The build-host role, under real releases rather than one rehearsal.
148	+	- s2idle drain on AMD, baseline against tuned.
149	+	- Multi-output reconcile with the laptop panel plus the BenQ, which no machine has exercised.
150	+	- fprintd enroll on the Goodix part, then unlock at swaylock and at a `run0` polkit prompt.
151	+	- The dark-mode L stops on this panel, as a second reading against the FW12's.
152	+
153	+	The migration also pays Alloy back here. fw13 on Pop runs systemd 255 and `run0` needs 256, which is why alloy task `3587c247` has been parked on needing a host. Fedora 43 clears it.
154	+
155	+	Way back: reboot into Pop, and either keep the dual boot or reclaim the Alloy partition. Nothing is lost that was pushed.
156	+
157	+	## Phase 5: reclaim `nvme0n1p3`
158	+
159	+	Only after Phase 4 returns a verdict, and only after the bootstrap rebuild has been exercised for real rather than dry-run.
160	+
161	+	This is the irreversible step. After it, Pop!_OS is gone and there is no reboot that brings the old build host back.
162	+
163	+	## What is not recoverable
164	+
165	+	Everything below is lost at Phase 5, and some of it is lost earlier if the Phase 2 shrink goes wrong. Check each one before, not after.
166	+
167	+	- Unpushed commits in any repo. The bootstrap manifest rebuilds the tree from the remotes. It cannot rebuild what was never pushed.
168	+	- Anything in `$HOME` that is not in a repo and not mirrored to astra. Signing keys and updater keys are mirrored, verified. Shell history, browser profiles, local scratch and anything else is not.
169	+	- The Pop-side build environment itself. The webkit, gtk, libsoup, patchelf and FUSE packages that make releases work today are apt packages on that install. Once p3 is gone, the fallback is gone with it, so the Alloy side has to be proven first rather than proven afterwards.
170	+	- Local caches. The podman image cache, `target/` directories and the sccache cache all regenerate from source. They are not backed up on purpose, and the cost of losing them is an hour or more of rebuild, once.
171	+
172	+	## Ahead of time, versus at the machine
173	+
174	+	Split deliberately, because machine time is the scarce half.
175	+
176	+	Can be done ahead, with fw13 fully in service: every item in Phase 0. Closing the blockers, the selection decisions, the image build, minting the medium, getting the sweep cell live on astra, regenerating the manifest, pushing every repo, writing down where the `$HOME` material lives, and the dry-run rebuild.
177	+
178	+	Has to happen at the machine: the Phase 1 boot test and its three judgments, the Phase 2 shrink, the Phase 3 install and unlock, fingerprint enrollment, and the display and suspend readings that only exist under a running session.
179	+
180	+	Happens over time and cannot be compressed: Phase 4.
181	+
182	+	## Open questions this runbook does not answer
183	+
184	+	- What prod runs for PostgreSQL, which decides how much of the version mismatch matters.
185	+	- Whether bentod grows a distrobox transport, or whether the build deps go in the image (`d77339f0`, `c4eb6d5e`).
186	+	- Whether the server profile is a product at all (`d1fed0d7`), which the build-host role argues for and which today's server profile cannot serve.
187	+	- Whether raising the AppImage glibc floor is acceptable, or whether Debian-family builds stay the rule.
188	+	- Whether the external-root option in Phase 2 is worth buying a drive for. Nobody has measured USB4 boot on this machine.