Skip to main content

max / everycycle

bom: flag that ZFS root does not survive a bootc OS Unit 1 is now the always-on build host and runs Alloy, which is bootc, which means an ostree root. ostree-on-ZFS is not a supported combination, and there is no ZFS in the Alloy image at all: on Fedora it is an out-of-tree kmod, and an immutable image is the worst place to carry one. Someone following the storage table at install time would have spent bring-up time building a root pool that cannot boot. Not resolved by rewriting the table, because a unit running a conventional distro could still do ZFS root and a customer machine's OS is not decided by this file. The rule is that the OS decides, so the pointer sits in the row and the reasoning sits below it. Storage 2 is untouched: raw XFS for kv-scratch was already the answer and the bandwidth reasoning still stands.
Author: Max Johnson <me@maxj.phd> · 2026-07-25 14:47 UTC
Signed with PGP, not checked
Commit: 79a13b0b850c8fdc7c0c3caefe749422ec2b0af6
Parent: 91df069
1 file changed, +16 insertions, -2 deletions
M docs/hardware/mm-v1-bom.md +16 -2
@@ -13,7 +13,7 @@
13 13 | CPU | AMD Threadripper Pro 7975WX | $3,900 | 32 cores, 5.3 GHz boost, 8-channel DDR5. Same memory controller and PCIe lanes as bigger SKUs; cores past ~32 starve on memory bandwidth for inference. |
14 14 | Motherboard | ASRock Rack WRX90D8-2L/2T | $1,200 | 7× PCIe 5.0 ×16, 8 DIMM slots, **ASPEED AST2600 BMC (OpenBMC-friendly)**, dual 10 GbE, 4× M.2 + 4× SlimSAS. |
15 15 | RAM | 8× 64 GB DDR5-5600 ECC RDIMM (512 GB total) | $2,500 | One DIMM per channel — 2 DIMMs per channel forces DDR5 down to ~4400 MT/s on WRX90, costing ~20% memory bandwidth (real impact on CPU-offload layers). Brand: Micron or Hynix off the board QVL. |
16 - | Storage 1 | 4 TB Gen5 NVMe (Samsung 9100 Pro or Crucial T705) | $600 | ZFS root pool: OS, models, Sando state, logs. |
16 + | Storage 1 | 4 TB Gen5 NVMe (Samsung 9100 Pro or Crucial T705) | $600 | ZFS root pool: OS, models, Sando state, logs. **Read the ZFS-root note below before partitioning.** |
17 17 | Storage 2 | 4 TB Gen5 NVMe (same) | $600 | **Raw XFS kv-scratch.** Not ZFS — ZFS caps Gen5 throughput; scratch is by definition disposable. EveryCycle uses this for kv-cache overflow. |
18 18 | PSU | 2000 W Titanium-class (Super Flower Leadex Titanium or equivalent) | $600 | Headroom for any GPU combination the loose-parts experimentation hits. |
19 19 | Chassis | 4U rackmount with ≥4 dual-slot GPU bays (Sliger CX4712 or similar) | $400 | Matches the eventual EveryCycle reference inference box; rackable from day one. |
@@ -54,7 +54,21 @@
54 54
55 55 - **Threadripper Pro WRX90 platform** chosen over consumer TRX50 for the 7× PCIe 5.0 ×16 slots and the AST2600 BMC. EveryCycle's `bmc-agent` crate needs a real BMC to talk to.
56 56 - **512 GB RAM** lets a 405B Q4 model fit fully in RAM if ever wanted; the immediate need is comfortable headroom for kv-cache experiments.
57 - - **Two Gen5 NVMes with different filesystems** because ZFS root is the right answer for the OS and model store, but ZFS would cap Gen5 bandwidth on the kv-scratch where bandwidth is the whole point.
57 + - **Two Gen5 NVMes with different filesystems** because ZFS root is the right answer for the OS and model store, but ZFS would cap Gen5 bandwidth on the kv-scratch where bandwidth is the whole point. **The first half of that no longer holds if the unit runs a bootc OS — see below.**
58 +
59 + ### ZFS root does not survive a bootc OS
60 +
61 + Added 2026-07-25, when unit 1 was assigned the always-on build-host role and Alloy was picked as its OS (wiki `tm-first-unit-build-host`, GoingsOn alloy task `d1fed0d7`).
62 +
63 + A bootc image-based OS puts an **ostree** root on the boot disk, and ostree-on-ZFS is not a supported combination. There is also no ZFS anywhere in the Alloy image: on Fedora it is an out-of-tree kmod, and an immutable image is the worst place to carry one.
64 +
65 + So on any unit running Alloy (or Silverblue, or CoreOS, or anything else bootc):
66 +
67 + - **Storage 1 root becomes xfs**, the bootc default. Do not spend bring-up time building a ZFS root pool; it will not boot.
68 + - **Storage 2 is unaffected.** Raw XFS kv-scratch was already the answer and the reasoning still stands.
69 + - ZFS survives only as an optional **data** pool on additional disks, if something later actually wants snapshots and checksums for the model store. Nothing does today.
70 +
71 + This is a live conflict in a settled BOM rather than a resolved one: the storage table still reads "ZFS root pool" because a unit running a conventional distro could still do that, and a customer machine's OS is not decided by this file. The rule is the OS decides. Check which OS the unit is getting before partitioning.
58 72 - **2000 W PSU** is sized for the worst-case 4-GPU configuration on the GPU thread, not for A1 alone.
59 73 - **4U rackmount chassis** because it doubles as a prototype of the eventual EveryCycle reference appliance form factor.
60 74