max / everycycle

hardware: bench-v1 after a skeptical pass over its own reasoning Four of this file's justifications did not survive being questioned. The PSU contradicted the document. 850 W was sized for one card while the same file claimed two are reachable by bifurcation; two 250 W passives plus a 170 W CPU is 88% load. Now 1200 W, and ATX 3.1 with native 12V-2x6 so the current-gen vertical needs no adapters. The CPU justification was wrong about the workload. It argued single-thread for incremental cargo, but the edit/compile loop lives in Helix on fw13 and this machine is fire-and-forget CI: clean builds, container builds, zstd. That is throughput. Same part, honest reason, and the 16-core platform ceiling is now stated rather than discovered. The 4U airflow argument was the weakest of the lot. mm-v1 budgets a shroud for the P40 *inside* the 4U, so the case was never replacing the blower — passive datacenter cards want static pressure no 140 mm fan produces. With noise deprioritized the better answer is real server blowers at the card, which frees the enclosure to be optimized purely for access: an open frame, plus a per-card test sled at fixed geometry and recorded RPM as the actual thermal instrument. Dust is the accepted cost and the frame is not shippable as-is; both named. "All-new means known-good" was half right. What new buys is an RMA path, so a host part never becomes a suspect. Known-good has to be measured, so there is now a characterization baseline that gates the first card, and a rule that the first card passing in each vertical becomes that vertical's reference rather than buying one. Also: Sando moves to astra rather than living on a machine that gets opened weekly, ECC monitoring tooling is named as missing, verticals open in sequence with their own sleds, and /var/lib/containers goes on the scratch disk.

Author: Max Johnson <me@maxj.phd> · 2026-07-25 15:30 UTC

Signed with PGP, not checked

Commit: 6fac3bfff2ca2c0e24b97f91194308131c27c33d

Parent: 39f3e51

1 file changed, +132 insertions, -21 deletions

M docs/hardware/bench-v1-bom.md +132 -21

			@@ -8,9 +8,17 @@
8	8
9	9		## What this machine is for, in priority order
10	10
11		-	1. Always-on build host. Alloy images and ISOs, the MNW build gate, Sando. The ecosystem
12		-	has no always-on x86_64 machine: astra is always-on and aarch64, fw13 is x86_64 and sleeps,
13		-	and production never builds binaries.
	11	+	1. Always-on build host, fire-and-forget. Alloy images and ISOs, the MNW build gate,
	12	+	scheduled rebuilds. The ecosystem has no always-on x86_64 machine: astra is always-on and
	13	+	aarch64, fw13 is x86_64 and sleeps, and production never builds binaries. Nobody waits on
	14	+	this machine interactively — the edit/compile loop stays in Helix on fw13 — so it is sized
	15	+	for throughput, not latency.
	16	+
	17	+	Sando does not live here. The original BOM assigned this box the Sando host role back
	18	+	when it was going to be one big always-on machine. That does not survive the box becoming
	19	+	a card-swapping bench: a production deploy controller should not share a chassis with
	20	+	unproven hardware, and a single non-redundant root NVMe makes it worse. Sando moves to
	21	+	astra — always-on, never opened, and arch-agnostic for a controller.
14	22		2. The known-good control for testing used parts one at a time. Every part here is new.
15	23		That is the point: when a used Collection card misbehaves on the bench, the host is not a
16	24		suspect. One variable at a time, against a reference that does not move.
			@@ -27,32 +35,70 @@
27	35
28	36		\| Component \| Choice \| Approx \| Notes \|
29	37		\|---\|---\|---\|---\|
30		-	\| CPU \| AMD EPYC 4565P (16c/32t "Zen 5", AM5) \| ~$700–800 \| Up to 5.7 GHz. Chosen for single-thread: incremental `cargo` builds are latency-bound, and this beats a 32-core Milan at the thing done fifty times a day. 2× DDR5-5600, 28 PCIe Gen 5 lanes, 170 W class \|
	38	+	\| CPU \| AMD EPYC 4565P (16c/32t "Zen 5", AM5) \| ~$700–800 \| 16 Zen 5 cores at high all-core clocks is competitive throughput-per-dollar for clean builds, container builds and the ISO's zstd pass. 2× DDR5-5600, 28 PCIe Gen 5 lanes, 170 W class. Note the ceiling: EPYC 4005 stops at 16 cores, so if build throughput ever binds, the answer is a new machine, not a new CPU. Acceptable for a declared stepping stone. Also live: a Ryzen 9950X is ~$150 cheaper with the same core count and higher clocks — the EPYC premium buys official ECC UDIMM validation and server firmware support, which is worth confirming against this board's Ryzen ECC story before paying it \|
31	39		\| Board \| ASRock Rack B650D4U (base variant) \| ~$400–450 \| mATX server board. AST2600 BMC with IPMI 2.0, iKVM, vMedia, dedicated management LAN. Base variant chosen over -2L2T/-2L2T/BCM specifically because it is the only one with two M.2 slots \|
32	40		\| RAM \| 2× 48 GB DDR5 ECC UDIMM (96 GB) \| ~$450 \| 1 DPC runs 5600; 2 DPC drops to 3600. Two slots stay empty on purpose \|
33	41		\| Storage 1 \| 2 TB NVMe on M2_1 (PCIe 5.0 x4) \| ~$180 \| xfs root. Not ZFS — see the note in `mm-v1-bom.md` \|
34	42		\| Storage 2 \| 4 TB NVMe on M2_2 (PCIe 4.0 x4) \| ~$280 \| Raw XFS scratch: kv-cache overflow later, container build scratch from day one \|
35		-	\| PSU \| 850 W Platinum ATX \| ~$150 \| One 250–300 W card plus a 170 W CPU. Sized for the machine that exists, not a 4-GPU machine that does not \|
36		-	\| Chassis \| 4U rackmount, ≥2 dual-slot bays (Sliger CX4712 or Rosewill RSV-L4500U) \| $120–400 \| Undecided. 4U for airflow, not for form-factor prototyping — see below \|
37		-	\| CPU cooler \| AM5 tower, ≤160 mm \| ~$100 \| Air. Height limit is the 4U lid \|
38		-	\| Case fans \| 4–6× 120/140 mm, high static pressure \| ~$120 \| Load-bearing for the bench role, see below \|
	43	+	\| PSU \| 1200 W Platinum, ATX 3.1 with native 12V-2x6 \| ~$220 \| Was 850 W, which contradicted this document's own claim that two cards are reachable: 2× 250 W passive + a 170 W CPU is ~750 W, or 88% load. 1200 W also covers one modern ~450 W card. ATX 3.1 is deliberate: it handles the transient spikes modern cards produce and provides 12V-2x6 natively, so no adapters when the current-gen vertical gets tested. The 500–600 W-class vertical triggers a PSU review \|
	44	+	\| Enclosure \| Open frame — see the enclosure section \| ~$150–250 \| Optimized for access. Airflow is solved at the card, not by the case \|
	45	+	\| CPU cooler \| AM5 tower, any height \| ~$100 \| Air. No lid to clear on an open frame \|
	46	+	\| Card airflow \| Per-card test sled: server blower + shroud, per vertical \| ~$150 setup \| The actual thermal instrument. See the enclosure section \|
39	47		\| Boot GPU \| none \| $0 \| The AST2600 has its own VGA and iKVM. `mm-v1-bom.md` carried a GT 1030 only because it was unsure the BMC would suffice; the manual settles it \|
40	48		\| Total \| \| ~$2,500–2,900 \| vs ~$10,280 for the `mm-v1` host platform \|
41	49
42	50		Vendor manuals for the decided parts are in `_private/docs/hardware/bench-v1/`.
43	51
44		-	## Why 4U, when a tower would be nicer to work in
	52	+	## Enclosure: open frame, and airflow solved at the card
45	53
46		-	A tower with a side panel is plainly better for swapping cards, and swapping cards is what
47		-	this machine does. It still loses.
	54	+	The earlier draft of this file argued for a 4U rackmount because a passive card "only cools
	55	+	in a chassis with a front-to-back static-pressure path." **That argument does not survive
	56	+	scrutiny.** `mm-v1-bom.md` itself budgets a fan shroud for the P40 inside the 4U, and it is
	57	+	right to: passive datacenter cards are engineered for 1U/2U server fans producing static
	58	+	pressure that no 140 mm case fan approaches. The shroud is needed either way, so the 4U was
	59	+	buying repeatability, not cooling — a weaker claim than the one it was sold on.
48	60
49		-	Collection cards are datacenter pulls, and a Tesla P40 is passively cooled — no fan at
50		-	all. It cools only in a chassis with a front-to-back static-pressure path. A tower needs a
51		-	strapped-on blower per card, which is exactly the sort of ad-hoc rig that makes two bench
52		-	results incomparable. A fixed 4U airflow path means every card is tested in the same thermal
53		-	environment, which is the only thing that makes a Care Tag's thermal trace worth printing.
	61	+	Once noise is not a constraint, there is a better answer available, and it inverts the
	62	+	problem.
54	63
55		-	The case fans are therefore a test instrument, not cooling. Do not substitute quiet fans.
	64	+	Airflow is solved at the card, with real server blowers. 40–80 mm server blowers at
	65	+	10k+ RPM produce the static pressure these cards were designed around. That is a thing a
	66	+	quiet build physically cannot do, and it is the single biggest quality difference in a bench
	67	+	test of a passive card. With sound deprioritized, use them.
	68	+
	69	+	Once airflow lives at the card, the enclosure only has to be good to work in. So:
	70	+
	71	+	- Open aluminium frame (mining-rig style, or an open bench-frame such as a Core P3 /
	72	+	BC1-class fixture). Nothing to unscrew, nothing to unrack, no lid, no cable gymnastics.
	73	+	Cards go in and out in seconds, which is the operation performed most.
	74	+	- Unlimited card clearance. Modern 3–3.5 slot, 350 mm cards fit trivially. A 4U would
	75	+	have constrained exactly the vertical most likely to need bench time later.
	76	+	- Everything visible. You can see fan spin, LEDs, and scorch marks on an unknown card
	77	+	before they become a smell.
	78	+
	79	+	### The test sled is the instrument
	80	+
	81	+	Repeatability comes from a fixture, not a case. Build one test sled per card class: a
	82	+	rigid bracket that holds the card, its shroud, and its blower at a fixed geometry, with
	83	+	the blower on a controller at a fixed, recorded RPM. Every card in that class is then
	84	+	tested in an identical thermal environment regardless of what is around it — better
	85	+	repeatability than a shared case gives, because a case's airflow changes with every other
	86	+	card and cable in it.
	87	+
	88	+	Consequences to hold:
	89	+
	90	+	- One sled per vertical. A passive P40, a 2-fan consumer card and a 3.5-slot current-gen
	91	+	card have nothing thermally in common. Sleds get built as verticals open, not up front.
	92	+	- Log RPM and ambient with every test. A thermal trace without them is not comparable.
	93	+	Until `bmc-agent` exists (and it may not — task `59335767`), blower RPM is set on a manual
	94	+	fan controller and written down, and BIOS-set chassis fan curves are recorded.
	95	+	- Dust is the accepted cost. An open frame running 24/7 in a living space collects dust
	96	+	in the CPU cooler and PSU intake. Mitigated by the blowers only running during tests, and
	97	+	by putting periodic cleaning in the standing ops list. This is a real trade and the reason
	98	+	a closed case would otherwise win.
	99	+	- It is not shippable as-is. When this machine is sold, it either gets rehoused in a case
	100	+	or sold as a parts bundle. Budget that, and do not let the frame make the machine
	101	+	unsellable by surprise.
56	102
57	103		## The GPU ceiling, stated exactly
58	104
			@@ -70,8 +116,9 @@
70	116		Two GPUs are reachable via bifurcation. BIOS exposes "Configure PCIE6 Link Width" with
71	117		`[x16]`, `[x8x8]`, `[x8x4x4]`. Gen5 x8 is roughly twice the bandwidth a Gen3 x16 card such
72	118		as a P40 can consume, so splitting costs those cards nothing. What it costs is a bifurcation
73		-	riser and a physical mounting problem: two dual-slot cards hanging off risers above an mATX
74		-	board in a 4U is a rig to solve, not a slot to populate. Unproven until someone builds it.
	119	+	riser and a physical mounting problem: two dual-slot cards on risers off an mATX board is a
	120	+	rig to solve, not a slot to populate. The open frame helps here — a frame is a mounting
	121	+	surface, where a case would have been a constraint. Unproven until someone builds it.
75	122
76	123		So against the EveryCycle roadmap:
77	124
			@@ -85,6 +132,47 @@
85	132		- B6 (frontier-size with CPU offload) — does not fit. Two DDR5 channels is ~80–90 GB/s
86	133		against ~358 GB/s for 8-channel DDR5-5600. CPU-offload work belongs to the replacement.
87	134
	135	+	## The bench is not a control until it is characterized
	136	+
	137	+	"All-new means known-good" is the right instinct with the wrong mechanism. New parts have
	138	+	infant mortality and firmware bugs; a used part with hours on it can be more reliable than
	139	+	a fresh one. What new actually buys is an RMA path — when a host part does fail, it gets
	140	+	replaced rather than debugged, and it never becomes a suspect in a card's test result. That
	141	+	holds up. The "known-good" part does not, until it is measured.
	142	+
	143	+	Before the first used card goes in, record a baseline and keep it in the runbook:
	144	+
	145	+	- memtest86+ clean pass over all installed DIMMs.
	146	+	- Sustained all-core `stress-ng` run: clocks held, package temp, no throttle.
	147	+	- `fio` on both NVMes: throughput and latency, so a future slow build has a reference.
	148	+	- Idle and full-load wall power, and the ambient temperature both were taken at.
	149	+	- ECC corrected-error count at zero, and proof you can read it — see below.
	150	+
	151	+	Designate a reference card per vertical. To tell "this card is bad" from "the slot, riser,
	152	+	PSU, sled or BIOS is bad" you need a known-good card to swap in. Buying a new one in each
	153	+	class is not worth it. Instead: **the first used card that passes cleanly in a vertical
	154	+	becomes that vertical's reference** — its numbers are recorded, and it is not sold. Cheap,
	155	+	and it makes every later test in that class a comparison rather than an absolute.
	156	+
	157	+	ECC you cannot read is not ECC. Rising correctable-error counts are the early warning the
	158	+	memory is paid for, and Alloy ships no `rasdaemon`, `edac-utils`, `smartmontools` or
	159	+	`ipmitool`. Those belong in the server profile (GoingsOn alloy task `d1fed0d7`) and the
	160	+	baseline above cannot be taken without them.
	161	+
	162	+	## Card verticals, opened one at a time
	163	+
	164	+	The Collection wants at least one cut in several verticals, and cannot fund testing all of
	165	+	them at once. So verticals open in sequence, and each one opens a small equipment bill:
	166	+
	167	+	\| Vertical \| Bench implications \|
	168	+	\|---\|---\|
	169	+	\| Passive datacenter pulls (P40, P100, MI50) \| Shroud + server blower sled. Some use CPU-style EPS 8-pin, not PCIe. Large BARs need Above 4G + ReBAR. First, because the roadmap names it and the cards are cheapest \|
	170	+	\| Used consumer / gaming \| Self-cooling, standard PCIe power, trivial sled. Needs a monitor on the card under test, since display output is part of what is being sold \|
	171	+	\| B-stock / open-box current-gen \| Standard cooling, may retain manufacturer warranty. Bench time still required to prove it \|
	172	+	\| Modern high-TDP (500–600 W class) \| 12V-2x6 native (the ATX 3.1 PSU covers this), real transient spikes, 3.5 slots, 350 mm. Triggers a PSU review past 1200 W and its own sled \|
	173	+
	174	+	Sequence the verticals; do not buy sleds or PSU headroom for a vertical that is not open yet.
	175	+
88	176		## What was deliberately not bought, and why
89	177
90	178		- 8-channel memory bandwidth. The single real loss. It costs a $3,900 CPU tier plus
			@@ -114,8 +202,31 @@
114	202		IPMI/Redfish or wants a dedicated dev board. **This question is not specific to this
115	203		board — it applies to the WRX90 spec identically**, and answering it on a $400 board is
116	204		cheaper.
117		-	- Chassis choice, and with it whether an mATX board plus a bifurcation riser can actually
118		-	mount two dual-slot cards.
	205	+	- 48 GB ECC UDIMM availability and price. The manual lists 48 GB as the maximum module
	206	+	size so the spec permits it, but ECC UDIMMs are commonly 16/32 GB. If 48 GB is scarce or
	207	+	badly priced, 2× 32 GB (64 GB at 5600) is honestly enough for 16 cores, and 4× 32 GB pays
	208	+	the 3600 penalty for capacity that a fire-and-forget CI host does not need.
	209	+	- Where the frame physically sits. There is no rack, and an open frame in a living space
	210	+	has different constraints than a case: dust, curiosity, and anything that can be knocked
	211	+	into it. Pick the spot before buying the frame.
	212	+	- The x8x8 riser rig, which an open frame makes easier than any case would: two dual-slot
	213	+	cards on risers need mounting, and a frame is a mounting surface.
	214	+
	215	+	## Sequence the bring-up
	216	+
	217	+	Too many unknowns arrive at once otherwise: a new hardware platform, Alloy in a role it has
	218	+	never held, no settled sshd story, bench duty, and CI. Add one at a time — the same
	219	+	discipline the bench itself exists to enforce.
	220	+
	221	+	1. Bare bring-up on the frame. BIOS, BMC, iKVM, management LAN.
	222	+	2. OS install, and find out what `alloy install` does on non-laptop hardware.
	223	+	3. Characterization baseline (above). This is the gate that makes it a control.
	224	+	4. Build load: images, ISOs, the MNW gate.
	225	+	5. First used card, in a sled, with the reference procedure.
	226	+
	227	+	`/var/lib/containers` mounts on the 4 TB scratch, not on root. On a bootc host `/var` is
	228	+	stateroot, so this is a deliberate step; get it wrong and every container build lands on the
	229	+	2 TB root.
119	230
120	231		## BIOS at first boot
121	232