Skip to main content

max / everycycle

15.0 KB · 190 lines History Blame Raw
1 # EveryCycle Appraisal
2
3 How EveryCycle measures a GPU as a tool for doing computation, and why the same measurement feeds both the scheduler and the eventual TailoredMachines buyback formula.
4
5 This is a design sketch, not a spec. The schema lives in `crates/hal/src/capability.rs`; this doc records the rules the schema was built against so they do not drift.
6
7 ## Why this exists
8
9 The scheduler needs to know what each device can do. Routing a layer to a card is an estimate of "how many ops/sec of precision P will land if I put this work here." The cleanest source of that estimate is the device itself, measured.
10
11 The TailoredMachines buyback program (see `todo.md`, cross-project hook) will eventually publish a price formula for used cards. The input that formula needs is the same input the scheduler needs: a vector of measured capabilities. Building one tool that serves both is leverage; the alternative is two diverging definitions of "what this card is worth doing."
12
13 Reference benchmarks against vLLM and TGI (architecture.md, Phase 3) are a different exercise. They answer "does our stack hold up against the field." The appraisal tool answers "what is this physical device capable of, today, in this rack."
14
15 ## The SKU-agnostic principle
16
17 No part of the scoring path looks up the card's model.
18
19 No PCIe-ID-to-expected-TFLOPs table. No "this is a 3090, you scored 94% of class median." No nameplate anywhere in the algorithm.
20
21 This is load-bearing for three reasons:
22
23 1. **It generalizes.** A binned die, a modded card, an oddball regional SKU, and a card that hasn't been released yet all get fair value from the same code path.
24 2. **It resists fraud.** A seller cannot inflate a report by claiming the card is something it is not. The numbers stand or do not.
25 3. **It matches EveryCycle's philosophy.** The whole project is built on "cards are throughput at a precision, route accordingly." A SKU-aware appraisal tool would be the one component that quietly rebuilds the model-table assumption the rest of the stack is designed to be rid of.
26
27 Identity fields (PCIe BDF, vendor:device:subsystem, VBIOS hash, serial) are recorded in the report for anti-fraud — they anchor "this is the same physical card across time" — but they are metadata. They are never inputs to scoring or routing.
28
29 ## The capability vector
30
31 The full schema is in `crates/hal/src/capability.rs`. The shape, summarized:
32
33 - **`PrecisionThroughput[]`** — sustained TOPS at each precision the device can execute. Vec, not fixed fields, because future precisions (FP6, microscaling, whatever comes after) drop in without a breaking change. Each entry carries a `matrix_engine: bool` honesty flag: a card with no reachable matrix path reports vector-ALU TOPS and says so.
34 - **`Memory`** — physical bytes, usable bytes (after driver and OS reservation), sustained read and write bandwidth, integrity-probe error count. The scheduler routes on usable; the report records both.
35 - **`Interconnect`** — measured host bandwidth, plus full pairwise directed peer measurements (one `PeerLink` per ordered `this -> peer`). No `link_kind` field: classifying a link as PCIe vs NVLink vs Infinity Fabric would be vendor knowledge, and the report stays purely measured. A reader can draw their own conclusions about why pair X is faster than pair Y. Cost is O(n²); fine because this is an appraisal run, not a hot path.
36 - **`LaunchOverhead`** — null-launch latency and 1 MiB host-to-device memcpy latency. Small fields, but on heterogeneous fleets they decide whether a card is worth shipping small work to at all.
37 - **`ThermalEnvelope`** — time-to-throttle (Option, because "did not throttle" is a real outcome), throttled throughput as a fraction of cold throughput, sustained power draw.
38 - **`DeviceIdentity`** — anti-fraud metadata only. See above.
39 - **`ProbeEnvironment`** — driver version, ambient inlet temp, run duration, probe-suite version. Two reports are only directly comparable when these agree, and the comparator can enforce that mechanically.
40
41 Everything is measured. Nothing is rated. The constructor for a `CapabilityVector` is a probe run, not a config file.
42
43 ## The probe trait
44
45 A probe is a backend implementation that fills in the vector. Sketch:
46
47 ```rust
48 pub trait Probe {
49 /// Name for the report (e.g. "vulkan-coopmat", "cuda-cublaslt").
50 fn name(&self) -> &str;
51
52 /// Returns the probe-suite version this probe contributes to.
53 fn version(&self) -> &str;
54
55 /// Run against one device and produce its capability vector.
56 fn run(&self, device: &Device) -> Result<CapabilityVector, ProbeError>;
57 }
58 ```
59
60 Backends planned in order of cross-vendor reach:
61
62 1. **Vulkan + `VK_KHR_cooperative_matrix`** — the primary path. One Rust probe via `ash` (or `wgpu` plus raw Vulkan for the cooperative-matrix piece) hits the matrix engines on Nvidia, AMD, and Intel. Apple via MoltenVK is partial; gaps fall to backend 3.
63 2. **BabelStream + mixbench wrappers** — not for the report, for ground-truth cross-checks. If our Vulkan bandwidth probe disagrees with BabelStream on the same card by more than a small tolerance, something in our probe is wrong, and we want to find out before a buyback formula trusts the number. These are shelled-out, not linked.
64 3. **Per-vendor BLAS fallbacks** — `cuBLASLt`, `hipBLASLt`, `MPSGraph`. Used only where backend 1 cannot reach the matrix engine. Each emits the same `CapabilityVector` shape; the report carries the probe name so the source is visible.
65 4. **CPU baseline** — same struct, used for routing decisions about offload and for the host-side numbers.
66
67 The scheduler consumes `CapabilityVector` directly. The appraisal CLI runs the same probe, attaches identity and environment, and signs the result.
68
69 ## Internal consistency in place of cohort comparison
70
71 A SKU-aware tool catches degradation by comparing a card to its class median. We do not have that lever. The substitute is the roofline.
72
73 A healthy device sits on its own arithmetic-intensity roofline. If the memory-bandwidth probe measures X GB/s and the matrix probe achieves Y% of what X predicts at the precision being tested, that ratio should land where the physics says it should. A card that measures bandwidth-X but only achieves a fraction of expected compute *given that bandwidth* is telling on itself — without anyone consulting a model table.
74
75 Mixbench's roofline plot is exactly this analysis. The appraisal tool will compute and report the on-roofline ratio per precision. A buyback formula can choose to discount cards that fall below a roofline threshold without ever needing to know the card's model.
76
77 A card with surprise-low bandwidth and proportionally-low compute is a slower card. A card with normal bandwidth and *disproportionately* low compute is a degraded card. The vector distinguishes these.
78
79 ## Cross-checks against standard tools
80
81 For credibility, the probe suite carries a `--cross-check` mode that runs BabelStream and mixbench against the same device and emits a delta against our internal probes. Two purposes:
82
83 1. **Catch our own bugs.** If our Vulkan bandwidth number disagrees with BabelStream by more than a small percent, our probe is wrong; report a warning and refuse to sign.
84 2. **Public legibility.** A published appraisal that says "our number, BabelStream's number, mixbench's number, delta within tolerance" is much harder to wave away than one that asks the reader to trust our probes alone.
85
86 Standard-tool comparison is also how we'd respond to "your TOPS number is unrealistic" — we publish the cross-check by default.
87
88 ## The signed report
89
90 The report is a single JSON file per card with the signature inlined. One file is the unit of transfer — it can be emailed, attached to a listing, pasted into a forum post, dropped into a verifier. A human can open it and read the numbers; the signature is a base64 blob you scroll past unless you care.
91
92 ### Card report envelope
93
94 ```json
95 {
96 "payload": {
97 "report_version": "1",
98 "probed_at": "2026-06-20T18:23:11Z",
99 "box_report": "<hex-encoded SHA-256 of canonical box-report payload>",
100 "capability": { /* CapabilityVector */ }
101 },
102 "signature": {
103 "alg": "ed25519",
104 "tenant_identity": "tailoredmachines",
105 "tenant_pubkey": "<base64 ed25519 public key>",
106 "sig": "<base64 ed25519 signature over canonical payload bytes>"
107 }
108 }
109 ```
110
111 Filename convention: `card-<vbios-hash-or-bus-address>.appraisal.json`. Filenames are cosmetic; the content carries the real identity.
112
113 ### Box report and linkage
114
115 A probe run produces N card reports plus one box report. The box report carries the topology context: which other devices were present, what host they were on, ambient temperature, driver bindings, probe-suite version. Card reports stand alone for normal use, but if a peer measurement looks anomalous later you fetch the box report and read the topology that produced it.
116
117 ```json
118 {
119 "payload": {
120 "report_version": "1",
121 "probed_at": "2026-06-20T18:23:11Z",
122 "host": {
123 "hostname": "astra",
124 "dmi_string": "...",
125 "cpu_model": "..."
126 },
127 "present_devices": ["0000:01:00.0", "0000:02:00.0", "..."],
128 "driver_versions": [
129 {"bus_address": "0000:01:00.0", "driver_version": "..."},
130 ...
131 ],
132 "ambient_c": 24.5,
133 "probe_suite_version": "0.1.0"
134 },
135 "signature": { ... }
136 }
137 ```
138
139 Filename convention: `box-<short-hash>.appraisal.json`.
140
141 The link from card to box is **content-hash, not UUID**: each card report's `box_report` field carries the hex SHA-256 of the canonical box-report payload. A verifier given both files re-hashes the box payload and confirms the link mechanically — no registry, no trust-on-first-use. Tampering with the box report breaks every card report that references it.
142
143 ### Trust model: per-tenant key
144
145 The operator (in-house team, homelab user, TailoredMachines itself) generates one Ed25519 keypair and signs every report they produce. The public key ships in each report; verifier trust is established the way a signed software release establishes it — you trust the pubkey, you trust the reports it signs. Cheap to build, well-understood failure mode, revocation is the operator's problem. Suitable while EveryCycle is mostly in-house. If TM publishes a buyback program later, additional trust layers (per-box BMC keys, identity-bound per-card keys) can sit on top of this format without breaking it.
146
147 ### What gets signed
148
149 For both card and box reports, the signature covers the entire `payload` object, serialized with a deterministic canonical encoding (stable key order, UTF-8, no insignificant whitespace). The canonicalizer is shipped with the writer and the verifier; we control both ends, so no canonical-JSON-ecosystem footgun.
150
151 Card payload signed bytes therefore include:
152
153 - The full capability vector — numbers cannot be edited after the fact.
154 - Identity fields (PCIe BDF, vendor:device:subsystem, VBIOS hash, serial) — you cannot lift a real card's vector onto a different identity.
155 - The box-report hash — if the box report is altered, the card's reference no longer resolves.
156 - Probe-suite version, driver version, ambient temperature, duration.
157
158 Box payload signed bytes include host identity, present devices, per-device driver bindings, ambient temperature, probe-suite version, probe timestamp.
159
160 Public verification means anyone — a buyer evaluating a listing, an auditor, a future TM competitor — can run the verifier and confirm the report binds to the identity claimed under the tenant's pubkey.
161
162 ## What is out of scope
163
164 - **Unified-memory devices.** Appraisal targets discrete GPUs with dedicated VRAM. Apple Silicon, integrated GPUs, APUs with shared memory, and NPUs are deliberately excluded — the buyback economics target discrete cards pulled from gaming PCs and ex-mining rigs, and the schema's separation of physical-vs-usable VRAM and host-vs-peer bandwidth assumes a discrete topology. The EveryCycle runtime may still support Asahi as a contributor-node target; that is a different scope, and this tool does not attempt to score those devices.
165 - **Market pricing.** This tool produces a capability vector. Converting that vector to dollars is a separate function published by TailoredMachines (and openly, if the platform is to mean anything). Other parties may publish their own functions.
166 - **Listing-side appraisal.** Pricing a card before you have physical access is market-comp work — aggregating sold listings, condition tiers, time-decay — not benchmarking. A separate tool, not this one.
167 - **Workload-specific scoring.** "How fast can this card serve Qwen-7B Q4_K_M" is a downstream calculation. The vector is the substrate; the workload model is the consumer. Keeping these separate keeps the appraisal valid across model generations.
168 - **Reference benchmarks vs vLLM and TGI.** Different exercise. Lives elsewhere in the roadmap.
169
170 ## Initial targets
171
172 **First target: astra + RTX 5070 Ti.** The probe suite is developed against **astra**, the aarch64 Linux build host on the tailnet. Address it by tailnet name. The card plugged into it is an **NVIDIA GeForce RTX 5070 Ti** (GB203 Blackwell, 16 GB GDDR7, PCI ID `10de:2c05` at BDF `0000:01:00.0`).
173
174 **Second target: gaming-PC host + RTX 5070.** Once Phase B works on astra, the next host is the operator's gaming PC running an **NVIDIA GeForce RTX 5070** (GB205 Blackwell, 12 GB GDDR7). Same vendor and generation as the first card, so the probe code path is the same, but it is a different SKU on a different host on a different ISA (x86_64 vs aarch64). Both cards run through the same enumeration and probe code, and the report differs only in measured values, which is the check that the SKU-agnostic rule holds in practice.
175
176 Choosing astra first is deliberate: it forces the Vulkan and BLAS paths to work on aarch64 Linux from day one, which is the harder of the two architectures EveryCycle targets, and shakes out any x86-only assumptions in dependencies before they calcify. The 5070 Ti is also a good first target on the merits:
177
178 - Blackwell exposes FP8 (E4M3, E5M2) and FP4 natively via 5th-gen tensor cores, so the `PrecisionThroughput` vector gets exercised against a card with real entries at the modern precisions, not a generation where half are N/A.
179 - Nvidia's Blackwell drivers expose `VK_KHR_cooperative_matrix` (and the newer `VK_NV_cooperative_matrix2`), so the primary cross-vendor path can be developed and verified here before the AMD and Intel ports land.
180 - 16 GB VRAM is real-but-modest, matching the EveryCycle target audience (cheap-and-slightly-old hardware) better than an H100 would. Memory-bound workloads behave the way they will in production.
181
182 The host also carries an ASPEED Graphics chip at `0003:02:00.0`, the server motherboard's BMC display rather than a compute device. The device-enumeration step will need a filter that picks discrete-compute GPUs only (PCI class plus vendor plus presence of a discrete VRAM pool is the cheap heuristic).
183
184 ## References
185
186 - Schema: `crates/hal/src/capability.rs`
187 - Buyback hook: `todo.md`, "Cross-project hook (TailoredMachines buyback program)"
188 - Warranty policy with the buyback deferral: the TailoredMachines warranty policy
189 - Architecture context (Phase 3 reference benchmarks, separate exercise): `docs/architecture.md`
190