Skip to main content

max / everycycle

12.9 KB · 190 lines History Blame Raw
1 # EveryCycle Architecture
2
3 An architectural sketch rather than a spec. It exists to make trade-offs visible and to give the Phase 0 demo a target.
4
5 ## The bet
6
7 We are not building a stack that beats vLLM on H100s. vLLM on H100s is excellent. We are building a stack that makes **heterogeneous, cheap, slightly-old hardware** serve real inference workloads at production quality, and we ship the box that makes that capability turnkey.
8
9 The wedge is software, not silicon. The box is the vehicle.
10
11 Three load-bearing observations:
12
13 1. **GPU pricing is set by training demand.** A used 3090 ($500–700) and an H100 ($25k) have a ~40× price gap and a ~3–6× inference-throughput gap on realistic memory-bound workloads. The remaining ~7–10× spread is captured by whoever writes the software to make the cheap path reliable.
14 2. **Existing inference engines assume homogeneity.** vLLM and TGI work best on identical modern GPUs. llama.cpp handles cheap and mixed hardware brilliantly at the single-user level but is not a serving runtime. The serving-grade-but-heterogeneous-aware engine doesn't exist in production form.
15 3. **Inference quality on quantized models has caught up.** Q4_K_M and IQ-quants on competent models are now indistinguishable from FP16 for most production tasks. The "you need full precision" reflex is out of date.
16
17 EveryCycle puts those three observations together.
18
19 ## Stack diagram
20
21 ```
22 +-------------------------------------------------------------------+
23 | OPERATOR SURFACE |
24 | TUI dashboard | HTTP admin | Prometheus / OTel export |
25 +-------------------------------------------------------------------+
26 | SERVING API LAYER |
27 | OpenAI-compatible HTTP | streaming SSE / gRPC |
28 | rate limit, auth, quotas | request tracing |
29 +-------------------------------------------------------------------+
30 | ORCHESTRATOR (control plane) |
31 | |
32 | request router | fleet manager | model registry |
33 | priority queue | health monitor | rolling upgrade |
34 | per-tenant SLO | capability map | observability bus |
35 +-------------------------------------------------------------------+
36 | SCHEDULER (per box) |
37 | |
38 | continuous batcher (paged kv) | admission control |
39 | spec-decode coordinator | preemption / cancel |
40 | heterogeneous shard planner | memory accountant |
41 +-------------------------------------------------------------------+
42 | MODEL RUNTIME (per box) |
43 | |
44 | pipeline-parallel executor | weight streaming |
45 | per-layer placement (CPU/GPU/N) | kv-cache offload |
46 | quantization-aware kernels | speculative draft host |
47 +-------------------------------------------------------------------+
48 | COMPUTE BACKENDS (plugin) |
49 | |
50 | cuda (Ampere, Ada, Hopper, Blackwell) via cudarc |
51 | rocm (RDNA3/4, CDNA) via custom + hipBLAS |
52 | vulkan / wgpu (universal fallback, Arc, Apple, etc.) |
53 | cpu (avx2, avx512, neon) native Rust + BLAS |
54 +-------------------------------------------------------------------+
55 | HARDWARE ABSTRACTION LAYER |
56 | |
57 | device enumeration | topology (PCIe, NVLink-if-present) |
58 | memory pools | power / thermal telemetry |
59 | IPC: shared mem, ring buffers, lockless queues |
60 +-------------------------------------------------------------------+
61 | BMC AGENT (separate binary) |
62 | |
63 | Rust agent on OpenBMC fork |
64 | structured telemetry (no IPMI web UI from 2008) |
65 | fan curves, PSU sequencing, GPU power capping |
66 | thermal model exposed to scheduler via gRPC |
67 +-------------------------------------------------------------------+
68 ```
69
70 ## Where the alpha sits, by layer
71
72 | Layer | Effort grade | Alpha story |
73 |-------|--------------|-------------|
74 | Operator surface | medium | Honest operator UX is a differentiator vs opaque appliances. TUI quality matters for brand more than for performance. |
75 | Serving API | low | OpenAI-compatible is table stakes. Build correctly, don't innovate here. |
76 | Orchestrator | **high** | Heterogeneous fleet awareness lives here. Routing a request to "the box whose idle 3090 has the right model warm" is real value nobody ships well. |
77 | Scheduler | **high** | Continuous batching tuned for low-VRAM cards, mixed GPU generations on one box, speculative decoding coordination across devices. Hardest layer, highest-leverage. |
78 | Model runtime | **high** | Aggressive per-layer placement, weight streaming, kv-cache offload to NVMe. llama.cpp does some of this; nobody does it in a serving context. |
79 | Compute backends | medium | We bind to existing kernel libraries. The work is making the abstraction over CUDA/ROCm/Vulkan honest enough that backends can be swapped without rewrites. |
80 | HAL | medium | Critical to get right (topology mistakes corrupt performance silently) but not novel. |
81 | BMC agent | medium | High brand and trust value (open firmware that talks structurally to the runtime). Modest performance value. |
82
83 The two layers worth most of the engineering investment are **scheduler** and **model runtime**, with **orchestrator** as the connecting tissue.
84
85 ## Request flow (single box, batched)
86
87 ```
88 client
89 |
90 v
91 [Serving] -- auth, rate limit, schema check
92 |
93 v
94 [Orchestrator] -- pick a box (warm model? lowest queue? right capability?)
95 | (gRPC to selected box)
96 v
97 [Scheduler] -- admit, place in continuous batch, allocate kv pages
98 |
99 v
100 [Runtime] -- forward pass across placed layers
101 | |
102 | +--> GPU 0 (3090, 24GB): layers 0-23, fp16 attention
103 | +--> GPU 1 (3090, 24GB): layers 24-47, fp16 attention
104 | +--> CPU (DDR5, 128GB): layers 48-55 + embedding head, q4
105 | +--> NVMe (kv overflow): pages evicted from GPU when queue spikes
106 |
107 v
108 [Scheduler] -- stream tokens back as they emerge (no wait for full sequence)
109 |
110 v
111 [Serving] -- SSE/gRPC stream to client
112 |
113 v
114 client
115 ```
116
117 The interesting moves:
118
119 - **Heterogeneous layer placement.** A 70B-class model on a single $1500 box (2× used 3090 + 128GB DDR5) is realistic if we put the tail layers on CPU. Throughput drops 2–3× vs all-GPU but cost drops 20×. For batch-size-1 chat with small concurrent user counts, that's the right trade.
120 - **NVMe as kv-cache overflow.** vLLM's PagedAttention assumes VRAM is the bound. On cheap boxes it isn't; total context budget is. Spilling cold kv pages to NVMe is a 10× capacity multiplier at the cost of a controlled latency penalty on reactivation. Nobody ships this in serving.
121 - **Speculative decoding routing.** Run a 1B draft on CPU or a small GPU; run the 70B target on the cheap fleet. Spec-decode is a 2–3× throughput win when the draft is well-matched; almost no serving runtime exposes it cleanly.
122
123 ## Heterogeneous fleet, multi-box
124
125 ```
126 +-------------------+
127 | ORCHESTRATOR |
128 | (HA pair) |
129 +---------+---------+
130 |
131 +------------------------+--------------------------+
132 | | |
133 +----+----+ +----+----+ +----+----+
134 | BOX A | | BOX B | | BOX C |
135 | | | | | |
136 | 4x 3090 | | 2x 4090 | | 8x 7900 |
137 | EPYC | | TR Pro | | XTX |
138 | 256GB | | 128GB | | EPYC |
139 +---------+ +---------+ +---------+
140 warm: 70B Q4 warm: 8B fp16 warm: 70B fp8
141 warm: 8B cold: 13B
142 draft host (rocm)
143 ```
144
145 Each box advertises its **capability map** to the orchestrator: which models it has warm, which it can warm quickly, current queue depth, thermal state. The orchestrator routes against that map. Naive round-robin treats this fleet badly; capability-aware routing recovers ~2× aggregate throughput.
146
147 This is exactly where vLLM and TGI underperform and where a control plane built around heterogeneity from the start can win. Also exactly the kind of thing that benefits from a Rust implementation — concurrent, latency-sensitive, operationally observable, and we want it to *not fall over at 3am*.
148
149 ## Quantization and accuracy posture
150
151 Default to **kv-cache-aware quantization**: Q4_K_M or IQ4_XS for weights, Q8 or fp16 for kv-cache, fp16 for attention math. Produces output indistinguishable from fp16 on most production benchmarks while halving VRAM.
152
153 Customers who want fp16/bf16 end-to-end can have it; their box serves fewer concurrent requests or runs a smaller model. The default is the cheap-hardware default because it is also the right answer for most workloads, and we are willing to say so.
154
155 We do not ship Q2 or Q3 defaults. Below Q4 the accuracy loss is real and the support burden of "the model gave a weird answer" goes up. Q3 is available as an option for customers who know what they want; it is not the default that surprises someone.
156
157 ## What we are explicitly not building
158
159 - **Training.** Not in v1, not in v3. Inference is a different software discipline and the operation should learn to do one thing first.
160 - **A novel tensor library.** We bind to `cudarc`, `candle` / `burn` primitives where solid, `wgpu` for the universal fallback, and write the scheduler and runtime — not the kernels. Writing our own kernels is a four-year detour we do not need.
161 - **A model zoo of our own.** We serve the open models that exist: Llama, Qwen, Mistral, DeepSeek, Phi, Whisper, Flux when image makes sense. We do not train or fine-tune as a product.
162 - **Kubernetes integration as a first-class feature.** Customers who want k8s can wrap our HTTP API. We will not add complexity to the runtime to accommodate it.
163 - **A web UI.** TUI and structured telemetry exporters are the operator surface. Grafana exists and consumes our Prometheus output cleanly.
164
165 ## Build order
166
167 Each phase is a working artifact, not a feature flag.
168
169 **Phase 0 — Single box, single model, single user (8 weeks).** Rust binary that loads a quantized 8B or 70B model, exposes OpenAI-compatible HTTP, runs on whatever GPU is plugged in. Continuous batching, paged kv, basic Prometheus export. TUI dashboard, one screen, real telemetry. The angel-raise demo.
170
171 **Phase 1 — Single box, heterogeneous placement (12 weeks).** Per-layer placement across mixed GPUs and CPU. kv-cache NVMe overflow. Quantization-aware kernel selection. BMC agent: structured telemetry, fan curves, thermal-aware power capping.
172
173 **Phase 2 — Multi-box orchestrator (16 weeks).** Orchestrator with capability map and routing. Speculative decoding with draft on separate device or box. Model registry: warm/cold state per box, rolling model loads. First customer-deployed cluster.
174
175 **Phase 3 — Hardening and openness (ongoing).** Publish BMC agent and runtime as open repos. Reference benchmarks against vLLM and TGI on equivalent hardware, published honestly. Documentation thorough enough that a hardware resident can read it in their first month.
176
177 ## What this sketch leaves open
178
179 - Exact factoring across the named crates: which logic lives in `runtime` vs `hal` vs `scheduler`. Phase 0 decision.
180 - Whether the BMC agent shares any code with the host runtime. Probably not, given the different security and reliability constraints, though `everycycle-hal` may expose a thin shared schema crate.
181 - HA story for the orchestrator. Phase 2 problem.
182 - Multi-tenant isolation guarantees. Start single-tenant per box; add per-tenant isolation when a customer asks.
183
184 ## What we would push back on if proposed
185
186 - "Let's also write the kernels in Rust." No. Bind to `cudarc` and friends. The scheduler is where time should go.
187 - "Let's support training as a stretch goal." No. Different software discipline, different customer, different cadence.
188 - "Let's start with the orchestrator." No. Phase 0 must be a single binary that runs on one box and looks great in a demo. The orchestrator is Phase 2.
189 - "Let's target H100s for the demo." No. The demo runs on hardware that cost $2000, which is the point.
190