# EveryCycle Architecture An architectural sketch rather than a spec. It exists to make trade-offs visible and to give the Phase 0 demo a target. ## The bet We are not building a stack that beats vLLM on H100s. vLLM on H100s is excellent. We are building a stack that makes **heterogeneous, cheap, slightly-old hardware** serve real inference workloads at production quality, and we ship the box that makes that capability turnkey. The wedge is software, not silicon. The box is the vehicle. Three load-bearing observations: 1. **GPU pricing is set by training demand.** A used 3090 ($500–700) and an H100 ($25k) have a ~40× price gap and a ~3–6× inference-throughput gap on realistic memory-bound workloads. The remaining ~7–10× spread is captured by whoever writes the software to make the cheap path reliable. 2. **Existing inference engines assume homogeneity.** vLLM and TGI work best on identical modern GPUs. llama.cpp handles cheap and mixed hardware brilliantly at the single-user level but is not a serving runtime. The serving-grade-but-heterogeneous-aware engine doesn't exist in production form. 3. **Inference quality on quantized models has caught up.** Q4_K_M and IQ-quants on competent models are now indistinguishable from FP16 for most production tasks. The "you need full precision" reflex is out of date. EveryCycle puts those three observations together. ## Stack diagram ``` +-------------------------------------------------------------------+ | OPERATOR SURFACE | | TUI dashboard | HTTP admin | Prometheus / OTel export | +-------------------------------------------------------------------+ | SERVING API LAYER | | OpenAI-compatible HTTP | streaming SSE / gRPC | | rate limit, auth, quotas | request tracing | +-------------------------------------------------------------------+ | ORCHESTRATOR (control plane) | | | | request router | fleet manager | model registry | | priority queue | health monitor | rolling upgrade | | per-tenant SLO | capability map | observability bus | +-------------------------------------------------------------------+ | SCHEDULER (per box) | | | | continuous batcher (paged kv) | admission control | | spec-decode coordinator | preemption / cancel | | heterogeneous shard planner | memory accountant | +-------------------------------------------------------------------+ | MODEL RUNTIME (per box) | | | | pipeline-parallel executor | weight streaming | | per-layer placement (CPU/GPU/N) | kv-cache offload | | quantization-aware kernels | speculative draft host | +-------------------------------------------------------------------+ | COMPUTE BACKENDS (plugin) | | | | cuda (Ampere, Ada, Hopper, Blackwell) via cudarc | | rocm (RDNA3/4, CDNA) via custom + hipBLAS | | vulkan / wgpu (universal fallback, Arc, Apple, etc.) | | cpu (avx2, avx512, neon) native Rust + BLAS | +-------------------------------------------------------------------+ | HARDWARE ABSTRACTION LAYER | | | | device enumeration | topology (PCIe, NVLink-if-present) | | memory pools | power / thermal telemetry | | IPC: shared mem, ring buffers, lockless queues | +-------------------------------------------------------------------+ | BMC AGENT (separate binary) | | | | Rust agent on OpenBMC fork | | structured telemetry (no IPMI web UI from 2008) | | fan curves, PSU sequencing, GPU power capping | | thermal model exposed to scheduler via gRPC | +-------------------------------------------------------------------+ ``` ## Where the alpha sits, by layer | Layer | Effort grade | Alpha story | |-------|--------------|-------------| | Operator surface | medium | Honest operator UX is a differentiator vs opaque appliances. TUI quality matters for brand more than for performance. | | Serving API | low | OpenAI-compatible is table stakes. Build correctly, don't innovate here. | | Orchestrator | **high** | Heterogeneous fleet awareness lives here. Routing a request to "the box whose idle 3090 has the right model warm" is real value nobody ships well. | | Scheduler | **high** | Continuous batching tuned for low-VRAM cards, mixed GPU generations on one box, speculative decoding coordination across devices. Hardest layer, highest-leverage. | | Model runtime | **high** | Aggressive per-layer placement, weight streaming, kv-cache offload to NVMe. llama.cpp does some of this; nobody does it in a serving context. | | Compute backends | medium | We bind to existing kernel libraries. The work is making the abstraction over CUDA/ROCm/Vulkan honest enough that backends can be swapped without rewrites. | | HAL | medium | Critical to get right (topology mistakes corrupt performance silently) but not novel. | | BMC agent | medium | High brand and trust value (open firmware that talks structurally to the runtime). Modest performance value. | The two layers worth most of the engineering investment are **scheduler** and **model runtime**, with **orchestrator** as the connecting tissue. ## Request flow (single box, batched) ``` client | v [Serving] -- auth, rate limit, schema check | v [Orchestrator] -- pick a box (warm model? lowest queue? right capability?) | (gRPC to selected box) v [Scheduler] -- admit, place in continuous batch, allocate kv pages | v [Runtime] -- forward pass across placed layers | | | +--> GPU 0 (3090, 24GB): layers 0-23, fp16 attention | +--> GPU 1 (3090, 24GB): layers 24-47, fp16 attention | +--> CPU (DDR5, 128GB): layers 48-55 + embedding head, q4 | +--> NVMe (kv overflow): pages evicted from GPU when queue spikes | v [Scheduler] -- stream tokens back as they emerge (no wait for full sequence) | v [Serving] -- SSE/gRPC stream to client | v client ``` The interesting moves: - **Heterogeneous layer placement.** A 70B-class model on a single $1500 box (2× used 3090 + 128GB DDR5) is realistic if we put the tail layers on CPU. Throughput drops 2–3× vs all-GPU but cost drops 20×. For batch-size-1 chat with small concurrent user counts, that's the right trade. - **NVMe as kv-cache overflow.** vLLM's PagedAttention assumes VRAM is the bound. On cheap boxes it isn't; total context budget is. Spilling cold kv pages to NVMe is a 10× capacity multiplier at the cost of a controlled latency penalty on reactivation. Nobody ships this in serving. - **Speculative decoding routing.** Run a 1B draft on CPU or a small GPU; run the 70B target on the cheap fleet. Spec-decode is a 2–3× throughput win when the draft is well-matched; almost no serving runtime exposes it cleanly. ## Heterogeneous fleet, multi-box ``` +-------------------+ | ORCHESTRATOR | | (HA pair) | +---------+---------+ | +------------------------+--------------------------+ | | | +----+----+ +----+----+ +----+----+ | BOX A | | BOX B | | BOX C | | | | | | | | 4x 3090 | | 2x 4090 | | 8x 7900 | | EPYC | | TR Pro | | XTX | | 256GB | | 128GB | | EPYC | +---------+ +---------+ +---------+ warm: 70B Q4 warm: 8B fp16 warm: 70B fp8 warm: 8B cold: 13B draft host (rocm) ``` Each box advertises its **capability map** to the orchestrator: which models it has warm, which it can warm quickly, current queue depth, thermal state. The orchestrator routes against that map. Naive round-robin treats this fleet badly; capability-aware routing recovers ~2× aggregate throughput. This is exactly where vLLM and TGI underperform and where a control plane built around heterogeneity from the start can win. Also exactly the kind of thing that benefits from a Rust implementation — concurrent, latency-sensitive, operationally observable, and we want it to *not fall over at 3am*. ## Quantization and accuracy posture Default to **kv-cache-aware quantization**: Q4_K_M or IQ4_XS for weights, Q8 or fp16 for kv-cache, fp16 for attention math. Produces output indistinguishable from fp16 on most production benchmarks while halving VRAM. Customers who want fp16/bf16 end-to-end can have it; their box serves fewer concurrent requests or runs a smaller model. The default is the cheap-hardware default because it is also the right answer for most workloads, and we are willing to say so. We do not ship Q2 or Q3 defaults. Below Q4 the accuracy loss is real and the support burden of "the model gave a weird answer" goes up. Q3 is available as an option for customers who know what they want; it is not the default that surprises someone. ## What we are explicitly not building - **Training.** Not in v1, not in v3. Inference is a different software discipline and the operation should learn to do one thing first. - **A novel tensor library.** We bind to `cudarc`, `candle` / `burn` primitives where solid, `wgpu` for the universal fallback, and write the scheduler and runtime — not the kernels. Writing our own kernels is a four-year detour we do not need. - **A model zoo of our own.** We serve the open models that exist: Llama, Qwen, Mistral, DeepSeek, Phi, Whisper, Flux when image makes sense. We do not train or fine-tune as a product. - **Kubernetes integration as a first-class feature.** Customers who want k8s can wrap our HTTP API. We will not add complexity to the runtime to accommodate it. - **A web UI.** TUI and structured telemetry exporters are the operator surface. Grafana exists and consumes our Prometheus output cleanly. ## Build order Each phase is a working artifact, not a feature flag. **Phase 0 — Single box, single model, single user (8 weeks).** Rust binary that loads a quantized 8B or 70B model, exposes OpenAI-compatible HTTP, runs on whatever GPU is plugged in. Continuous batching, paged kv, basic Prometheus export. TUI dashboard, one screen, real telemetry. The angel-raise demo. **Phase 1 — Single box, heterogeneous placement (12 weeks).** Per-layer placement across mixed GPUs and CPU. kv-cache NVMe overflow. Quantization-aware kernel selection. BMC agent: structured telemetry, fan curves, thermal-aware power capping. **Phase 2 — Multi-box orchestrator (16 weeks).** Orchestrator with capability map and routing. Speculative decoding with draft on separate device or box. Model registry: warm/cold state per box, rolling model loads. First customer-deployed cluster. **Phase 3 — Hardening and openness (ongoing).** Publish BMC agent and runtime as open repos. Reference benchmarks against vLLM and TGI on equivalent hardware, published honestly. Documentation thorough enough that a hardware resident can read it in their first month. ## What this sketch leaves open - Exact factoring across the named crates: which logic lives in `runtime` vs `hal` vs `scheduler`. Phase 0 decision. - Whether the BMC agent shares any code with the host runtime. Probably not, given the different security and reliability constraints, though `everycycle-hal` may expose a thin shared schema crate. - HA story for the orchestrator. Phase 2 problem. - Multi-tenant isolation guarantees. Start single-tenant per box; add per-tenant isolation when a customer asks. ## What we would push back on if proposed - "Let's also write the kernels in Rust." No. Bind to `cudarc` and friends. The scheduler is where time should go. - "Let's support training as a stretch goal." No. Different software discipline, different customer, different cadence. - "Let's start with the orchestrator." No. Phase 0 must be a single binary that runs on one box and looks great in a demo. The orchestrator is Phase 2. - "Let's target H100s for the demo." No. The demo runs on hardware that cost $2000, which is the point.