max / everycycle

23.3 KB · 216 lines History Blame Raw

1	# EveryCycle Roadmap
2
3	This is the live thread roadmap. Work advances on parallel threads, not numbered phases. The project's "current state" is the snapshot of where every thread stands together.
4
5	When a thread is ready to advance, the next milestone on it is the next bit of work — independent of where other threads are. A resident or intern can take a thread or even a single milestone without having to follow a linear sequence.
6
7	## Identity
8
9	EveryCycle is a Linux system service for heterogeneous GPU compute. The daemon (`everycycled`) owns the GPU fleet on a host and intermediates between the system and the silicon — across vendors, generations, and the long tail of hardware whose vendors have moved on.
10
11	The category EveryCycle is creating is the GPU-compute analog of PipeWire (audio), NetworkManager (networking), CUPS (printing): a system service that owns a device class and provides a stable API to userspace, replacing the chaos in which every program directly drives the GPU via vendor SDKs.
12
13	## The product and the business
14
15	EveryCycle is software. The daemon (`everycycled` + clients + libraries + operator tools) is the product, MIT-licensed, distributed via standard distro packaging.
16
17	Make Creative, LLC is a for-profit company with good intentions. It funds the software through TailoredMachines — a small-batch atelier that cuts heterogeneous-GPU Linux workstations pre-configured with EveryCycle. TailoredMachines revenue pays for the open-source work. We sell machines, not compute-as-a-service.
18
19	MNW is a TailoredMachines customer like any other: it buys machines from the TailoredMachines line at fair internal-transfer pricing and runs its own workloads on them. External sales and internal sales are the same business — both fund EveryCycle.
20
21	This is a clean separation of concerns: software identity lives in EveryCycle, hardware design and revenue live in TailoredMachines, and MNW is a customer. See the TailoredMachines plan and its deeper brand and product docs.
22
23	## Wedge
24
25	1. Heterogeneity is the default. Cheap, mixed-generation, mixed-vendor silicon serves real workloads when the runtime above the kernels is honest about the hardware. We do not assume a homogeneous fleet.
26	2. Defensive curation. When vendors abandon old hardware, EveryCycle catches it. We watch for support discontinuation, fork upstreams when they drop too aggressively, and partner with OSS driver and kernel projects (nouveau, NVK, Mesa, RADV) to extend useful life past vendor end-of-life. Old hardware that can still serve at all is a first-class supported tier.
27
28	## First named workloads
29
30	TailoredMachines has narrowed the appliance line's target to 1–6 programmers per box doing bursty, high-intensity inference (see the TailoredMachines plan). That narrowing gives EveryCycle two named reference workloads instead of "generic LLM serving":
31
32	- Interactive pair-programming (reference: the pair project, `~/Documents/Content/projects/pair/`). Single-user batch-1, latency-first, prefix-cached repo context, spec-decoded token stream. This is EveryCycle Thread A1/A2's exact shape and pair is the first named internal customer against which the batch-1 features are designed.
33	- Audit / fuzz sweeps (reference: the operator's Claude skill library — `code-fuzz`, `test-fuzz`, `doc-fuzz`, `nitpick`, `audit`, `ultra-fuzz`). Non-interactive, long-form, whole-repo scans. Naturally scheduled as background work between interactive turns. This is Thread C's preemption + QoS-classes story with a real workload attached.
34
35	Together these two workloads exercise Thread C's fair-share + preemption path at small N, which is exactly the scale a TM House Cut runs at. Design against them explicitly.
36
37	The work is curatorial and operational, not greenfield kernel engineering. We bind to existing kernel collections (ggml-cuda for NVIDIA, ggml-rocm for AMD) vendored in-tree so we can fork when needed. The novel work is the layer above: the supervisor, the planner, the audit, the operator UX, the client API.
38
39	## Platform and distribution
40
41	- Linux only by design. macOS-native is not on the roadmap. Apple Silicon support, if it comes, comes via Asahi Linux.
42	- A Fedora downstream is the reference target, loosely. Loosely is the operative word: the codebase is generic Linux, the reference host is where it is built and probed first, and nothing here is a hard pin. Other distros stay best-effort. EveryCycle's target is the box with the GPUs in it, not the box serving the website, so the reference host does not track MNW production's distro and the two test matrices are separate.
43	- Glibc build, RPM packaging. The daemon is a standard glibc binary. Executor isolation (in-process dlopen vs out-of-process sidecar) is decided per executor by vendor SDK stability and crash blast-radius, not by libc compatibility.
44	- Immutability is the one thing that is genuinely different, and it is a kernel-module question. An image-based host (bootc, rpm-ostree) has no DKMS at update time, so a vendor kernel module cannot be built on the machine the way `nvidia-dkms` does on Ubuntu. It has to be baked into the image or layered as an akmod. Everything else about an immutable host is friendlier than it sounds: `/opt` and `/usr/local` are `/var`-backed and writable, so CUDA and ROCm install where their installers expect, `/etc` and `/var` persist, and systemd, udev and `/dev` are unchanged. Treat the module delivery path as real work and the rest as packaging format.
45
46	## Threads
47
48	### Thread A — GPU complexity
49
50	\| Milestone \| Cards \| Approx cost \| What it teaches \|
51	\|---\|---\|---\|---\|
52	\| A1 \| 1× Tesla P40 (24 GB GDDR5, Pascal) + GT 710 boot card \| ~$230 \| Audit, plan, serve, ggml-cuda wrap, first defensive-curation regression suite. Pascal is the next NVIDIA shoe to drop, so the wedge is in immediate practice. \|
53	\| A2 \| 2× Tesla P40 \| ~$400 \| Same-executor tensor-split, per-card capability reporting, planner spreads across same-vendor devices. \|
54	\| A3 \| 1× P40 + 1× P100 (16 GB HBM2) \| ~$500 \| Mixed-arch within one executor (cc 6.1 + cc 6.0, different memory bandwidth and fp16 support). Capability-aware placement. \|
55	\| A4 \| 1× P40 + 1× MI50 (16 GB HBM2, Vega 20) \| ~$550 \| Inter-executor activation protocol; AMD executor; cross-vendor supervisor coordination; the wedge demo. Cross-vendor enters here, not earlier. \|
56	\| A5 \| + 3090 + Arc card \| varies \| Real heterogeneous fleet; planner makes real tradeoffs. \|
57
58	Single-vendor Pascal-era Tesla through A3. Cross-vendor enters at A4. The executor trait must be forward-compatible with cross-vendor handoff from A1 even though no impl uses it until A4.
59
60	### Thread B — Model complexity
61
62	Smaller models first. Qwen family is the default — open weights, well-quantized in the wild, multiple sizes that step up cleanly. The progression demonstrates the runtime, not the model.
63
64	\| Milestone \| Model \| Why \|
65	\|---\|---\|---\|
66	\| B1 \| Qwen 3 0.6B Q8 (or Qwen 2.5 0.5B Q8) \| Smoke test. Validates audit → plan → serve end-to-end with a model that loads in seconds. \|
67	\| B2 \| Qwen 3 1.7B Q4 (or Qwen 2.5 1.5B Q4) \| First "useful" responses. Streaming tokens, OpenAI-compatible chat. \|
68	\| B3 \| Qwen 3 8B Q4 (or Qwen 2.5 7B Q4) \| The workhorse demo model. Fits a Tesla P40 with comfortable kv-cache headroom. \|
69	\| B4 \| Qwen 2.5 32B Q4 \| Pushes the P40 harder; still fits 24 GB. \|
70	\| B5 \| Qwen 2.5 72B Q4 \| Saturates the P40's VRAM; first encounter with kv-cache pressure that wants NVMe overflow. \|
71	\| B6 \| Frontier-size with offload \| Layer placement across multiple devices + CPU + NVMe. Couples tightly with A4/A5 on the GPU thread. \|
72
73	### Thread C — Supervisor and multi-client arbitration
74
75	The supervisor owns the device fleet on behalf of concurrent clients. Single-tenant operation is not a stable resting point — it's a degenerate case of the multi-client design. PipeWire is fundamentally multi-tenant from day one; so is EveryCycle.
76
77	Design scale: N=1–6 cooperating programmers per box (the TM House Cut scale). Multi-tenant fair-share, preemption, and QoS classes matter at small N because the reference workloads (interactive pair vs. background audit/fuzz) routinely coexist on the same box — a fuzz sweep and a pair session sharing one GPU is the normal case, not an edge case. This is not a datacenter multi-tenant story where N is thousands and tenants are adversarial; it's a small-team story where the classes are cooperating and the QoS gap between "interactive latency-sensitive" and "background throughput-happy" is the load-bearing distinction.
78
79	Shape: enumerate-and-dispatch (A1, single client) → two-client arbitration with priority + fair-share (A2, the load-bearing milestone for the analogy) → crash isolation (per-executor watchdog + structured panic handling) → hot-plug → preemption and QoS classes → multi-box fleet routing.
80
81	Two clients sharing a GPU under daemon-enforced policy is a required A2 demo, not a late-roadmap luxury. L5 (preemptive multi-tenant scheduling) moves into the early path on this thread. Classifying workloads by QoS class (interactive vs. background, matching the pair vs. audit/fuzz split) lands with or shortly after A2 — earlier than "preemption and QoS classes" appears in the shape above suggests, because TM's House Cut assumes it.
82
83	### Thread D — Audit depth
84
85	Open. Shape: enumeration (boot, ms) → calibration cache (first-boot per fingerprint, seconds) → live profiling (background, minutes) → drift detection (continuous; recalibrate when observed perf deviates from cache by N%).
86
87	### Thread E — Defensive curation
88
89	Open. Shape: regression suite stub (just records what hardware passes today) → watch list of vendors' end-of-life announcements (Pascal, Vega, Polaris are the near-term candidates) → first OSS partnership (likely nouveau/NVK for Pascal post-CUDA, or RADV for Vega post-ROCm) → first real fork-and-maintain event when an upstream drops a generation we promised to keep alive.
90
91	### Thread F — BMC and hardware product
92
93	Open. Shape: wrap stock Redfish on whatever BMC is on the TailoredMachines Platform board (ASRock Rack AST2600) → flash OpenBMC → bmc-agent emits structured telemetry → scheduler consumes thermal data for placement decisions. This thread runs in parallel with the runtime work; it does not block A1.
94
95	### Thread G — Operator UX
96
97	Open. Shape: audit screen (the brand moment — beautiful, structured, says "this runtime understands your hardware") → live ops dashboard → multi-box view → incident response views. The TUI exists as `everycycle-tui` already; what's missing is the actual screen content.
98
99	### Thread H — Appliance line (the business that funds the software)
100
101	The hardware line is how the operation pays for EveryCycle's development. Machine BOMs, sourcing routes and pricing live in the TailoredMachines wiki rather than in this repo.
102
103	The line is a small-batch atelier cutting workstations for 1–6 programmers doing bursty pair-programming and audit/fuzz workloads. The product architecture is a single Platform SKU (v1: "The House Cut") plus three explicit GPU paths (Alterations / Off-the-Rack / The Collection). See the TailoredMachines plan and brand docs.
104
105	Thread H's active work happens against TailoredMachines' plan, not against this repo.
106
107	### Thread I — Client API and adoption
108
109	This is the long-pole project. PipeWire's decade of distro defaults and upstream integrations was the work; same applies here. The thread starts at A1, not after A3 — the API surface clients write against has to be designed before the runtime locks shape around its first internal users.
110
111	Shape: native client API sketch (A1, alongside the executor trait) → API v0 published with explicit "unstable" marker (A2) → reference shims and SDKs (A3) → API v1 freeze with semver and compat charter (A4 or when the second external client ships against it, whichever first) → distro packaging and default-daemon advocacy (multi-year) → upstream integrations as alternative backends in llama.cpp / vLLM / TGI / ComfyUI / etc.
112
113	The v1 freeze is a gate, not a date. Until it lands, every client is opting into churn; after it lands, breaking the API requires a major version and a migration story. This commitment is what makes EveryCycle a platform rather than a tool.
114
115	### Thread K — Batch-1 latency features (for pair and interactive audit)
116
117	Interactive workloads (pair-programming, single-turn audit responses) live or die on per-turn latency, not throughput. The features that move that number don't come for free from Threads A/D — they need their own thread because pair's roadmap explicitly depends on them landing on EveryCycle rather than being maintained as a private vLLM config.
118
119	Shape:
120
121	\| Milestone \| Feature \| Why it matters for the named workloads \|
122	\|---\|---\|---\|
123	\| K1 \| Prefix caching across turns \| Repo context isn't re-prefilled every turn. Highest impact-per-hour lever for pair; also helps interactive audit's "here's my repo, look at file X" patterns. Ships alongside B2/B3. \|
124	\| K2 \| Speculative decoding with per-model draft \| 2–3× throughput at batch 1 when the draft is tuned. Requires draft-model registry and coordinated scheduling. Ships alongside A2 (so the second device can host the draft). \|
125	\| K3 \| KV-cache quantization (Q8 default, mixed-precision recipe per model) \| Halves KV pressure with negligible accuracy loss on production benchmarks. Ships wherever the appraisal work first exposes precision knobs to the scheduler. \|
126	\| K4 \| Adaptive reasoning budget (per-request `effort` knob) \| Terse on a rename, deliberate on architecture. Requested by pair Phase 5; needs a scheduler-visible knob so the batcher can plan around variable per-request compute. \|
127
128	K1 is the single most impactful feature on this thread and should ship as soon as the request path is stable enough to hang a cache off of. K2–K4 are gated on measurement — apply after the profile (pair e02, EveryCycle's own dogfooding) says they're the next real bottleneck.
129
130	### Thread J — Compatibility shims
131
132	The adoption mechanism. PipeWire displaced PulseAudio and JACK not by asking apps to port, but by shipping `pipewire-pulse` and `pipewire-jack` — drop-in replacements that spoke the existing protocols. Existing apps pointed at the new daemon without modification.
133
134	EveryCycle's analogs, in rough priority order:
135
136	\| Shim \| What it impersonates \| Why it matters \|
137	\|---\|---\|---\|
138	\| J1 \| OpenAI Chat Completions / Responses API \| Largest existing client ecosystem; every LLM tool already speaks it. \|
139	\| J2 \| llama.cpp server (`/completion`, `/v1/chat/completions` extensions) \| Captures the local-inference tool ecosystem (Open WebUI, LM Studio clients, etc.). \|
140	\| J3 \| Ollama API \| Captures the hobbyist surface. \|
141	\| J4 \| vLLM / TGI HTTP surfaces \| Captures production-inference clients without code change. \|
142	\| J5 \| ComfyUI / diffusion backend protocols \| Extends past text into image/video workloads on the same daemon. \|
143
144	J1 ships alongside B2 (first useful responses). The shims are separate processes that translate to the native EveryCycle client API — they are clients of Thread I's API, not special-cased daemon code. This keeps the native API honest: if a shim can't express something cleanly, that's a Thread I design problem, not a shim problem.
145
146	## Settled architectural decisions
147
148	- Backend approach: wrap vendored kernel collections per vendor. Don't write kernels from scratch. Vendor the source in-tree so we can fork when needed. NVIDIA executor wraps ggml-cuda; AMD executor wraps ggml-rocm; Apple executor wraps MLX; CPU executor is the always-present fallback.
149	- Cross-vendor layer placement: belongs to A4. The executor trait is forward-compatible from day one (activation-handoff signatures present even if no impl uses them); the inter-executor protocol gets built when A4 is reached.
150	- Project structure: parallel threads, not numbered phases. Each thread has its own pace; project state is the snapshot across all threads.
151	- First GPU: Tesla P40. Walks the defensive-curation talk on day one. Pascal facing imminent CUDA-legacy transition makes this card the canonical example of what EveryCycle exists to support.
152	- Host platform: dual-use with the TailoredMachines line. The substrate is a workstation-class host with ECC memory, a BMC, and Gen5 NVMe; GPUs are fungible above it. Exact BOMs live in the TailoredMachines wiki.
153	- First model family: Qwen, starting small. B1 is the smallest Qwen variant that loads; subsequent milestones step up. Choice driven by open weights, broad quant availability, and clean size laddering.
154	- Client API stability is a sacred commitment. Once Thread I reaches v1 freeze, the client-facing API follows strict semver: minor versions are additive, patch versions are bug-fix-only, major versions require a migration document and a deprecation window of at least one full release cycle on every supported distro. Before v1 freeze, the API is explicitly marked unstable and may break on any release. The freeze is the line that turns EveryCycle from a tool into a platform; everything downstream (compatibility shims, third-party clients, distro defaults) depends on this commitment being honored.
155	- Multi-client is the default, not a feature. The daemon assumes concurrent clients from A1, even when only one is connected. The internal data model, scheduler, and audit surface are designed around N clients sharing M devices under policy; the N=1 case is just the limit. This is the structural difference between a system service and a runtime library.
156	- Compatibility shims are clients, not daemon code. Each protocol shim (OpenAI, llama.cpp, Ollama, vLLM, ComfyUI) is a separate process that speaks the native EveryCycle client API. The shims do not get privileged access to internals. This forces the native API to be expressive enough that the shims are thin — and keeps the daemon free of protocol-specific code paths.
157	- Implementation languages. Main binary is Rust, built natively for the reference Fedora downstream (glibc, RPM-packaged). Module boundary is a stable C ABI from a reasonable point — early enough that the second executor doesn't lock the trait shape into Rust-internals. Module implementations may be in any language that emits a C-ABI shared library (Rust, Zig, C). The BMC agent is a separate binary and is the leading candidate for being written in Zig rather than Rust; that decision deferred until the agent is real.
158	- Crate naming. The daemon-internal native client API lives in `crates/api/`. Thread J shims (OpenAI, llama.cpp, Ollama, vLLM, ComfyUI) will land as their own crates (`crates/shim-openai`, etc.) when they ship — they are not modules of `crates/api`. The HTTP "serving" framing is gone from the daemon by design.
159
160	## Model stance
161
162	Open weights only, on principle. EveryCycle serves models whose weights are publicly available. We do not wire to closed-weight model APIs and we do not ship closed weights as part of the runtime distribution. Operators are free to load whatever weights they want — we just don't make non-open-weight serving the default path.
163
164	Models are commodities. We expect the model market to be fully commoditized: many capable open families, rapid generation turnover, no durable moat at any particular model. EveryCycle's architecture follows this stance:
165
166	- No model-specific kernels. Backend kernels handle architectures generically (attention, RMSNorm, rotary, MoE routing) via the wrapped kernel collections.
167	- No model-specific loaders. Standard formats (GGUF, safetensors) only; loaders for new architectures land in the wrapped backends, not in EveryCycle.
168	- No model registry of our own. Operators bring their own models; we serve them.
169	- The serving API stays OpenAI-compatible — that interface is stable across model churn.
170	- Marketing and demos focus on what the runtime does to the hardware, not on what model is loaded. The model is a workload, not a feature.
171
172	When a new model family emerges (Llama 4, Mistral Next, DeepSeek N+1, etc.), supporting it should be a backend-version bump, not an EveryCycle code change.
173
174	## Open questions (will be resolved iteratively)
175
176	- Executor isolation: in-process dlopen + watchdog threads + structured panic handling, or out-of-process with local-socket IPC. Per executor, driven by vendor SDK stability and crash blast-radius. In-process is the default for executors that behave; out-of-process when an executor empirically destabilizes the host process. Standardizing on one glibc reference host removes libc compatibility as a forcing factor, since CUDA and ROCm SDKs target glibc natively, so sidecar separation is a stability choice rather than a packaging workaround. That turns on glibc, not on which distro ships it.
177	- CPU executor placement: statically linked into the base binary (always-present fallback) vs another module
178	- A1 dispatch planner: greedy heuristic only, or invest in a real optimizer up front
179	- BMC agent's actual A1-equivalent role: just enumerate Redfish, or already running on flashed OpenBMC
180	- First-boot online vs offline default
181
182	## Long-arc vision: the OS-for-compute level decomposition
183
184	EveryCycle's identity grows over time toward "operating system for heterogeneous GPU compute." The progression, in increasing order of difficulty:
185
186	\| Level \| What \| Status \|
187	\|---\|---\|---\|
188	\| L0 \| Unified inventory (audit) \| Thread D, A1 \|
189	\| L1 \| Unified routing (per-request placement) \| Thread C, A1 \|
190	\| L2 \| Per-model layer placement (across same-vendor devices) \| Thread A, A2-A3 \|
191	\| L3 \| Virtual VRAM with paging (kv-cache first; weights later) \| Thread D, mid-roadmap \|
192	\| L4 \| Cross-device kernel sharding (same-vendor first, cross-vendor later) \| A4+ \|
193	\| L5 \| Preemptive multi-tenant scheduling \| Thread C, A2 (fair-share) → mid-roadmap (preemption + QoS classes) \|
194	\| L6 \| Transparent failure recovery (request-level early; mid-token much later) \| Mid roadmap and beyond \|
195	\| L7 \| Unified telemetry surface (operators reason about "the cluster," not individual devices) \| Thread G + Thread F \|
196	\| L8 \| Full programmability against the abstract device (clients write to EveryCycle's API, never to vendor SDKs) \| Thread I; the adoption goal \|
197
198	Levels through L4 (same-vendor) are achievable solo + residency. L4 (cross-vendor), full-fidelity L3 (weight paging), L5's preemption+QoS layer (fair-share at L5 is in scope solo), and L6 (mid-token recovery) are team-required and are honest team-scale R&D. Worth naming because the levels frame what each thread is for and where each milestone fits in the larger picture.
199
200	What this forbids:
201	- Performance parity across topologies. "One big GPU" means unified abstraction, not magical speed. Cross-vendor will be slower than single-vendor; that's physics.
202	- Hiding hardware completely. Operators still need to see what's there. The abstraction is programmer-facing; the operator surface stays physically honest.
203	- Forcing the abstraction. Workloads that need locality guarantees can pin. "One big GPU" is the opt-in default, not the only mode.
204
205	## Non-goals
206
207	Restated for emphasis:
208
209	- Not training. Not in v1, not in v3.
210	- Not writing kernels from scratch. We bind, we wrap, we fork.
211	- Not Kubernetes-first. Client API; orchestrators can wrap it.
212	- Not a web UI. TUI + Prometheus/OTel exporters.
213	- Not a model zoo of our own. We serve the open models that exist.
214	- Not macOS-native. Linux only; Apple Silicon via Asahi if at all.
215	- Not a license-paid product. Software is MIT; revenue comes from selling TailoredMachines workstations (external and internal-transfer to MNW). Not compute-as-a-service.
216