| 1 |
# Contributing to audiofiles |
| 2 |
|
| 3 |
Patterns, conventions, and rules for working on the audiofiles codebase. |
| 4 |
|
| 5 |
## Project Structure |
| 6 |
|
| 7 |
``` |
| 8 |
audiofiles/ (workspace root) |
| 9 |
Cargo.toml # Workspace definition |
| 10 |
crates/ |
| 11 |
audiofiles-core/ # Domain logic, storage, VFS, analysis, database |
| 12 |
audiofiles-browser/ # eframe/egui GUI, Backend trait, state management |
| 13 |
audiofiles-app/ # App entry point (thin shell) |
| 14 |
audiofiles-sync/ # SyncKit cloud sync integration |
| 15 |
audiofiles-rhai/ # Device plugin runtime (TOML manifests + Rhai hooks) |
| 16 |
audiofiles-bench/ # Benchmarks (dev-only binary) |
| 17 |
audiofiles-rhai/plugins/bundled/ # Bundled device profiles (SP-404, MPC, etc.) |
| 18 |
dist/ # Build scripts for macOS, Windows, Linux |
| 19 |
``` |
| 20 |
|
| 21 |
### Crate Boundaries |
| 22 |
|
| 23 |
|
| 24 |
|
| 25 |
| `audiofiles-core` | All domain logic | **Sync-only** (no async runtime) | |
| 26 |
| `audiofiles-browser` | GUI + backend abstraction | Owns egui state, polls workers | |
| 27 |
| `audiofiles-app` | Entry point | Thin shell, creates backend + launches GUI | |
| 28 |
| `audiofiles-sync` | Cloud sync | Uses tokio + `spawn_blocking` for rusqlite | |
| 29 |
| `audiofiles-rhai` | Device plugins | TOML manifests + sandboxed Rhai hooks | |
| 30 |
| `audiofiles-bench` | Benchmarks | Dev-only, not shipped. Needs a corpus (`scripts/corpus.py`) | |
| 31 |
|
| 32 |
**Critical rule:** `audiofiles-core` is entirely synchronous. `rusqlite::Connection` is `!Send`, so all database operations are synchronous. Long-running operations (import, analysis, export) use dedicated worker threads with channel-based message passing. |
| 33 |
|
| 34 |
## Content-Addressed Storage |
| 35 |
|
| 36 |
Samples are identified by SHA-256 hash. The hash IS the primary key. There are no UUIDs or auto-increment IDs for samples. |
| 37 |
|
| 38 |
```rust |
| 39 |
pub struct SampleStore { |
| 40 |
root: PathBuf, |
| 41 |
} |
| 42 |
``` |
| 43 |
|
| 44 |
**Import flow:** |
| 45 |
1. Stream file through SHA-256 hasher |
| 46 |
2. Check if blob already exists at `samples/{hash}.{ext}`; if so, skip copy (dedup) |
| 47 |
3. `INSERT OR IGNORE INTO samples`, dedup at DB level too |
| 48 |
4. Return the hash |
| 49 |
|
| 50 |
**Rules:** |
| 51 |
- Never store samples by filename or path. The hash is the only identifier. |
| 52 |
- The `cloud_only` flag marks samples whose local blobs have been evicted but still exist in cloud sync. |
| 53 |
- Hash validation rejects anything that isn't exactly 64 lowercase hex characters. |
| 54 |
- Extension validation rejects directory traversal attempts. |
| 55 |
|
| 56 |
## Worker Thread Pattern |
| 57 |
|
| 58 |
Long-running operations use dedicated threads with channel-based message passing: |
| 59 |
|
| 60 |
```rust |
| 61 |
pub struct WorkerHandle { |
| 62 |
cmd_tx: mpsc::Sender<WorkerCommand>, // Send commands to worker |
| 63 |
event_rx: Mutex<mpsc::Receiver<WorkerEvent>>, // Receive results |
| 64 |
cancel_flag: Arc<AtomicBool>, // Lock-free cancellation |
| 65 |
thread: Option<JoinHandle<()>>, // Join on drop |
| 66 |
} |
| 67 |
``` |
| 68 |
|
| 69 |
**Command/Event protocol:** |
| 70 |
- `WorkerCommand::AnalyzeBatch { samples, config }` → worker processes samples |
| 71 |
- `WorkerEvent::Progress { completed, total, current_name }` → UI updates progress bar |
| 72 |
- `WorkerEvent::SampleDone { result, suggestions }` → UI stores result |
| 73 |
- `WorkerEvent::BatchComplete` → UI finishes operation |
| 74 |
|
| 75 |
**Rules:** |
| 76 |
- `Mutex<Receiver>` satisfies `Send + Sync` requirements for egui state. |
| 77 |
- `Arc<AtomicBool>` for lock-free cancellation; worker checks before each sample. |
| 78 |
- `Drop` implementation sends `Shutdown` command and joins the thread. |
| 79 |
- The browser crate calls `try_recv()` each frame to poll for events. |
| 80 |
|
| 81 |
## Backend Trait |
| 82 |
|
| 83 |
`Backend` is an async trait that abstracts all data operations: |
| 84 |
|
| 85 |
```rust |
| 86 |
pub trait Backend: Send + Sync { |
| 87 |
fn list_vfs(&self) -> BackendResult<Vec<Vfs>>; |
| 88 |
fn import_file(&self, path: &Path) -> BackendResult<String>; |
| 89 |
fn start_analysis(&self, samples: Vec<(String, String)>, config: AnalysisConfig) -> BackendResult<()>; |
| 90 |
fn poll_events(&self) -> Vec<BackendEvent>; |
| 91 |
// ... ~30 methods covering VFS, tags, search, analysis, export |
| 92 |
} |
| 93 |
``` |
| 94 |
|
| 95 |
`DirectBackend` is the sole implementation, wrapping `Mutex<Database>` + `SampleStore`. This indirection exists to support potential future backends without changing UI code. UI code always calls `backend.method()`, never accesses the database directly. |
| 96 |
|
| 97 |
## VFS Abstraction |
| 98 |
|
| 99 |
Users organize samples through virtual file systems (VFS), not by moving files on disk. |
| 100 |
|
| 101 |
- `vfs_nodes` is a self-referential tree (`parent_id` FK to own table) |
| 102 |
- Nodes are either `Directory` or `Sample` (with a `sample_hash` FK) |
| 103 |
- A sample can appear in multiple VFS locations without duplication |
| 104 |
- Move/rename operations only touch VFS metadata, not the blob store |
| 105 |
- `UNIQUE(vfs_id, parent_id, name)` prevents duplicate names in the same directory |
| 106 |
|
| 107 |
**Enriched queries** join VFS nodes with analysis data and sample metadata in a single query, avoiding N+1 patterns. |
| 108 |
|
| 109 |
## Analysis Pipeline |
| 110 |
|
| 111 |
The pipeline runs in a worker thread and processes each sample through these stages: |
| 112 |
|
| 113 |
1. **Decode**: Symphonia decodes any audio format to mono f32 |
| 114 |
2. **Loudness**: Peak dB, RMS dB, LUFS (fast, uses full signal) |
| 115 |
3. **Spectral**: STFT → centroid, flatness, rolloff, bandwidth, ZCR, onset strength |
| 116 |
4. **MFCC**: Extract MFCCs from magnitude frames into the persisted 35-feature vector |
| 117 |
5. **BPM**: Tempo detection (skipped for non-rhythmic samples if `smart_skip` enabled) |
| 118 |
6. **Key**: Musical key detection (skipped for non-pitched samples if `smart_skip` enabled) |
| 119 |
7. **Loop**: Loop point detection |
| 120 |
8. **Fingerprint**: Peak envelope for near-duplicate detection (VP-tree similarity search) |
| 121 |
|
| 122 |
All results stored in `audio_analysis` table (one row per hash). The `smart_skip` feature reads cheap raw features (duration, spectral flatness) to skip stages that cannot apply, e.g. no BPM detection for a clip too short to carry tempo. |
| 123 |
|
| 124 |
### Adding a New Analysis Stage |
| 125 |
|
| 126 |
1. Add the computation in `crates/audiofiles-core/src/analysis/` |
| 127 |
2. Add column(s) to `audio_analysis` table via inline migration in `db.rs` |
| 128 |
3. Wire into the pipeline in `analysis/mod.rs` |
| 129 |
4. Add to `AnalysisResult` struct |
| 130 |
5. Expose in the enriched VFS query if needed for the UI |
| 131 |
|
| 132 |
## Unsafe FFI |
| 133 |
|
| 134 |
Platform-specific drag-and-drop requires FFI: |
| 135 |
- **macOS:** `drag_out/macos.rs`: objc2 message sends, libdispatch async to main thread |
| 136 |
- **Windows:** `drag_out/windows.rs`: COM/OLE `DoDragDrop` |
| 137 |
|
| 138 |
**Rules:** |
| 139 |
- Every `unsafe` block MUST have a `// SAFETY:` comment explaining the invariant. |
| 140 |
- macOS FFI uses `MainThreadMarker` to guarantee AppKit calls happen on the main thread. |
| 141 |
- `RcBlock` prevents use-after-free in async dispatch. |
| 142 |
- The `DRAG_ACTIVE` atomic flag prevents concurrent drag sessions. |
| 143 |
|
| 144 |
## Error Handling |
| 145 |
|
| 146 |
Three error types, one per major crate: |
| 147 |
|
| 148 |
```rust |
| 149 |
// audiofiles-core |
| 150 |
pub enum CoreError { |
| 151 |
Db(rusqlite::Error), |
| 152 |
Io { path: PathBuf, source: std::io::Error }, |
| 153 |
SampleNotFound(String), |
| 154 |
VfsNotFound(VfsId), |
| 155 |
Analysis(AnalysisError), |
| 156 |
// ... |
| 157 |
} |
| 158 |
|
| 159 |
// audiofiles-sync |
| 160 |
pub enum SyncError { |
| 161 |
Db(rusqlite::Error), |
| 162 |
Client(String), |
| 163 |
Auth(String), |
| 164 |
Io(std::io::Error), |
| 165 |
} |
| 166 |
|
| 167 |
// audiofiles-browser |
| 168 |
pub enum BackendError { |
| 169 |
Core(CoreError), |
| 170 |
Other(String), |
| 171 |
} |
| 172 |
``` |
| 173 |
|
| 174 |
Use typed variants with context. Use `?` for propagation. `From` impls enable automatic conversion between error types. |
| 175 |
|
| 176 |
## Database |
| 177 |
|
| 178 |
### Inline Migrations |
| 179 |
|
| 180 |
Migrations are `const` strings in `crates/audiofiles-core/src/db.rs`, applied sequentially on database open: |
| 181 |
|
| 182 |
```rust |
| 183 |
const MIGRATION_001: &str = r#" |
| 184 |
CREATE TABLE samples ( |
| 185 |
hash TEXT PRIMARY KEY, |
| 186 |
original_name TEXT NOT NULL, |
| 187 |
... |
| 188 |
); |
| 189 |
"#; |
| 190 |
``` |
| 191 |
|
| 192 |
Production uses a file-backed SQLite database. Tests use `:memory:`. |
| 193 |
|
| 194 |
When adding a migration, make it **replay-safe**: every `CREATE TABLE / INDEX / TRIGGER` should be `IF NOT EXISTS` (or preceded by `DROP IF EXISTS` for triggers whose body changes), and any seed insert should be `INSERT OR IGNORE`. The `migration_replay_from_version_two_against_full_schema` test in `db.rs` rolls `user_version` back to 2 and re-runs every migration from M003 onward against a populated schema; non-idempotent CREATEs fail it. M001 (initial schema) and M002 (`DROP TABLE tags; ALTER tags_v2 RENAME TO tags`) are inherently one-shot and excluded from the replay test. |
| 195 |
|
| 196 |
The connection registers a custom `hash_row_id(salt, key)` SQLite function on open (rusqlite `functions` feature). It's used by the M018 sync triggers; if you write a migration that creates new sync triggers, prefer it for any row_id that would otherwise leak user content. |
| 197 |
|
| 198 |
### Sync Changelog Triggers |
| 199 |
|
| 200 |
Every synced table has triggers that insert into `sync_changelog` on INSERT/UPDATE/DELETE. A `sync_state` row (`applying_remote = '1'`) suppresses triggers during pull operations to prevent recursion. |
| 201 |
|
| 202 |
Per migration M018, `sync_changelog.row_id` is hashed via `hash_row_id(row_id_salt, canonical_key)` for sensitive tables (samples, audio_analysis, tags, collection_members) so the server never sees raw sample hashes or tag strings. The salt is generated per device, stored in `sync_state`, never synced. DELETE triggers also emit the canonical PK in the encrypted `data` field, which `resolve::apply_delete` reads to reconstruct WHERE clauses without parsing the (now-opaque) row_id. When adding a new synced table, follow the same pattern: wrap row_id in `hash_row_id(...)` if it carries user content, and emit the canonical PK into `data` for DELETE. |
| 203 |
|
| 204 |
### rusqlite + async |
| 205 |
|
| 206 |
`rusqlite::Connection` is `!Send`. In the sync crate (which uses tokio), all database operations go through `tokio::task::spawn_blocking`. In core (sync-only), no async runtime is needed. |
| 207 |
|
| 208 |
## SyncKit Integration |
| 209 |
|
| 210 |
Cloud sync is optional. The `SyncManager` coordinates push/pull: |
| 211 |
|
| 212 |
- **Tables synced** (in FK-safe order): `vfs`, `samples`, `collections`, `vfs_nodes`, `audio_analysis`, `tags`, `collection_members`, `user_config`, `edit_history` (smart_folders merged into `collections.filter_json` in M015 and the standalone table dropped) |
| 213 |
- **Delete order** is reversed (children first) |
| 214 |
- **Column whitelist:** `table_columns()` restricts which columns sync to prevent schema drift |
| 215 |
- **Blob sync:** Sample files sync to cloud storage for VFS entries with `sync_files = true` |
| 216 |
- **`cloud_only` flag:** Marks samples whose local blobs have been evicted |
| 217 |
|
| 218 |
## Device Plugins |
| 219 |
|
| 220 |
TOML manifests in `plugins/devices/` define hardware constraints. Optional Rhai scripts in `hooks/` run sandboxed. |
| 221 |
|
| 222 |
### Hook Style |
| 223 |
|
| 224 |
Optional Rhai hooks follow the cross-project Rhai style guide. Run `_private/scripts/lint-rhai.sh` to check formatting. Key points: 4-space indent, `snake_case` functions, `UPPER_CASE` constants, header comment block. |
| 225 |
|
| 226 |
### Manifest Contract |
| 227 |
|
| 228 |
```toml |
| 229 |
[device] |
| 230 |
name = "SP-404 MKII" |
| 231 |
manufacturer = "Roland" |
| 232 |
|
| 233 |
[audio] |
| 234 |
formats = ["wav"] |
| 235 |
sample_rates = [44100, 48000] |
| 236 |
bit_depths = [16, 24] |
| 237 |
channels = "both" |
| 238 |
|
| 239 |
[naming] |
| 240 |
case = "upper" |
| 241 |
max_length = 12 |
| 242 |
|
| 243 |
[hooks] |
| 244 |
validate_sample = "hooks/validate.rhai" |
| 245 |
transform_filename = "hooks/filename.rhai" |
| 246 |
``` |
| 247 |
|
| 248 |
### Hook Functions |
| 249 |
|
| 250 |
|
| 251 |
|
| 252 |
| `validate_sample` | `info` (sample metadata) | `bool` | Accept/reject sample for device | |
| 253 |
| `transform_filename` | `name`, `ctx` | `String` | Rename for device conventions | |
| 254 |
| `pre_export` | `ctx` | none | Run before export batch | |
| 255 |
| `post_export` | `ctx` | none | Run after export batch | |
| 256 |
|
| 257 |
## Concurrency |
| 258 |
|
| 259 |
- `parking_lot::Mutex` everywhere (not `std::sync::Mutex`), no poisoning, shorter lock API. |
| 260 |
- `#[instrument(skip_all)]` on all significant functions. |
| 261 |
- Worker threads for long-running operations (never block the UI thread). |
| 262 |
- The egui render loop polls `backend.poll_events()` each frame for worker results. |
| 263 |
|
| 264 |
## Testing |
| 265 |
|
| 266 |
- **Core tests:** In-file `#[cfg(test)]` modules with `test_helpers::insert_fake_sample` for fixtures |
| 267 |
- **Sync tests:** Unit tests in sync crate modules |
| 268 |
- **No GUI tests:** Immediate-mode UI is tested manually |
| 269 |
- Test databases use in-memory SQLite (`:memory:`) with migrations applied via `Database::open` |
| 270 |
|
| 271 |
## Building and Distribution |
| 272 |
|
| 273 |
Summary of platform-specific builds: |
| 274 |
|
| 275 |
|
| 276 |
|
| 277 |
| macOS arm64 | Native cargo, signed + notarized DMG | |
| 278 |
| Windows x86_64 | `cargo-xwin` cross-compile, MSI + EXE | |
| 279 |
| Linux aarch64 | Native cargo on Astra, AppImage + .deb | |
| 280 |
| Linux x86_64 | Cross-compile on Astra, AppImage + .deb | |
| 281 |
|
| 282 |
Every release must have all 7 artifacts before uploading. |
| 283 |
|