max / audiofiles
- Co-Authored-By
- Claude Opus 5 (1M context) <noreply@anthropic.com>
57 files changed,
+470 insertions,
-2610 deletions
| @@ -9,6 +9,9 @@ | |||
| 9 | 9 | ### Added | |
| 10 | 10 | - Forge: resample overshoot handling. Conforming to an integer target now detects true-peak overshoot (>1.0); by default the signal is left untouched and a warning is shown that it will clip. A new Settings > Forge toggle, "Auto-trim resample overshoot", opts in to the gentlest reversible fix (a single linear gain to full scale), reported when applied. 32-bit float targets are unaffected (lossless passthrough). | |
| 11 | 11 | ||
| 12 | + | ### Removed | |
| 13 | + | - The single sample-class label, and everything that consumed it. Measured at 33.4% accuracy on exact class match, with two of its seven drum classes unreachable by any rule; the features it read carry coarse family structure, not instrument identity. Browsing moves to the measured axes, instrument names to filename rules and your own tags, and "more like this" stays on k-NN. Gone with it: the Class column and its sort, the class filter checkboxes, the class badge, the `{class}` rename token, the `classification` field in the export sidecar, the per-class tag suggestions and their dismissal list, and the `classification` / `classification_confidence` database columns. A saved rename pattern containing `{class}` reports an unknown token rather than quietly dropping it, and a saved search that filtered on the class loses that criterion. | |
| 14 | + | ||
| 12 | 15 | ### Fixed | |
| 13 | 16 | - Store: importing identical bytes under a different file extension no longer leaves an unreachable orphan blob on disk; the import reuses the canonical content-addressed blob, restoring "import twice is a no-op". | |
| 14 | 17 |
| @@ -113,13 +113,13 @@ | |||
| 113 | 113 | 1. **Decode**: Symphonia decodes any audio format to mono f32 | |
| 114 | 114 | 2. **Loudness**: Peak dB, RMS dB, LUFS (fast, uses full signal) | |
| 115 | 115 | 3. **Spectral**: STFT → centroid, flatness, rolloff, bandwidth, ZCR, onset strength | |
| 116 | - | 4. **MFCC + ML**: Extract MFCCs from magnitude frames, run through embedded neural classifier | |
| 116 | + | 4. **MFCC**: Extract MFCCs from magnitude frames into the persisted 35-feature vector | |
| 117 | 117 | 5. **BPM**: Tempo detection (skipped for non-rhythmic samples if `smart_skip` enabled) | |
| 118 | 118 | 6. **Key**: Musical key detection (skipped for non-pitched samples if `smart_skip` enabled) | |
| 119 | 119 | 7. **Loop**: Loop point detection | |
| 120 | 120 | 8. **Fingerprint**: Peak envelope for near-duplicate detection (VP-tree similarity search) | |
| 121 | 121 | ||
| 122 | - | All results stored in `audio_analysis` table (one row per hash). The `smart_skip` feature uses ML classification to skip irrelevant stages (e.g., no BPM detection for ambient textures). | |
| 122 | + | All results stored in `audio_analysis` table (one row per hash). The `smart_skip` feature reads cheap raw features (duration, spectral flatness) to skip stages that cannot apply, e.g. no BPM detection for a clip too short to carry tempo. | |
| 123 | 123 | ||
| 124 | 124 | ### Adding a New Analysis Stage | |
| 125 | 125 |
| @@ -32,7 +32,7 @@ | |||
| 32 | 32 | ||
| 33 | 33 | | Crate | Path | Role | | |
| 34 | 34 | |-------|------|------| | |
| 35 | - | | `audiofiles-core` | `crates/audiofiles-core/` | Domain library. SQLite database, content-addressed store (SHA-256), audio decoding (Symphonia), analysis pipeline (loudness, BPM, key, spectral, classification), VFS, tag system, VP-tree similarity index. | | |
| 35 | + | | `audiofiles-core` | `crates/audiofiles-core/` | Domain library. SQLite database, content-addressed store (SHA-256), audio decoding (Symphonia), analysis pipeline (loudness, BPM, key, spectral features), VFS, tag system, VP-tree similarity index. | | |
| 36 | 36 | | `audiofiles-browser` | `crates/audiofiles-browser/` | Shared egui UI. File list, detail panel, waveform display, search/filter, import wizard, analysis progress, export, themes. | | |
| 37 | 37 | | `audiofiles-app` | `crates/audiofiles-app/` | Standalone desktop app via eframe. System audio (cpal), drag-and-drop import, native drag-out to Finder/DAWs, system tray, CLI import, OTA updates. | | |
| 38 | 38 | | `audiofiles-sync` | `crates/audiofiles-sync/` | Cloud sync via SyncKit. Pushes/pulls sample metadata, tags, and VFS structure across devices. E2E encrypted. | | |
| @@ -54,14 +54,14 @@ | |||
| 54 | 54 | ||
| 55 | 55 | ### Audio Analysis | |
| 56 | 56 | - **Analysis pipeline**: loudness (peak/RMS/LUFS), BPM detection, key detection, spectral analysis | |
| 57 | - | - **Classification**: deterministic DSP features feed a coarse `SampleClass` label plus a layered tag pipeline (user-authored rules, then k-NN over the samples you have already tagged). No trained model ships in the binary. See `docs/ml_classifier.md` | |
| 57 | + | - **Classification**: deterministic DSP features feed a layered tag pipeline (user-authored rules, then k-NN over the samples you have already tagged). Multi-label, and every tag records where it came from. No trained model ships in the binary. See `docs/ml_classifier.md` | |
| 58 | 58 | - **Loop detection**: identifies seamless loops via amplitude envelope analysis | |
| 59 | 59 | - **Similarity search**: VP-tree indexed fingerprinting for finding similar and duplicate samples (O(log n) lookup) | |
| 60 | 60 | - **Waveform display**: pre-computed peak data with click-to-seek playback | |
| 61 | 61 | ||
| 62 | 62 | ### Search & Filtering | |
| 63 | 63 | - **Text search**: FTS5 indexed across filenames, tags, and metadata | |
| 64 | - | - **Parameter filters**: BPM range, duration range, key selector, classification category | |
| 64 | + | - **Parameter filters**: BPM range, duration range, loudness range, key selector | |
| 65 | 65 | - **Tag prefix matching**: type a tag prefix to filter by hierarchy | |
| 66 | 66 | ||
| 67 | 67 | ### Editing | |
| @@ -91,7 +91,7 @@ | |||
| 91 | 91 | | What | Where | | |
| 92 | 92 | |------|-------| | |
| 93 | 93 | | Domain library | `crates/audiofiles-core/src/` | | |
| 94 | - | | Classification rules | `crates/audiofiles-core/src/analysis/classify.rs` | | |
| 94 | + | | Tag rules (Layer A) | `crates/audiofiles-core/src/rules.rs` | | |
| 95 | 95 | | Benchmarks + corpus builder | `crates/audiofiles-bench/`, `scripts/corpus.py` | | |
| 96 | 96 | | UI components | `crates/audiofiles-browser/src/` | | |
| 97 | 97 | | Desktop app shell | `crates/audiofiles-app/src/` | |
| @@ -10,7 +10,7 @@ | |||
| 10 | 10 | ||
| 11 | 11 | ### audiofiles-core | |
| 12 | 12 | ||
| 13 | - | Pure data layer with no UI or async dependencies. Owns the SQLite database schema (versioned migrations), content-addressed `SampleStore`, VFS tree operations, tag management, search/filter, smart folders, audio analysis pipeline (BPM, key, loudness, spectral, classification, loop detection), fingerprinting for duplicate detection, similarity search, instrument zone definitions, and export logic including device profile support. All operations are synchronous. This crate is the single source of truth for business logic. | |
| 13 | + | Pure data layer with no UI or async dependencies. Owns the SQLite database schema (versioned migrations), content-addressed `SampleStore`, VFS tree operations, tag management, search/filter, smart folders, audio analysis pipeline (BPM, key, loudness, spectral features, loop detection), fingerprinting for duplicate detection, similarity search, instrument zone definitions, and export logic including device profile support. All operations are synchronous. This crate is the single source of truth for business logic. | |
| 14 | 14 | ||
| 15 | 15 | ### audiofiles-browser | |
| 16 | 16 | ||
| @@ -56,7 +56,7 @@ | |||
| 56 | 56 | | Table | Purpose | | |
| 57 | 57 | |-------|---------| | |
| 58 | 58 | | `samples` | Content-addressed sample metadata (hash PK, original name, extension, size, timestamps) | | |
| 59 | - | | `audio_analysis` | Per-sample analysis results (BPM, key, duration, sample rate, channels, loudness, spectral features, classification) | | |
| 59 | + | | `audio_analysis` | Per-sample analysis results (BPM, key, duration, sample rate, channels, loudness, spectral features) | | |
| 60 | 60 | | `vfs` | Virtual filesystem roots (name, timestamps, sync_files flag) | | |
| 61 | 61 | | `vfs_nodes` | Tree nodes (directory or sample link, parent reference, unique name per parent) | | |
| 62 | 62 | | `tags` | Flat tag strings per sample hash (simplified from original name/value model) | | |
| @@ -81,10 +81,10 @@ | |||
| 81 | 81 | - **BPM**: Onset-based tempo estimation using spectral flux and autocorrelation. | |
| 82 | 82 | - **Key**: Musical key detection from chroma features. | |
| 83 | 83 | - **MFCC**: Mel-Frequency Cepstral Coefficients computed from existing STFT magnitudes (26-band mel filterbank, log energy, DCT-II, 13 coefficients). Aggregated as mean + variance across frames (26 features total). | |
| 84 | - | - **Classification**: Deterministic DSP only. A priority-ordered threshold tree (`analysis/classify.rs`) maps the 35-feature vector (9 spectral/waveform + 26 MFCC) to a `SampleClass`; first rule to match wins. It reports no confidence, being a threshold tree, and is measured at 23.6% strict accuracy, so it is being retired rather than tuned. The 35-feature vector is persisted to `sample_features` and feeds the layered tag pipeline (rules, exemplar k-NN, optional trained head, `.afcl` layers). Nothing model-derived ships in the binary today; the accepted plan is to bundle an official `.afcl` layer built from a CC-BY 4.0 corpus, with attribution carried in the manifest. See `ml_classifier.md`. Accuracy is measured out-of-tree by `audiofiles-bench` against a labeled corpus. | |
| 84 | + | - **Classification**: Deterministic DSP only, and multi-label. `analysis/features.rs` assembles the 35-feature vector (9 spectral/waveform + 26 MFCC), persists it to `sample_features`, and the layered tag pipeline reads it (rules, exemplar k-NN, optional trained head, `.afcl` layers). The single-label `SampleClass` and the threshold tree behind it were removed: 33.4% strict accuracy with two classes unreachable, and the features carry family structure rather than instrument identity. Nothing model-derived ships in the binary today; the accepted plan is to bundle an official `.afcl` layer built from a CC-BY 4.0 corpus, with attribution carried in the manifest. See `ml_classifier.md`. | |
| 85 | 85 | - **Loop detection**: Identifies whether a sample is a seamless loop. | |
| 86 | 86 | - **Fingerprinting**: Computes an amplitude envelope fingerprint for near-duplicate detection across the library. | |
| 87 | - | - **Tag suggestion**: Generates tag suggestions from analysis results (classification, BPM range, key, duration bracket) with confidence scores and human-readable reasons. | |
| 87 | + | - **Tag suggestion**: Generates tag suggestions from analysis results (BPM range, key, duration bracket, loudness) with confidence scores and human-readable reasons. | |
| 88 | 88 | ||
| 89 | 89 | Analysis runs in a background worker thread using rayon for parallel processing. A configurable analysis cap (`max_analysis_seconds`, default 30s) limits expensive operations (STFT, BPM/key) to the first N seconds of audio while cheap operations (peak/RMS, fingerprint) use the full signal. An `AtomicBool` cancel flag allows interrupting in-flight parallel work. | |
| 90 | 90 | ||
| @@ -93,7 +93,7 @@ | |||
| 93 | 93 | Export converts VFS subtrees into standalone file hierarchies on disk. The pipeline: | |
| 94 | 94 | ||
| 95 | 95 | 1. **Collect items**: Walk the VFS subtree, resolving each sample link to its content-addressed blob path and enriching with tags. | |
| 96 | - | 2. **Configure**: User selects format (original, WAV, AIFF), sample rate, bit depth, channel configuration, structure (preserve tree or flatten), naming pattern (with tokens like `{name}`, `{bpm}`, `{key}`, `{class}`), metadata sidecar, and destination directory. | |
| 96 | + | 2. **Configure**: User selects format (original, WAV, AIFF), sample rate, bit depth, channel configuration, structure (preserve tree or flatten), naming pattern (with tokens like `{name}`, `{bpm}`, `{key}`, `{duration}`), metadata sidecar, and destination directory. | |
| 97 | 97 | 3. **Device profiles**: Optionally select a hardware sampler profile (from the Rhai plugin registry) which pre-fills format constraints and may run custom hook scripts during export. | |
| 98 | 98 | 4. **Execute**: Background worker copies or transcodes each file, applying format conversion (via hound + rubato for resampling), channel conversion (mono/stereo), and naming rules. Progress is reported per file. Each output is written atomically via a `write_atomic(dest, |tmp| ...)` helper: the encoder/copier targets `dest.audiofiles_tmp`, then `fs::rename`s into place on success. A killed export never leaves a partial file in the user's export directory. | |
| 99 | 99 | ||
| @@ -114,8 +114,6 @@ | |||
| 114 | 114 | ||
| 115 | 115 | Which theme is active is two questions. `theme::chosen()` is the user's standing selection, `Follow` or `Fixed(id)`; `theme::active_id()` is what it resolves to against the desktop's current light/dark appearance. Only the first is stored, under `ConfigKey::Theme`, as `"system"` or an id. | |
| 116 | 116 | ||
| 117 | - | Sample classification colors are hardcoded (not theme-driven) so that semantic categories maintain consistent visual identity across themes. | |
| 118 | - | ||
| 119 | 117 | ## Audio Thread Model | |
| 120 | 118 | ||
| 121 | 119 | The cpal audio output callback runs on a real-time thread where blocking should be avoided. The design uses `parking_lot::Mutex` with `try_lock`: | |
| @@ -143,7 +141,7 @@ | |||
| 143 | 141 | 2. **Strategy selection**: User chooses flat (all links in current directory), new VFS (preserve directory structure), or merge into existing VFS. | |
| 144 | 142 | 3. **Import loop**: For each file: hash, copy to store, create VFS node. Duplicates (name conflicts) are counted but not errors. Progress and errors are reported back to the GUI via channel events. | |
| 145 | 143 | 4. **Folder tagging**: After import completes, the user can assign comma-separated tags to each imported top-level folder, applied to all samples within. | |
| 146 | - | 5. **Analysis**: Optionally run configurable analysis (loudness, BPM, key, spectral, classification, loop detection, fingerprint, auto-suggest tags) on imported samples. | |
| 144 | + | 5. **Analysis**: Optionally run configurable analysis (loudness, BPM, key, spectral features, loop detection, fingerprint, auto-suggest tags) on imported samples. | |
| 147 | 145 | 6. **Tag review**: If auto-suggest was enabled, the user reviews suggested tags with accept/reject per suggestion before committing. | |
| 148 | 146 | ||
| 149 | 147 | Cancellation is checked between files, keeping the UI responsive during large imports. |
| @@ -60,14 +60,17 @@ | |||
| 60 | 60 | | `spectral_flatness` | REAL | Migration 003, nullable | | |
| 61 | 61 | | `spectral_rolloff` | REAL | Migration 003, nullable | | |
| 62 | 62 | | `zero_crossing_rate` | REAL | Migration 003, nullable | | |
| 63 | - | | `classification` | TEXT | ML category label (migration 003), nullable | | |
| 64 | 63 | | `spectral_bandwidth` | REAL | Migration 010, nullable | | |
| 65 | 64 | | `centroid_variance` | REAL | Migration 010, nullable | | |
| 66 | 65 | | `crest_factor` | REAL | Migration 010, nullable | | |
| 67 | 66 | | `attack_time` | REAL | Migration 010, nullable | | |
| 68 | - | | `classification_confidence` | REAL | ML confidence score (migration 011), nullable | | |
| 69 | 67 | ||
| 70 | - | Indexes: `bpm`, `musical_key`, `duration`, `classification`. | |
| 68 | + | Indexes: `bpm`, `musical_key`, `duration`. | |
| 69 | + | ||
| 70 | + | `classification` (M003) and `classification_confidence` (M011) held the single | |
| 71 | + | sample-class label. Both are gone, along with the classifier that produced them; see | |
| 72 | + | `ml_classifier.md`. The columns were removed from the migration bodies that added them | |
| 73 | + | rather than dropped in a later migration, so no database creates them. | |
| 71 | 74 | ||
| 72 | 75 | ### sample_features | |
| 73 | 76 | The 35-element feature vector (9 scalar + 26 MFCC) per sample. Migration 020. Foundation | |
| @@ -288,7 +291,7 @@ | |||
| 288 | 291 | ## Key Indexes | |
| 289 | 292 | ||
| 290 | 293 | - **Content lookup** on samples (original_name), tags (sample_hash, tag) | |
| 291 | - | - **Analysis** on audio_analysis (bpm, musical_key, duration, classification) | |
| 294 | + | - **Analysis** on audio_analysis (bpm, musical_key, duration) | |
| 292 | 295 | - **VFS tree** on vfs_nodes (parent_id, vfs_id, sample_hash) | |
| 293 | 296 | - **History** on edit_history (source_hash, result_hash) | |
| 294 | 297 | - **Sync** on sync_changelog (pushed) for pending push detection |
| @@ -50,7 +50,7 @@ | |||
| 50 | 50 | ### Classification (16 Categories) | |
| 51 | 51 | Kick, Snare, HiHat, Cymbal, Percussion, Bass, Vocal, Synth, Pad, FX, Noise, Music, Ambience, Impact, Foley, Texture | |
| 52 | 52 | ||
| 53 | - | Classification runs on deterministic signal measurement: a priority-ordered rule tree over 35 extracted features (spectral shape, crest factor, attack time, MFCCs). No trained model ships in the app, so nothing about your library leaves it and the classifier carries no training-data licence baggage. | |
| 53 | + | Classification runs on deterministic signal measurement: 35 extracted features per sample (spectral shape, crest factor, attack time, MFCCs), then rules you write and k-NN over the samples you have already tagged. No trained model ships in the app, so nothing about your library leaves it and the classifier carries no training-data licence baggage. | |
| 54 | 54 | ||
| 55 | 55 | ### Tag System | |
| 56 | 56 | - Hierarchical dot-notation tags (e.g., `genre.electronic.house`, `instrument.drum.kick`) | |
| @@ -73,7 +73,7 @@ | |||
| 73 | 73 | ||
| 74 | 74 | ### Search + Filtering | |
| 75 | 75 | - Text search by sample name (folder or global scope, 500 result limit) | |
| 76 | - | - Filter panel: BPM range, duration range, key selector (with compatibility mode), classification checkboxes, tag prefix filter | |
| 76 | + | - Filter panel: BPM range, duration range, loudness range, key selector (with compatibility mode), tag prefix filter | |
| 77 | 77 | - Filters combine with AND logic | |
| 78 | 78 | - `/` to focus search bar | |
| 79 | 79 | - Similarity search: weighted Euclidean distance on analysis vectors ("Find Similar" in context menu) | |
| @@ -142,13 +142,13 @@ | |||
| 142 | 142 | ### Bulk Operations | |
| 143 | 143 | - Bulk delete, move, rename, tag (add/remove) across multi-selection | |
| 144 | 144 | - 50-deep undo stack (Cmd+Z) | |
| 145 | - | - Rename pattern engine: {name}, {ext}, {bpm}, {key}, {class}, {duration}, {n}/{nn}/{nnn} tokens | |
| 145 | + | - Rename pattern engine: {name}, {ext}, {bpm}, {key}, {duration}, {n}/{nn}/{nnn} tokens | |
| 146 | 146 | - Live preview of rename results before applying | |
| 147 | 147 | - Bulk modal UIs: tag editor (Cmd+T), directory picker, pattern builder (F2) | |
| 148 | 148 | ||
| 149 | 149 | ### Detail Panel | |
| 150 | 150 | - Waveform display with click-to-seek (rendered from pre-generated peak data) | |
| 151 | - | - Metadata: duration, sample rate, channels, peak dB, RMS dB, LUFS, BPM, key, classification | |
| 151 | + | - Metadata: duration, sample rate, channels, peak dB, RMS dB, LUFS, BPM, key | |
| 152 | 152 | - Tag editor (add/remove chips) | |
| 153 | 153 | - Auto-hides when window is narrow (< 700px) | |
| 154 | 154 | - Resizable (200-400px) | |
| @@ -233,7 +233,7 @@ | |||
| 233 | 233 | ||
| 234 | 234 | ### Testing | |
| 235 | 235 | ||
| 236 | - | Tests cover: database, store, VFS, tags, analysis (incl. ML classifier), search, export, instrument, smart folders, rename, fingerprint, similarity, state orchestration, theme parsing, preview decoding, updater, license activation, and full end-to-end pipelines. | |
| 236 | + | Tests cover: database, store, VFS, tags, analysis, search, export, instrument, smart folders, rename, fingerprint, similarity, state orchestration, theme parsing, preview decoding, updater, license activation, and full end-to-end pipelines. | |
| 237 | 237 | ||
| 238 | 238 | ### Status | |
| 239 | 239 |
| @@ -65,7 +65,6 @@ | |||
| 65 | 65 | | `hover_surface()` | alias | = `surface_sunken`. Shared row/button hover. | | |
| 66 | 66 | | `selection()` | derived | Selection fill = `lerp(page, action, 0.3)`. | | |
| 67 | 67 | | `trim_mute_overlay()` | derived | Edit panel's trimmed-region wash; dims toward the theme's own page colour. | | |
| 68 | - | | `classification_color(c)` | domain | Sample-class palette. Stable across themes for muscle memory. | | |
| 69 | 68 | | `piano_white_key()`, `piano_black_key()` | domain | Instrument panel only. | | |
| 70 | 69 | ||
| 71 | 70 | ### Theme selection | |
| @@ -210,7 +209,6 @@ | |||
| 210 | 209 | ||
| 211 | 210 | - **`tag_chip(ui, tag) -> Response`**: existing. | |
| 212 | 211 | - **`tag_chip_removable(ui, tag) -> bool`**: existing. | |
| 213 | - | - **`classification_badge(ui, class)`**: existing. | |
| 214 | 212 | - **`segmented_control(ui, current, options: &[(value, label, tooltip)]) -> Option<value>`**: mutually-exclusive choices: Folder/All, Exact/Compatible, Add/Remove tag, help tabs. Segments abut with no gap and each carries its own bevel, so the group reads as one object with one part pushed in. Was `toggle_pills`, which spaced them and rounded them and therefore read as two unrelated buttons. | |
| 215 | 213 | ||
| 216 | 214 | ### Inputs |
| @@ -5,13 +5,12 @@ | |||
| 5 | 5 | writes, and a k-NN layer whose exemplars are the samples the user has already tagged. | |
| 6 | 6 | There is no neural network and no training step at build time. | |
| 7 | 7 | ||
| 8 | - | Two things run per sample, and they are separate systems: | |
| 8 | + | One system does this, the tag pipeline: multi-label, provenance-tracked, per-library. | |
| 9 | 9 | ||
| 10 | - | - **The `SampleClass` column.** One coarse label per sample, produced today by a | |
| 11 | - | hardcoded threshold tree. It feeds the filter panel, theme colours, the `{class}` | |
| 12 | - | rename token, and export resolution. The tree is being retired (see below). | |
| 13 | - | - **The tag pipeline.** Multi-label, provenance-tracked, per-library. This is where the | |
| 14 | - | work is going. | |
| 10 | + | There used to be a second one alongside it, a single `SampleClass` label per sample from a | |
| 11 | + | hardcoded threshold tree, feeding the filter panel, theme colours, the `{class}` rename | |
| 12 | + | token, and export resolution. It is gone, tree and column both. Why, and what took over, | |
| 13 | + | is below. | |
| 15 | 14 | ||
| 16 | 15 | ## Per-sample analysis | |
| 17 | 16 | ||
| @@ -24,7 +23,6 @@ | |||
| 24 | 23 | bandwidth, centroid variance | |
| 25 | 24 | -> mfcc: 13 means + 13 variances (capped at 30s) | |
| 26 | 25 | -> 35-feature vector -> sample_features (synced) | |
| 27 | - | -> classify_full() -> SampleClass -> audio_analysis (no confidence) | |
| 28 | 26 | ``` | |
| 29 | 27 | ||
| 30 | 28 | The 35-feature vector is the durable asset. It is stored per sample in `sample_features` | |
| @@ -34,11 +32,11 @@ | |||
| 34 | 32 | ||
| 35 | 33 | ### Smart-skip | |
| 36 | 34 | ||
| 37 | - | The expensive BPM/key/loop stages are gated on cheap raw features rather than on the | |
| 38 | - | classification label: BPM and loop detection run only for clips long enough to carry | |
| 39 | - | tempo, and key detection only for clips that are long enough and not noise-like (high | |
| 40 | - | spectral flatness). The gate skips the clearly pointless cases and otherwise runs, so a | |
| 41 | - | misread class cannot cause a wrong skip. | |
| 35 | + | The expensive BPM/key/loop stages are gated on cheap raw features: BPM and loop detection | |
| 36 | + | run only for clips long enough to carry tempo, and key detection only for clips that are | |
| 37 | + | long enough and not noise-like (high spectral flatness). The gate skips the clearly | |
| 38 | + | pointless cases and otherwise runs. It was keyed off the sample class once, which could | |
| 39 | + | skip wrongly on a misread label; there is no label now, and nothing to misread. | |
| 42 | 40 | ||
| 43 | 41 | ## The tag pipeline | |
| 44 | 42 | ||
| @@ -56,30 +54,43 @@ | |||
| 56 | 54 | ||
| 57 | 55 | All five are wired into the app today. | |
| 58 | 56 | ||
| 59 | - | ## The threshold tree is being retired | |
| 57 | + | ## The threshold tree is gone, and so is the class column | |
| 60 | 58 | ||
| 61 | - | `classify_full()` in `crates/audiofiles-core/src/analysis/classify.rs` assigns a | |
| 62 | - | `SampleClass` from a priority-ordered tree of about 40 hand-set thresholds over cheap DSP | |
| 63 | - | features. Rules are evaluated in order and the first match wins. | |
| 59 | + | `classify_full()` assigned a `SampleClass` from a priority-ordered tree of about 40 | |
| 60 | + | hand-set thresholds, first match wins. Measured against 1,049 labelled drum samples it was | |
| 61 | + | 33.4% accurate on exact class match, and two of the seven drum classes, `Clap` and `Tom`, | |
| 62 | + | could not be emitted by any rule, so a sample of either was wrong every time. | |
| 64 | 63 | ||
| 65 | - | Measured against 1,049 labelled drum samples it is 23.6% accurate on exact class match. | |
| 66 | - | Two of the seven drum classes, `Clap` and `Tom`, cannot be emitted by any rule, so a | |
| 67 | - | sample of either is wrong every time. Coarse questions do far better than fine ones on | |
| 68 | - | these features: a single feature and a single threshold separate low drums from bright | |
| 69 | - | drums at 92.4%. Instrument identity is not in the features; family is. | |
| 64 | + | Tuning it could not have worked. Roughly a quarter of the corpus was unreachable by | |
| 65 | + | construction, so a sweep over 40 parameters would have fitted noise against a target it | |
| 66 | + | could not hit. The measurements are in the wiki note `af-browse-axes`; the short version is | |
| 67 | + | that instrument identity is not present in these features and coarse family structure is. | |
| 68 | + | One unfitted threshold on the spectral centroid separates low drums from bright drums at | |
| 69 | + | 92.4%, against 33.4% for seven-way identity with 40 tuned constants. Users also mostly do | |
| 70 | + | not ask for the specific name: across 9,493 Freesound sounds, 60.5% carry a family word | |
| 71 | + | and 13.5% a specific instrument. | |
| 70 | 72 | ||
| 71 | - | So the tree is being replaced rather than tuned. Tuning about 40 parameters against a | |
| 72 | - | target a quarter of the corpus cannot reach would fit noise. Retiring the tree does not | |
| 73 | - | retire `SampleClass`: the taxonomy is used in 239 places across 41 files and stays. What | |
| 74 | - | goes is the constants in one function, and the open question is what feeds the column in | |
| 75 | - | its place. | |
| 73 | + | So the label was not replaced, it was dropped, along with `SampleClass` and the | |
| 74 | + | `audio_analysis.classification` / `.classification_confidence` columns. What took over: | |
| 76 | 75 | ||
| 77 | - | One consequence is already in effect. `classify_ml()` reports no confidence at all | |
| 78 | - | (`Option::None`), because a threshold tree has no probability behind it. Tag suggestion | |
| 79 | - | used to read that absence as a 0.0 and substitute a fixed 0.7, which presented a | |
| 80 | - | 23.6%-accurate guess as a moderately confident one. It now suggests a classification tag | |
| 81 | - | only when a real confidence is reported and clears 0.5, so the rule tree proposes nothing. | |
| 82 | - | The class still shows in the UI. | |
| 76 | + | - **Browsing** by continuous measured axes (register, length, tonal vs noisy, attack). | |
| 77 | + | Nothing to misclassify, no thresholds to tune. | |
| 78 | + | - **Instrument names** from filename rules (Layer A over `RuleField::Name`) plus the | |
| 79 | + | user's own tags. Our own ground truth came from filenames: `corpus.py` labelled 1,049 of | |
| 80 | + | 1,074 one-shots that way, 97.7%, on the same files the tree scored 33.4% on. Real packs | |
| 81 | + | name their files `Kick.wav`. | |
| 82 | + | - **"More like this"** by the existing k-NN over the 35-feature vector. | |
| 83 | + | ||
| 84 | + | What went with the column: the `{class}` rename token (a saved pattern containing it now | |
| 85 | + | fails to parse rather than resolving to an empty string), the `classification` key in the | |
| 86 | + | export sidecar, the class filter checkboxes, the class column and its sort, the per-class | |
| 87 | + | badge and colour table, and the class-keyed tag-suggestion table with its dismissal store. | |
| 88 | + | Schema side: the columns came out of the migration bodies that added them (M003, M011), | |
| 89 | + | along with the index (M004, M028) and the changelog-trigger payload keys, so no database | |
| 90 | + | creates them and none needs to drop them. There is no data migration because there is no | |
| 91 | + | deployed data: a pre-existing dev vault keeps two dead columns until it is rebuilt, and a | |
| 92 | + | saved search that filtered on the class silently loses that criterion (`SearchFilter` is | |
| 93 | + | `#[serde(default)]`, so the stored key deserializes into nothing). | |
| 83 | 94 | ||
| 84 | 95 | ## What ships in the binary, and under what licence | |
| 85 | 96 | ||
| @@ -140,7 +151,6 @@ | |||
| 140 | 151 | ||
| 141 | 152 | | Table | Columns | Description | | |
| 142 | 153 | |-------|---------|-------------| | |
| 143 | - | | audio_analysis | classification, classification_confidence | `SampleClass` string; confidence is NULL for the rule-based path, which reports none | | |
| 144 | 154 | | sample_features | hash, feat_version, vector, computed_at | The persisted 35-feature vector (JSON array) | | |
| 145 | 155 | | classifier_exemplars | id, layer_id, feat_version, vector, tags | Exemplars from an imported `.afcl`, no sample row | | |
| 146 | 156 | | trained_head | feat_version, exemplar_count, model, trained_at | Single-row local cache of the distilled model, not synced | | |
| @@ -149,7 +159,7 @@ | |||
| 149 | 159 | ||
| 150 | 160 | | What | Where | | |
| 151 | 161 | |------|-------| | |
| 152 | - | | Threshold tree + `SampleClass` taxonomy | `crates/audiofiles-core/src/analysis/classify.rs` | | |
| 162 | + | | Feature vector assembly + `FEATURE_VERSION` | `crates/audiofiles-core/src/analysis/features.rs` | | |
| 153 | 163 | | Tag rules (Layer A) | `crates/audiofiles-core/src/rules.rs` | | |
| 154 | 164 | | Exemplar k-NN (Layer B) | `crates/audiofiles-core/src/analysis/exemplar.rs` | | |
| 155 | 165 | | Trained head | `crates/audiofiles-core/src/analysis/trained_head.rs` | |
| @@ -18,7 +18,7 @@ | |||
| 18 | 18 | ||
| 19 | 19 | - Your audio sample files, stored by content hash in a content-addressed store | |
| 20 | 20 | - A local database of sample metadata: original filenames, tags, collections, and your virtual folder structure | |
| 21 | - | - Analysis results computed on your device: BPM, musical key, loudness, spectral features, and instrument classification | |
| 21 | + | - Analysis results computed on your device: BPM, musical key, loudness, spectral features, and the tags you apply | |
| 22 | 22 | - Waveform data and an edit history for undo/redo | |
| 23 | 23 | ||
| 24 | 24 | Separately, a small configuration folder holds your preferences, your cached license, and a random per-install identifier (see "License Activation" below). | |
| @@ -54,7 +54,7 @@ | |||
| 54 | 54 | ||
| 55 | 55 | ## Audio Analysis and Classification | |
| 56 | 56 | ||
| 57 | - | All audio analysis (loudness, BPM, key, spectral features, and the instrument classifier) runs entirely on your device using a model bundled with the app. No audio is uploaded for analysis or classification. Your audio leaves the device only through cloud sync if you enable it, and only as encrypted blobs. | |
| 57 | + | All audio analysis (loudness, BPM, key, spectral features, and tag suggestion) runs entirely on your device. No audio is uploaded for analysis or classification. Your audio leaves the device only through cloud sync if you enable it, and only as encrypted blobs. | |
| 58 | 58 | ||
| 59 | 59 | ## Plugins | |
| 60 | 60 |
| @@ -1,8 +1,9 @@ | |||
| 1 | 1 | #!/usr/bin/env python3 | |
| 2 | 2 | """Fetch and lay out the benchmark sample corpus. | |
| 3 | 3 | ||
| 4 | - | audiofiles-bench scores classification against folder labels, so it needs a | |
| 5 | - | corpus laid out as samples/training/<class>/ and samples/test-suite/<kind>/. | |
| 4 | + | audiofiles-bench times the analysis pipeline over a corpus laid out as | |
| 5 | + | samples/training/<class>/ and samples/test-suite/<kind>/. The per-class folders | |
| 6 | + | are also the filename-derived ground truth the browse-axes measurements used. | |
| 6 | 7 | That tree is gitignored and has to be rebuilt per machine. This script fetches | |
| 7 | 8 | the source datasets and maps them into that layout. | |
| 8 | 9 | ||
| @@ -135,9 +136,9 @@ | |||
| 135 | 136 | Files matching more than one class are dropped rather than resolved by rule | |
| 136 | 137 | order. These packs contain genuinely ambiguous names -- "Kick_Cowbell.wav", | |
| 137 | 138 | "Tom-Cymbal.wav" -- and first-match-wins would silently assign one of the | |
| 138 | - | two at random. A wrong label is worse than a missing one here, because the | |
| 139 | - | bench reports accuracy against these folders and a mislabeled file looks | |
| 140 | - | like a classifier error forever. | |
| 139 | + | two at random. A wrong label is worse than a missing one here: these folders | |
| 140 | + | are the ground truth the browse-axes measurements are read against, so a | |
| 141 | + | mislabeled file looks like a real result forever. | |
| 141 | 142 | """ | |
| 142 | 143 | hay = " ".join([path.name, path.parent.name, path.parent.parent.name]).lower() | |
| 143 | 144 | ||
| @@ -234,8 +235,8 @@ | |||
| 234 | 235 | shutil.copy2(src, dest) | |
| 235 | 236 | counts[key] += 1 | |
| 236 | 237 | ||
| 237 | - | # Written out rather than just counted: when accuracy looks off, the first | |
| 238 | - | # question is always whether the corpus or the classifier is wrong. | |
| 238 | + | # Written out rather than just counted: when a measurement looks off, the | |
| 239 | + | # first question is always whether the corpus or the code is wrong. | |
| 239 | 240 | (samples / "DROPPED.txt").write_text("\n".join(sorted(dropped)) + "\n") | |
| 240 | 241 | ||
| 241 | 242 | return counts |