Skip to main content

max / audiofiles

Retire trained classifier, persist feature vectors (classifier Phase 0) Remove the copyright surface and lay the foundation for the hybrid rules + k-NN tag pipeline. - Delete the four embedded layer2_*.json Random Forest models and the audiofiles-train crate; strip the RF inference path (TreeNode, RandomForestModel, classify_broad/BroadClass, predict_with_model). classify_ml now delegates to the rule-based classify_full, kept as an interim bridge until the rules engine lands. - Migration 020: sample_features(hash, feat_version, vector JSON, computed_at) + sync triggers; register in audiofiles-sync. analyze_sample assembles and persists the sanitized 35-feature vector; FEATURE_VERSION stamps the extraction layout. - Re-key smart_skip off cheap raw features (duration + flatness) via should_detect_bpm/should_detect_key, replacing the class-based gate so it survives the classifier's removal. - Update docs/ml_classifier.md and docs/database_schema.md.
Co-Authored-By
Claude Opus 4.8 <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-06-14 22:35 UTC
Signed with PGP, not checked
Commit: 6abab38e5245b673d98340f45c19102ff986a472
Parent: 8bce2db
19 files changed, +367 insertions, -1318 deletions
M Cargo.lock -11
@@ -498,17 +498,6 @@
498 498 "uuid",
499 499 ]
500 500
501 - [[package]]
502 - name = "audiofiles-train"
503 - version = "0.5.0"
504 - dependencies = [
505 - "audiofiles-core",
506 - "rand",
507 - "rayon",
508 - "serde",
509 - "serde_json",
510 - ]
511 -
512 501 [[package]]
513 502 name = "autocfg"
514 503 version = "1.5.1"
M Cargo.toml +1 -1
@@ -1,5 +1,5 @@
1 1 [workspace]
2 - members = ["crates/audiofiles-core", "crates/audiofiles-browser", "crates/audiofiles-app", "crates/audiofiles-sync", "crates/audiofiles-rhai", "crates/audiofiles-train", "crates/audiofiles-bench"]
2 + members = ["crates/audiofiles-core", "crates/audiofiles-browser", "crates/audiofiles-app", "crates/audiofiles-sync", "crates/audiofiles-rhai", "crates/audiofiles-bench"]
3 3 default-members = ["crates/audiofiles-core", "crates/audiofiles-browser", "crates/audiofiles-app", "crates/audiofiles-sync", "crates/audiofiles-rhai"]
4 4 resolver = "2"
5 5
M docs/database_schema.md +17 -2
@@ -1,6 +1,6 @@
1 1 # audiofiles Database Schema
2 2
3 - SQLite schema reference. 19 inline migrations. Migrations are embedded as Rust string constants in `crates/audiofiles-core/src/db.rs` and applied via `PRAGMA user_version` tracking -- not separate SQL files.
3 + SQLite schema reference. 20 inline migrations. Migrations are embedded as Rust string constants in `crates/audiofiles-core/src/db.rs` and applied via `PRAGMA user_version` tracking -- not separate SQL files.
4 4
5 5 ## Domain Map
6 6
@@ -69,6 +69,19 @@
69 69
70 70 Indexes: `bpm`, `musical_key`, `duration`, `classification`.
71 71
72 + ### sample_features
73 + The 35-element feature vector (9 scalar + 26 MFCC) per sample. Migration 020. Foundation
74 + for the rules + k-NN tag pipeline; synced across a user's own devices. One row per sample.
75 +
76 + | Column | Type | Notes |
77 + |--------|------|-------|
78 + | `hash` | TEXT PK FK | -> samples (CASCADE) |
79 + | `feat_version` | INTEGER | Feature-extraction layout version (`FEATURE_VERSION`); stale rows are recomputed, not mixed |
80 + | `vector` | TEXT | JSON array of 35 f64 (non-finite components sanitized to 0.0) |
81 + | `computed_at` | INTEGER | Unix timestamp |
82 +
83 + Index: `feat_version`.
84 +
72 85 ### waveform_data
73 86 Pre-computed waveform visualizations. Migration 004.
74 87
@@ -230,7 +243,7 @@
230 243 - **Sync-excluded keys:** `user_config` sync triggers skip keys matching `sync_%` to avoid syncing sync-internal state
231 244 - **Cloud-only samples:** `samples.cloud_only` flag allows local blob eviction while keeping metadata and cloud copy
232 245 - **Hashed row IDs (M018):** sensitive `sync_changelog.row_id` values go through `hash_row_id(row_id_salt, canonical_key)` so the server never sees raw sample hashes or tag strings. DELETE triggers also emit the canonical PK into the encrypted `data` field so pull-side replay doesn't need to parse row_id.
233 - - **Synced tables:** `samples`, `audio_analysis`, `vfs`, `vfs_nodes`, `tags`, `collections`, `collection_members`, `user_config`, `edit_history`
246 + - **Synced tables:** `samples`, `sample_features`, `audio_analysis`, `vfs`, `vfs_nodes`, `tags`, `collections`, `collection_members`, `user_config`, `edit_history`
234 247
235 248 ## Key Indexes
236 249
@@ -262,3 +275,5 @@
262 275 | 016 | Exclude `loose_files` user_config key from sync (security: server can't flip the mode) |
263 276 | 017 | Rename `unsafe_mode` user_config key to `loose_files` (trigger recreation only; row-copy lives in main.rs) |
264 277 | 018 | Hash `sync_changelog.row_id` for sensitive tables; DELETE triggers emit canonical PK in `data`; per-user `row_id_salt` in sync_state |
278 + | 019 | `samples.deleted_at` soft-delete tombstone + partial index; seed `sample_tombstone_retain_days`; re-emit samples sync triggers |
279 + | 020 | `sample_features` table (persisted 35-feature vector) + sync triggers; retired the embedded RF classifier models |
M docs/ml_classifier.md +59 -136
@@ -1,172 +1,95 @@
1 - # audiofiles -- ML Classification System
1 + # audiofiles -- Sample Classification
2 2
3 - Two-layer system that classifies audio samples into 16 categories. Layer 1 uses rule-based heuristics for broad classification. Layer 2 uses a 200-tree Random Forest for fine-grained drum sub-classification.
3 + audiofiles classifies samples from deterministic DSP features only. No trained model
4 + ships in the binary: the feature extraction is plain signal measurement, and tagging is
5 + driven by the user's own library. This keeps the classifier free of any training-data
6 + copyright surface.
4 7
5 - ## Architecture
8 + The system is being built out in phases toward a hybrid tag pipeline — a deterministic
9 + rules layer plus an exemplar k-NN layer learned from the user's library, feeding one
10 + provenance-tracked tag resolution step. The sections below describe what exists today.
11 +
12 + ## What exists today
6 13
7 14 ```
8 15 Audio file
9 - ↓ decode (Symphonia → mono f32)
10 - ↓ feature extraction (9 spectral + 26 MFCC = 35 features)
11 - ↓
12 - Layer 1: classify_broad() ← rule-based heuristics
13 - ├─ Drum → Layer 2: predict_layer2() ← Random Forest (200 trees)
14 - │ └─ Kick / Snare / HiHat / Cymbal / Percussion
15 - └─ Non-drum → return directly
16 - └─ Bass / Vocal / Synth / Pad / Noise / Music / Ambience / Impact / Foley / Texture / Misc
16 + -> decode (Symphonia -> mono f32)
17 + -> feature extraction (9 spectral/basic + 26 MFCC = 35 features)
18 + -> classify_full() rule-based threshold tree -> SampleClass
19 + -> persist 35-feature vector (sample_features) + classification (audio_analysis)
17 20 ```
18 21
19 - ### Layer 1: Rule-Based Broad Classifier
22 + ### Rule-based classification (interim)
20 23
21 - `classify_broad()` in `crates/audiofiles-core/src/analysis/classify.rs`
24 + `classify_full()` in `crates/audiofiles-core/src/analysis/classify.rs` assigns a
25 + `SampleClass` from a priority-ordered threshold tree over cheap DSP features (duration,
26 + spectral centroid/flatness/rolloff/ZCR/bandwidth, crest factor, attack time). Rules are
27 + evaluated in order; the first match wins. This is an interim bridge: it is scheduled to be
28 + superseded by the user-editable rules layer, at which point the hardcoded thresholds retire.
22 29
23 - Routes samples into broad categories using spectral and waveform features:
30 + The trained Random Forest models that previously refined drum/bass/vocal/synth
31 + sub-classes were removed. The captured thresholds and the full taxonomy are archived in
32 + `_private/docs/audiofiles/design-user-classifier.md` for reference.
24 33
25 - | Rule | Condition | Category |
26 - |------|-----------|----------|
27 - | Noise | flatness > 0.7 | Noise |
28 - | Drum | duration < 2.0 AND (attack < 0.05 OR crest > 2.5) | Drum → Layer 2 |
29 - | Bass | centroid < 400 AND flatness < 0.15 | Bass |
30 - | Ambience | duration > 5.0 AND low centroid_variance AND 0.15 < flatness < 0.5 | Ambience |
31 - | Impact | crest > 10.0 AND attack < 0.005 | Impact |
32 - | Texture | duration > 2.0 AND centroid_variance > 500,000 | Texture |
34 + ### Persisted feature vector
33 35
34 - Rules are evaluated in priority order. Confidence values range 0.75--0.95 depending on how strongly the sample matches.
36 + Every analyzed sample's 35-feature vector is stored in the `sample_features` table
37 + (`hash`, `feat_version`, `vector` as a JSON array, `computed_at`). It is the foundation
38 + for the forthcoming rules + k-NN tag pipeline, syncs across a user's own devices, and is
39 + stamped with `FEATURE_VERSION` so a change to the extraction layout invalidates stale
40 + vectors rather than silently mixing incompatible feature spaces.
35 41
36 - ### Layer 2: Random Forest Drum Classifier
42 + ### Smart-skip
37 43
38 - `predict_layer2()` in `crates/audiofiles-core/src/analysis/classify.rs`
44 + The expensive BPM/key/loop stages are gated on cheap raw features rather than the
45 + classification label: BPM/loop run only for clips long enough to carry tempo; key runs
46 + only for clips that are long enough and not noise-like (high spectral flatness). The gate
47 + skips only the clearly-pointless cases and otherwise runs, so it never wrongly skips on a
48 + misread class.
39 49
40 - - **Model**: 200 decision trees, majority vote
41 - - **Classes**: Kick (0), Snare (1), HiHat (2), Cymbal (3), Percussion (4)
42 - - **Confidence**: fraction of trees voting for the majority class (e.g., 0.85 = 170/200 agreed)
43 - - **Fallback**: if the model file has empty trees, reverts to `classify_full()` (16-class rule-based)
50 + ## Feature vector
44 51
45 - ### Graceful Degradation
52 + 35 features: 9 scalar + 13 MFCC means + 13 MFCC variances.
46 53
47 - If `layer2_drum.json` contains an empty trees vector, the system falls back to `classify_full()` -- a comprehensive 16-class rule-based classifier covering all categories. The app never crashes on classification.
48 -
49 - ---
50 -
51 - ## Feature Vector
52 -
53 - 35 features total: 9 scalar + 13 MFCC means + 13 MFCC variances.
54 -
55 - ### Scalar Features (indices 0--8)
54 + ### Scalar features (indices 0--8)
56 55
57 56 | Index | Feature | Source | Description |
58 57 |-------|---------|--------|-------------|
59 58 | 0 | duration | basic.rs | Total length in seconds |
60 59 | 1 | centroid | spectral.rs | Spectral center of mass in Hz |
61 - | 2 | flatness | spectral.rs | 0.0 (pure tone) to 1.0 (white noise), geometric/arithmetic mean of magnitudes |
62 - | 3 | zcr | spectral.rs | Zero-crossing rate (fraction of sign changes per sample) |
63 - | 4 | onset_strength | spectral.rs | Sum of positive spectral flux across STFT frames |
60 + | 2 | flatness | spectral.rs | 0.0 (pure tone) to 1.0 (white noise) |
61 + | 3 | zcr | spectral.rs | Zero-crossing rate |
62 + | 4 | onset_strength | spectral.rs | Sum of positive spectral flux across frames |
64 63 | 5 | bandwidth | spectral.rs | Spectral standard deviation around centroid in Hz |
65 - | 6 | centroid_variance | spectral.rs | Variance of per-frame centroids (high = evolving spectrum) |
66 - | 7 | crest_factor | basic.rs | Peak / RMS in linear domain (high > 8 = impacts) |
67 - | 8 | attack_time | basic.rs | Time to reach 90% of peak amplitude in seconds |
64 + | 6 | centroid_variance | spectral.rs | Variance of per-frame centroids (spectral evolution) |
65 + | 7 | crest_factor | basic.rs | Peak / RMS in linear domain |
66 + | 8 | attack_time | basic.rs | Time to 90% of peak amplitude in seconds |
68 67
69 - ### MFCC Features (indices 9--34)
68 + ### MFCC features (indices 9--34)
70 69
71 70 | Indices | Feature | Description |
72 71 |---------|---------|-------------|
73 - | 9--21 | MFCC means | Mean of first 13 MFCCs across all STFT frames |
74 - | 22--34 | MFCC variances | Variance of first 13 MFCCs across all STFT frames |
72 + | 9--21 | MFCC means | Mean of the first 13 MFCCs across STFT frames |
73 + | 22--34 | MFCC variances | Variance of the first 13 MFCCs across STFT frames |
75 74
76 - MFCC computation: 26-bin mel filterbank applied to STFT magnitude frames, log energy transform, DCT-II, keep first 13 coefficients.
75 + MFCCs: mel filterbank over STFT magnitude frames, log-energy, DCT-II, first 13 coefficients.
76 + Non-finite components are sanitized to 0.0 before persistence.
77 77
78 - ### STFT Parameters
78 + ## Database integration
79 79
80 - - FFT size: 2048 points with Hann window
81 - - Hop size: 512 samples
80 + | Table | Columns | Description |
81 + |-------|---------|-------------|
82 + | audio_analysis | classification, classification_confidence | `SampleClass` string; confidence 0.0 for the rule-based path |
83 + | sample_features | hash, feat_version, vector, computed_at | The persisted 35-feature vector (JSON array) |
82 84
83 - ---
84 -
85 - ## Training Pipeline
86 -
87 - Binary: `crates/audiofiles-train/src/main.rs` (not built by default).
88 -
89 - ### Data
90 -
91 - - Source: `~/Git/Drums/test_data/` with subdirectories per class
92 - - Classes: `kick/`, `snare/`, `hihat/`, `cymbal/`, `clap/`, `tom/`, `percussion/`
93 - - Class mapping: kick→0, snare→1, hihat→2, cymbal→3, clap/tom/percussion→4
94 - - Dataset: 4,343 labeled drum samples
95 -
96 - ### Algorithm
97 -
98 - - **200 decision trees**, each trained on a bootstrap sample (random with replacement)
99 - - **Max depth**: 25 levels per tree
100 - - **Min leaf**: 3 samples minimum per leaf node
101 - - **Features per split**: sqrt(35) = ~6 random features sampled per split decision
102 - - **Split criterion**: Gini impurity
103 - - **Parallelism**: Trees trained in parallel via rayon
104 -
105 - ### Evaluation
106 -
107 - - **5-fold stratified cross-validation** (preserves class distribution)
108 - - **94.4% strict accuracy** on 4,343 samples
109 - - Per-class precision, recall, and F1 computed across all folds
110 -
111 - ### Output
112 -
113 - - Model file: `crates/audiofiles-core/models/layer2_drum.json` (4.0 MB)
114 - - Format: JSON array of 200 trees + class metadata
115 - - Each tree node is either a `Split { feature, threshold, left, right }` or `Leaf { class }`
116 -
117 - ---
118 -
119 - ## Model Loading
120 -
121 - The model is embedded at compile time and deserialized lazily on first use:
122 -
123 - ```rust
124 - static LAYER2_MODEL: OnceLock<RandomForestModel> = OnceLock::new();
125 -
126 - fn layer2_model() -> &'static RandomForestModel {
127 - LAYER2_MODEL.get_or_init(|| {
128 - serde_json::from_slice(LAYER2_MODEL_BYTES)
129 - .expect("embedded Layer 2 model is invalid JSON")
130 - })
131 - }
132 - ```
133 -
134 - - `include_bytes!` embeds `layer2_drum.json` into the binary
135 - - `OnceLock` ensures deserialization happens exactly once
136 - - After init, all subsequent calls return a static reference (zero cost)
137 -
138 - ---
139 -
140 - ## Database Integration
141 -
142 - Classification results are stored in the `audio_analysis` table:
143 -
144 - | Column | Type | Description |
145 - |--------|------|-------------|
146 - | classification | TEXT | SampleClass as lowercase string (e.g., "kick") |
147 - | classification_confidence | REAL | 0.0--1.0; RF vote fraction for drums, heuristic confidence for non-drums |
148 -
149 - ---
150 -
151 - ## Retraining
152 -
153 - To retrain the model with new or updated training data:
154 -
155 - 1. Organize labeled samples in `~/Git/Drums/test_data/{class}/`
156 - 2. Run `cargo run -p audiofiles-train`
157 - 3. The binary outputs cross-validation metrics and writes `layer2_drum.json`
158 - 4. Rebuild audiofiles to embed the updated model
159 -
160 - ---
161 -
162 - ## Key Files
85 + ## Key files
163 86
164 87 | What | Where |
165 88 |------|-------|
166 - | Two-layer classifier | `crates/audiofiles-core/src/analysis/classify.rs` |
89 + | Rule-based classifier + taxonomy | `crates/audiofiles-core/src/analysis/classify.rs` |
90 + | Analysis orchestrator + smart-skip | `crates/audiofiles-core/src/analysis/mod.rs` |
167 91 | Spectral features | `crates/audiofiles-core/src/analysis/spectral.rs` |
168 92 | MFCC computation | `crates/audiofiles-core/src/analysis/mfcc.rs` |
169 93 | Crest factor, attack time | `crates/audiofiles-core/src/analysis/basic.rs` |
170 - | Training pipeline | `crates/audiofiles-train/src/main.rs` |
171 - | Embedded model | `crates/audiofiles-core/models/layer2_drum.json` |
172 - | Analysis orchestrator | `crates/audiofiles-core/src/analysis/mod.rs` |
94 + | Feature-vector schema (migration 020) | `crates/audiofiles-core/src/db.rs` |
95 + | Full design + retired-heuristics reference | `_private/docs/audiofiles/design-user-classifier.md` |
M crates/audiofiles-core/src/db.rs +49 -6
@@ -1117,6 +1117,47 @@
1117 1117 END;
1118 1118 "#;
1119 1119
1120 + const MIGRATION_020: &str = r#"
1121 + -- Phase 0 of the hybrid tag classifier: persist the 35-element feature vector
1122 + -- (9 scalar + 26 MFCC) per sample as the foundation for the rules + k-NN pipeline.
1123 + -- The vector is deterministic DSP (non-reversible to audio), stored as a JSON array
1124 + -- of f64. feat_version stamps the extraction layout so stale vectors can be recomputed
1125 + -- rather than silently mixed.
1126 + CREATE TABLE IF NOT EXISTS sample_features (
1127 + hash TEXT PRIMARY KEY REFERENCES samples(hash) ON DELETE CASCADE,
1128 + feat_version INTEGER NOT NULL,
1129 + vector TEXT NOT NULL,
1130 + computed_at INTEGER NOT NULL
1131 + );
1132 + CREATE INDEX IF NOT EXISTS idx_sample_features_version ON sample_features(feat_version);
1133 +
1134 + -- Sync triggers (mirror audio_analysis: cleartext hash row_id, JSON payload).
1135 + CREATE TRIGGER IF NOT EXISTS sync_sample_features_insert AFTER INSERT ON sample_features
1136 + WHEN (SELECT value FROM sync_state WHERE key = 'applying_remote') != '1'
1137 + BEGIN
1138 + INSERT INTO sync_changelog (table_name, op, row_id, data)
1139 + VALUES ('sample_features', 'INSERT', NEW.hash,
1140 + json_object('hash', NEW.hash, 'feat_version', NEW.feat_version,
1141 + 'vector', NEW.vector, 'computed_at', NEW.computed_at));
1142 + END;
1143 +
1144 + CREATE TRIGGER IF NOT EXISTS sync_sample_features_update AFTER UPDATE ON sample_features
1145 + WHEN (SELECT value FROM sync_state WHERE key = 'applying_remote') != '1'
1146 + BEGIN
1147 + INSERT INTO sync_changelog (table_name, op, row_id, data)
1148 + VALUES ('sample_features', 'UPDATE', NEW.hash,
1149 + json_object('hash', NEW.hash, 'feat_version', NEW.feat_version,
1150 + 'vector', NEW.vector, 'computed_at', NEW.computed_at));
1151 + END;
1152 +
1153 + CREATE TRIGGER IF NOT EXISTS sync_sample_features_delete AFTER DELETE ON sample_features
1154 + WHEN (SELECT value FROM sync_state WHERE key = 'applying_remote') != '1'
1155 + BEGIN
1156 + INSERT INTO sync_changelog (table_name, op, row_id, data)
1157 + VALUES ('sample_features', 'DELETE', OLD.hash, NULL);
1158 + END;
1159 + "#;
1160 +
1120 1161 /// Register `hash_row_id(salt, key) -> TEXT` as a deterministic SQLite
1121 1162 /// function on the given connection. Used by the M018 sync triggers so the
1122 1163 /// `sync_changelog.row_id` field never carries cleartext content (tag strings,
@@ -1216,6 +1257,7 @@
1216 1257 MIGRATION_017,
1217 1258 MIGRATION_018,
1218 1259 MIGRATION_019,
1260 + MIGRATION_020,
1219 1261 ];
1220 1262
1221 1263 for (i, sql) in MIGRATIONS.iter().enumerate() {
@@ -1373,6 +1415,7 @@
1373 1415 "collections",
1374 1416 "edit_history",
1375 1417 "fingerprints",
1418 + "sample_features",
1376 1419 "samples",
1377 1420 "sync_changelog",
1378 1421 "sync_state",
@@ -1392,7 +1435,7 @@
1392 1435 .conn()
1393 1436 .query_row("PRAGMA user_version", [], |row| row.get(0))
1394 1437 .unwrap();
1395 - assert_eq!(version, 19);
1438 + assert_eq!(version, 20);
1396 1439 }
1397 1440
1398 1441 #[test]
@@ -1403,7 +1446,7 @@
1403 1446 .conn()
1404 1447 .query_row("PRAGMA user_version", [], |row| row.get(0))
1405 1448 .unwrap();
1406 - assert_eq!(version, 19);
1449 + assert_eq!(version, 20);
1407 1450 }
1408 1451
1409 1452 /// Open a fresh file-backed DB, close, reopen. The second open re-enters
@@ -1422,7 +1465,7 @@
1422 1465 .conn()
1423 1466 .query_row("PRAGMA user_version", [], |row| row.get(0))
1424 1467 .unwrap();
1425 - assert_eq!(version, 19);
1468 + assert_eq!(version, 20);
1426 1469 }
1427 1470
1428 1471 /// Simulates the worst-case recovery path: a prior partial migration left
@@ -1466,7 +1509,7 @@
1466 1509 .conn()
1467 1510 .query_row("PRAGMA user_version", [], |row| row.get(0))
1468 1511 .unwrap();
1469 - assert_eq!(version, 19);
1512 + assert_eq!(version, 20);
1470 1513 }
1471 1514
1472 1515 /// M018 contract: the `sync_changelog.row_id` for sensitive tables must
@@ -1688,7 +1731,7 @@
1688 1731 let initial_version: i32 = conn
1689 1732 .query_row("PRAGMA user_version", [], |row| row.get(0))
1690 1733 .unwrap();
1691 - assert_eq!(initial_version, 19);
1734 + assert_eq!(initial_version, 20);
1692 1735
1693 1736 let batch = format!(
1694 1737 "BEGIN;\n{}\nPRAGMA user_version = 999;\nCOMMIT;",
@@ -1751,7 +1794,7 @@
1751 1794 .conn()
1752 1795 .query_row("PRAGMA user_version", [], |row| row.get(0))
1753 1796 .unwrap();
1754 - assert_eq!(version, 19);
1797 + assert_eq!(version, 20);
1755 1798 }
1756 1799
1757 1800 #[test]
M crates/audiofiles-core/src/similarity.rs +2
@@ -494,6 +494,8 @@
494 494 crest_factor: Some(3.0),
495 495 attack_time: Some(0.01),
496 496 classification_confidence: None,
497 + feature_vector: None,
498 + feature_version: None,
497 499 };
498 500 analysis::save_analysis(db, &result).unwrap();
499 501 }
M crates/audiofiles-core/src/test_helpers.rs +2
@@ -56,6 +56,8 @@
56 56 crest_factor: None,
57 57 attack_time: None,
58 58 classification_confidence: None,
59 + feature_vector: None,
60 + feature_version: None,
59 61 };
60 62 crate::analysis::save_analysis(db, &result).unwrap();
61 63
M crates/audiofiles-browser/src/state/tests.rs +2
@@ -757,6 +757,8 @@
757 757 crest_factor: None,
758 758 attack_time: None,
759 759 classification_confidence: None,
760 + feature_vector: None,
761 + feature_version: None,
760 762 },
761 763 suggestions: vec![],
762 764 });
M crates/audiofiles-core/src/analysis/classify.rs +20 -438
@@ -1,12 +1,11 @@
1 - //! Two-layer ML classification system.
1 + //! Rule-based sample classification.
2 2 //!
3 - //! Layer 1 (rule-based): Broad class detection (Drum vs Bass/Vocal/Synth/etc.)
4 - //! Layer 2 (Random Forest): Fine-grained drum sub-classification (Kick/Snare/HiHat/Cymbal/Percussion)
5 - //!
6 - //! The RF model is trained offline by `audiofiles-train` and embedded via `include_bytes!`.
7 - //! If no trained model is available (empty trees), falls back to the rule-based `classify_full()`.
8 -
9 - use std::sync::OnceLock;
3 + //! Assigns a `SampleClass` from a priority-ordered threshold tree over cheap DSP
4 + //! features (`classify_full`). The trained Random Forest models were retired in the
5 + //! Phase 0 classifier rework; the captured thresholds + taxonomy live in
6 + //! `_private/docs/audiofiles/design-user-classifier.md`. The 35-feature vector this
7 + //! module assembles is now persisted (`sample_features`) as the foundation for the
8 + //! rules + k-NN tag pipeline.
10 9
11 10 use super::mfcc::MfccFeatures;
12 11 use super::spectral::SpectralFeatures;
@@ -15,6 +14,11 @@
15 14 /// Number of features in the classification feature vector.
16 15 pub const NUM_FEATURES: usize = 35;
17 16
17 + /// Version stamp for the feature-extraction layout/params. Bump whenever the
18 + /// 35-feature vector's composition changes so persisted vectors (and shared
19 + /// `.afcl` bundles) can be invalidated/recomputed rather than silently mixed.
20 + pub const FEATURE_VERSION: u32 = 1;
21 +
18 22 // ── SampleClass enum (unchanged) ──
19 23
20 24 /// High-level classification of a sample's content.
@@ -163,25 +167,6 @@
163 167 }
164 168 }
165 169
166 - // ── Broad class (Layer 1) ──
167 -
168 - /// Broad classification for Layer 1 routing.
169 - #[derive(Debug, Clone, Copy, PartialEq, Eq)]
170 - pub enum BroadClass {
171 - Drum,
172 - Bass,
173 - Vocal,
174 - Synth,
175 - Pad,
176 - Misc,
177 - Noise,
178 - Music,
179 - Ambience,
180 - Impact,
181 - Foley,
182 - Texture,
183 - }
184 -
185 170 // ── Classification result ──
186 171
187 172 /// Result of the two-layer classifier.
@@ -191,55 +176,6 @@
191 176 pub confidence: f64,
192 177 }
193 178
194 - // ── Decision tree types (custom format for embedded inference) ──
195 -
196 - /// A node in a serialized decision tree.
197 - #[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
198 - pub enum TreeNode {
199 - Split {
200 - feature: usize,
201 - threshold: f64,
202 - left: Box<TreeNode>,
203 - right: Box<TreeNode>,
204 - },
205 - Leaf {
206 - class: u8,
207 - },
208 - }
209 -
210 - impl TreeNode {
211 - pub fn predict(&self, features: &[f64; NUM_FEATURES]) -> u8 {
212 - match self {
213 - TreeNode::Split {
214 - feature,
215 - threshold,
216 - left,
217 - right,
218 - } => {
219 - if *feature >= NUM_FEATURES {
220 - return 0; // fallback for malformed model
221 - }
222 - let val = features[*feature];
223 - // NaN goes left (conservative path) instead of always right
224 - if val.is_nan() || val <= *threshold {
225 - left.predict(features)
226 - } else {
227 - right.predict(features)
228 - }
229 - }
230 - TreeNode::Leaf { class } => *class,
231 - }
232 - }
233 - }
234 -
235 - /// A trained Random Forest model (collection of decision trees).
236 - #[derive(Debug, Clone, serde::Serialize, serde::Deserialize)]
237 - pub struct RandomForestModel {
238 - pub trees: Vec<TreeNode>,
239 - pub num_classes: u8,
240 - pub class_names: Vec<String>,
241 - }
242 -
243 179 // ── ClassifyInput ──
244 180
245 181 /// Input bundle for classification — collects all features used by the classifier.
@@ -317,314 +253,19 @@
317 253 }
318 254 }
319 255
320 - // ── Model loading ──
321 -
322 - /// Layer 2 drum model embedded at compile time.
323 - const LAYER2_DRUM_BYTES: &[u8] = include_bytes!("../../models/layer2_drum.json");
324 - const LAYER2_BASS_BYTES: &[u8] = include_bytes!("../../models/layer2_bass.json");
325 - const LAYER2_VOCAL_BYTES: &[u8] = include_bytes!("../../models/layer2_vocal.json");
326 - const LAYER2_SYNTH_BYTES: &[u8] = include_bytes!("../../models/layer2_synth.json");
327 -
328 - /// Lazily deserialized Layer 2 models (one per broad class).
329 - static LAYER2_DRUM_MODEL: OnceLock<RandomForestModel> = OnceLock::new();
330 - static LAYER2_BASS_MODEL: OnceLock<RandomForestModel> = OnceLock::new();
331 - static LAYER2_VOCAL_MODEL: OnceLock<RandomForestModel> = OnceLock::new();
332 - static LAYER2_SYNTH_MODEL: OnceLock<RandomForestModel> = OnceLock::new();
333 -
334 - /// Fallback empty model — classification falls back to rule-based layer 1 only.
335 - fn empty_model() -> RandomForestModel {
336 - RandomForestModel {
337 - trees: vec![],
338 - num_classes: 0,
339 - class_names: vec![],
340 - }
341 - }
342 -
343 - fn layer2_model() -> &'static RandomForestModel {
344 - LAYER2_DRUM_MODEL.get_or_init(|| {
345 - serde_json::from_slice(LAYER2_DRUM_BYTES).unwrap_or_else(|e| {
346 - tracing::error!("Failed to deserialize embedded drum model: {e}");
347 - empty_model()
348 - })
349 - })
350 - }
351 -
352 - fn layer2_bass_model() -> &'static RandomForestModel {
353 - LAYER2_BASS_MODEL.get_or_init(|| {
354 - serde_json::from_slice(LAYER2_BASS_BYTES).unwrap_or_else(|e| {
355 - tracing::error!("Failed to deserialize embedded bass model: {e}");
356 - empty_model()
357 - })
358 - })
359 - }
360 -
361 - fn layer2_vocal_model() -> &'static RandomForestModel {
362 - LAYER2_VOCAL_MODEL.get_or_init(|| {
363 - serde_json::from_slice(LAYER2_VOCAL_BYTES).unwrap_or_else(|e| {
364 - tracing::error!("Failed to deserialize embedded vocal model: {e}");
365 - empty_model()
366 - })
367 - })
368 - }
369 -
370 - fn layer2_synth_model() -> &'static RandomForestModel {
371 - LAYER2_SYNTH_MODEL.get_or_init(|| {
372 - serde_json::from_slice(LAYER2_SYNTH_BYTES).unwrap_or_else(|e| {
373 - tracing::error!("Failed to deserialize embedded synth model: {e}");
374 - empty_model()
375 - })
376 - })
377 - }
378 -
379 - /// Class labels per Layer 2 model. Index corresponds to class ID in TreeNode::Leaf.
380 - const DRUM_CLASSES: [SampleClass; 7] = [
381 - SampleClass::Kick,
382 - SampleClass::Snare,
383 - SampleClass::HiHat,
384 - SampleClass::Cymbal,
385 - SampleClass::Clap,
386 - SampleClass::Tom,
387 - SampleClass::Percussion,
388 - ];
389 -
390 - const BASS_CLASSES: [SampleClass; 3] = [
391 - SampleClass::GuitarBass,
392 - SampleClass::SynthBass,
393 - SampleClass::SubBass,
394 - ];
395 -
396 - const VOCAL_CLASSES: [SampleClass; 3] = [
397 - SampleClass::VocalChop,
398 - SampleClass::VocalPhrase,
399 - SampleClass::VocalChoir,
400 - ];
401 -
402 - const SYNTH_CLASSES: [SampleClass; 4] = [
403 - SampleClass::SynthLead,
404 - SampleClass::SynthStab,
405 - SampleClass::SynthPluck,
406 - SampleClass::SynthChord,
407 - ];
408 -
409 - // ── Layer 1: Rule-based broad classifier ──
410 -
411 - /// Classify a sample into a broad category (Layer 1).
412 - ///
413 - /// Uses simplified rules from the original `classify_full()` to determine if a sample
414 - /// is a drum hit vs other content types. Drum detection heuristic:
415 - /// short duration + fast attack + high crest factor.
416 - fn classify_broad(input: &ClassifyInput) -> (BroadClass, f64) {
417 - let d = input.duration;
418 - let c = input.centroid;
419 - let flat = input.flatness;
420 - let zcr = input.zcr;
421 - let bw = input.bandwidth;
422 - let cv = input.centroid_variance;
423 - let crest = input.crest_factor;
424 - let attack = input.attack_time;
425 -
426 - // Noise: energy spread nearly uniformly — but not short percussive sounds
427 - // (cymbals/hihats can have high flatness but are still drums)
428 - if flat > 0.7 && (d > 2.0 || attack > 0.1) {
429 - return (BroadClass::Noise, 0.9);
430 - }
431 -
432 - // Bright metallic sounds (cymbals, crashes) are drums even when sustained
433 - if d < 10.0 && c > 3000.0 && flat > 0.2 {
434 - return (BroadClass::Drum, 0.85);
435 - }
436 -
437 - // Drum detection: short + percussive transient
438 - if d < 2.0 && (attack < 0.05 || crest > 2.5) {
439 - let mut conf: f64 = 0.8;
440 - if attack < 0.02 {
441 - conf += 0.05;
442 - }
443 - if crest > 4.0 {
444 - conf += 0.05;
445 - }
446 - if d < 1.0 {
447 - conf += 0.05;
448 - }
449 - return (BroadClass::Drum, conf.min(0.95));
450 - }
451 -
452 - // Extended drum catch: short sounds with fast attack that didn't match above
453 - // (catches deeper kicks with low crest, toms with moderate attack)
454 - if d < 2.0 && attack < 0.1 && crest > 1.5 {
455 - return (BroadClass::Drum, 0.75);
456 - }
457 -
458 - // Impact: very sharp non-drum transient (must be longer than typical drums)
459 - if d > 1.0 && d < 5.0 && crest > 10.0 && attack < 0.005 {
460 - return (BroadClass::Impact, 0.85);
461 - }
462 -
463 - // Ambience: long, spectrally static, moderate noise
464 - if d > 5.0 && cv < 100_000.0 && flat > 0.15 && flat < 0.5 {
465 - return (BroadClass::Ambience, 0.85);
466 - }
467 -
468 - // Bass: low-frequency, tonal — but not short percussive sounds (kicks)
469 - if c < 400.0 && flat < 0.15 && d > 2.0 {
470 - return (BroadClass::Bass, 0.85);
471 - }
472 -
473 - // Vocal: mid-range, tonal, smooth waveform — require longer duration to avoid catching toms
474 - if d > 1.0 && c > 300.0 && c < 3000.0 && flat < 0.2 && zcr < 0.08 && crest < 2.0 {
475 - return (BroadClass::Vocal, 0.8);
476 - }
477 -
478 - // Pad: long, tonal, mid-range
479 - if d > 2.0 && flat < 0.2 && c > 200.0 && c < 2000.0 {
480 - return (BroadClass::Pad, 0.8);
481 - }
482 -
483 - // Texture: long, spectrally evolving
484 - if d > 2.0 && cv > 500_000.0 {
485 - return (BroadClass::Texture, 0.8);
486 - }
487 -
488 - // Foley: broadband, moderate noise — require longer duration to avoid catching drums
489 - if d > 1.0 && bw > 2000.0 && flat > 0.1 && flat < 0.5 {
490 - return (BroadClass::Foley, 0.75);
491 - }
492 -
493 - // Synth: tonal, mid-to-high centroid — require low crest to exclude drums
494 - if c > 500.0 && flat < 0.3 && zcr < 0.1 && crest < 2.0 {
495 - return (BroadClass::Synth, 0.75);
496 - }
497 -
498 - // Music: long catch-all
499 - if d > 3.0 {
500 - return (BroadClass::Music, 0.6);
501 - }
502 -
503 - // Short unclassified sounds are likely drums
504 - if d < 2.0 {
505 - return (BroadClass::Drum, 0.6);
506 - }
507 -
508 - // Misc: final catch-all (nothing else matched)
509 - (BroadClass::Misc, 0.5)
510 - }
511 -
512 - /// Map a BroadClass (non-Drum) to the corresponding SampleClass.
513 - fn broad_to_sample_class(broad: BroadClass) -> SampleClass {
514 - match broad {
515 - BroadClass::Drum => SampleClass::Percussion, // shouldn't reach here
516 - BroadClass::Bass => SampleClass::Bass,
517 - BroadClass::Vocal => SampleClass::Vocal,
518 - BroadClass::Synth => SampleClass::Synth,
519 - BroadClass::Pad => SampleClass::Pad,
520 - BroadClass::Misc => SampleClass::Misc,
521 - BroadClass::Noise => SampleClass::Noise,
522 - BroadClass::Music => SampleClass::Music,
523 - BroadClass::Ambience => SampleClass::Ambience,
524 - BroadClass::Impact => SampleClass::Impact,
525 - BroadClass::Foley => SampleClass::Foley,
526 - BroadClass::Texture => SampleClass::Texture,
527 - }
528 - }
529 -
530 - // ── Layer 2: Random Forest inference ──
531 -
532 - /// Run Layer 2 classification using a model and its class mapping.
533 - ///
534 - /// Returns (SampleClass, confidence) where confidence is the vote fraction.
535 - /// If the model has no trees, returns the fallback class with 0 confidence.
536 - fn predict_with_model(
537 - model: &RandomForestModel,
538 - classes: &[SampleClass],
539 - fallback: SampleClass,
540 - features: &[f64; NUM_FEATURES],
541 - ) -> (SampleClass, f64) {
542 - if model.trees.is_empty() {
543 - return (fallback, 0.0);
544 - }
545 -
546 - let mut votes = vec![0u32; model.num_classes as usize];
547 - for tree in &model.trees {
548 - let class_id = tree.predict(features) as usize;
549 - if class_id < votes.len() {
550 - votes[class_id] += 1;
551 - }
552 - }
553 -
554 - let total = model.trees.len() as f64;
555 - let (best_class_id, &best_count) = votes
556 - .iter()
557 - .enumerate()
558 - .max_by_key(|(_, count)| **count)
559 - .unwrap_or((0, &0));
560 -
561 - let confidence = best_count as f64 / total;
562 - let class = classes
563 - .get(best_class_id)
564 - .copied()
565 - .unwrap_or(fallback);
566 -
567 - (class, confidence)
568 - }
569 -
570 256 // ── Main entry point ──
571 257
572 - /// Two-layer ML classifier.
258 + /// Classify a sample into a `SampleClass` using the rule-based threshold tree.
573 259 ///
574 - /// Layer 1 (rule-based) determines broad class. Layer 2 (Random Forest) provides
575 - /// fine-grained sub-classification for Drum, Bass, Vocal, and Synth classes.
576 - /// If a Layer 2 model is not trained (empty trees), falls back to the broad class.
260 + /// The trained Random Forest layer was retired in the Phase 0 rework; this now
261 + /// delegates to `classify_full`. Confidence is reported as `0.0` to signal a
262 + /// rule-based (non-probabilistic) result — the same signal the prior no-model
263 + /// fallback emitted, which downstream tag suggestion already handles.
577 264 #[instrument(skip_all)]
578 265 pub fn classify_ml(input: &ClassifyInput) -> ClassificationResult {
579 - let drum_model = layer2_model();
580 -
581 - // If no drum model at all, fall back entirely to rule-based
582 - if drum_model.trees.is_empty() {
583 - return ClassificationResult {
584 - class: classify_full(input),
585 - confidence: 0.0,
586 - };
587 - }
588 -
589 - let (broad, broad_conf) = classify_broad(input);
590 - let features = input.to_feature_array();
591 -
592 - match broad {
593 - BroadClass::Drum => {
594 - let (class, conf) = predict_with_model(drum_model, &DRUM_CLASSES, SampleClass::Percussion, &features);
595 - ClassificationResult { class, confidence: conf }
596 - }
597 - BroadClass::Bass => {
598 - let model = layer2_bass_model();
599 - if model.trees.is_empty() {
600 - ClassificationResult { class: SampleClass::Bass, confidence: broad_conf }
601 - } else {
602 - let (class, conf) = predict_with_model(model, &BASS_CLASSES, SampleClass::Bass, &features);
603 - ClassificationResult { class, confidence: conf }
604 - }
605 - }
606 - BroadClass::Vocal => {
607 - let model = layer2_vocal_model();
608 - if model.trees.is_empty() {
609 - ClassificationResult { class: SampleClass::Vocal, confidence: broad_conf }
610 - } else {
611 - let (class, conf) = predict_with_model(model, &VOCAL_CLASSES, SampleClass::Vocal, &features);
612 - ClassificationResult { class, confidence: conf }
613 - }
614 - }
615 - BroadClass::Synth => {
616 - let model = layer2_synth_model();
617 - if model.trees.is_empty() {
618 - ClassificationResult { class: SampleClass::Synth, confidence: broad_conf }
619 - } else {
620 - let (class, conf) = predict_with_model(model, &SYNTH_CLASSES, SampleClass::Synth, &features);
621 - ClassificationResult { class, confidence: conf }
622 - }
623 - }
624 - _ => ClassificationResult {
625 - class: broad_to_sample_class(broad),
626 - confidence: broad_conf,
627 - },
266 + ClassificationResult {
267 + class: classify_full(input),
268 + confidence: 0.0,
628 269 }
629 270 }
630 271
@@ -1050,60 +691,6 @@
1050 691 );
1051 692 }
1052 693
1053 - #[test]
1054 - fn tree_node_prediction() {
1055 - let tree = TreeNode::Split {
1056 - feature: 1,
1057 - threshold: 1000.0,
1058 - left: Box::new(TreeNode::Leaf { class: 0 }), // kick (low centroid)
1059 - right: Box::new(TreeNode::Leaf { class: 2 }), // hihat (high centroid)
1060 - };
1061 - let mut features = [0.0; NUM_FEATURES];
1062 - features[1] = 500.0; // centroid below threshold
1063 - assert_eq!(tree.predict(&features), 0);
1064 - features[1] = 5000.0; // centroid above threshold
1065 - assert_eq!(tree.predict(&features), 2);
1066 - }
1067 -
1068 - #[test]
1069 - fn broad_classifier_detects_drums() {
1070 - let input = ClassifyInput {
1071 - duration: 0.3,
1072 - centroid: 600.0,
1073 - flatness: 0.15,
1074 - zcr: 0.04,
1075 - onset_strength: 50.0,
1076 - bandwidth: 500.0,
1077 - centroid_variance: 10_000.0,
1078 - crest_factor: 5.0,
1079 - attack_time: 0.003,
1080 - mfcc_means: [0.0; 13],
1081 - mfcc_variances: [0.0; 13],
1082 - };
1083 - let (broad, conf) = classify_broad(&input);
1084 - assert_eq!(broad, BroadClass::Drum);
1085 - assert!(conf >= 0.8);
1086 - }
1087 -
1088 - #[test]
1089 - fn broad_classifier_detects_noise() {
1090 - let input = ClassifyInput {
1091 - duration: 2.0,
1092 - centroid: 5000.0,
1093 - flatness: 0.8,
1094 - zcr: 0.3,
1095 - onset_strength: 10.0,
1096 - bandwidth: 5000.0,
1097 - centroid_variance: 100_000.0,
1098 - crest_factor: 1.5,
1099 - attack_time: 0.5,
1100 - mfcc_means: [0.0; 13],
1101 - mfcc_variances: [0.0; 13],
1102 - };
1103 - let (broad, _) = classify_broad(&input);
1104 - assert_eq!(broad, BroadClass::Noise);
1105 - }
1106 -
1107 694 #[test]
1108 695 fn smart_skip_drum_classes_skip_bpm_key() {
1109 696 // All drum sub-classes should skip BPM/key
@@ -1177,35 +764,6 @@
1177 764 assert!(SampleClass::Pad.has_rhythm());
1178 765 }
1179 766
1180 - #[test]
1181 - fn classify_ml_bass_routes_to_sub_class() {
1182 - // Bass-like input: low centroid, tonal, sustained
1183 - let input = ClassifyInput {
1184 - duration: 3.0,
Lines truncated
M crates/audiofiles-core/src/analysis/mod.rs +181 -25
M crates/audiofiles-core/src/analysis/suggest.rs +2
M crates/audiofiles-sync/src/service/mod.rs +4
M crates/audiofiles-sync/src/service/state.rs +28
D crates/audiofiles-core/models/layer2_bass.json -1
D crates/audiofiles-core/models/layer2_drum.json -1
D crates/audiofiles-core/models/layer2_synth.json -1
D crates/audiofiles-core/models/layer2_vocal.json -1
D crates/audiofiles-train/Cargo.toml -15
D crates/audiofiles-train/src/main.rs -680