Skip to main content

max / audiofiles

Rewrite the classifier docs: current pipeline, and the bundled-data reversal docs/ml_classifier.md presented the threshold tree as what exists today and the hybrid pipeline as forthcoming. Most of the pipeline is built and reachable from the app: rules (Layer A), exemplar k-NN (Layer B), the optional trained head, .afcl layer sharing, and the clustering cold start. The doc now separates the SampleClass column from the tag pipeline, describes all five layers, and records the tree's measured 23.6% strict accuracy with Clap and Tom unreachable. It also claimed no trained model ships in the binary, "which keeps the classifier free of any training-data copyright surface". Bundling an official .afcl built from Reverb Drum Machines reverses that, accepted deliberately: CC-BY 4.0 permits derivatives, attribution rides in AfclManifest.license_note, and 26-of-35-MFCC vectors are not reversible to audio. Licence discipline is load-bearing here, so the doc states plainly what ships, what it is derived from, and under what attribution, along with the corpus limitation (drum one-shots only) and the fact that nothing produces an official layer yet. Three copies of the same stale claim elsewhere: README advertised a 200-tree random forest at 94.4% for a component that was removed, architecture.md repeated the copyright-surface line, and classify.rs pointed at a design doc that no longer exists.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-07-29 23:46 UTC
Signed with PGP, not checked
Commit: a67309ea65a77c21cf9c283c1a30221ffc7e9503
Parent: 8bc8156
4 files changed, +116 insertions, -45 deletions
M README.md +1 -1
@@ -54,7 +54,7 @@
54 54
55 55 ### Audio Analysis
56 56 - **Analysis pipeline**: loudness (peak/RMS/LUFS), BPM detection, key detection, spectral analysis
57 - - **ML classification**: two-layer system: rule-based broad categories, then 200-tree random forest for drum sub-classification (94.4% accuracy on 4,343 samples)
57 + - **Classification**: deterministic DSP features feed a coarse `SampleClass` label plus a layered tag pipeline (user-authored rules, then k-NN over the samples you have already tagged). No trained model ships in the binary. See `docs/ml_classifier.md`
58 58 - **Loop detection**: identifies seamless loops via amplitude envelope analysis
59 59 - **Similarity search**: VP-tree indexed fingerprinting for finding similar and duplicate samples (O(log n) lookup)
60 60 - **Waveform display**: pre-computed peak data with click-to-seek playback
M docs/architecture.md +1 -1
@@ -81,7 +81,7 @@
81 81 - **BPM**: Onset-based tempo estimation using spectral flux and autocorrelation.
82 82 - **Key**: Musical key detection from chroma features.
83 83 - **MFCC**: Mel-Frequency Cepstral Coefficients computed from existing STFT magnitudes (26-band mel filterbank, log energy, DCT-II, 13 coefficients). Aggregated as mean + variance across frames (26 features total).
84 - - **Classification**: Deterministic DSP only. A priority-ordered threshold tree (`analysis/classify.rs`) maps the 35-feature vector (9 spectral/waveform + 26 MFCC) to a `SampleClass`; first rule to match wins. The 35-feature vector is persisted to `sample_features` for the forthcoming rules + k-NN tag pipeline. No trained model ships in the binary, which keeps the classifier free of training-data copyright surface; the Random Forest that previously refined drum sub-classes was removed. See `ml_classifier.md`. Accuracy is measured out-of-tree by `audiofiles-bench` against a labeled corpus.
84 + - **Classification**: Deterministic DSP only. A priority-ordered threshold tree (`analysis/classify.rs`) maps the 35-feature vector (9 spectral/waveform + 26 MFCC) to a `SampleClass`; first rule to match wins. It reports no confidence, being a threshold tree, and is measured at 23.6% strict accuracy, so it is being retired rather than tuned. The 35-feature vector is persisted to `sample_features` and feeds the layered tag pipeline (rules, exemplar k-NN, optional trained head, `.afcl` layers). Nothing model-derived ships in the binary today; the accepted plan is to bundle an official `.afcl` layer built from a CC-BY 4.0 corpus, with attribution carried in the manifest. See `ml_classifier.md`. Accuracy is measured out-of-tree by `audiofiles-bench` against a labeled corpus.
85 85 - **Loop detection**: Identifies whether a sample is a seamless loop.
86 86 - **Fingerprinting**: Computes an amplitude envelope fingerprint for near-duplicate detection across the library.
87 87 - **Tag suggestion**: Generates tag suggestions from analysis results (classification, BPM range, key, duration bracket) with confidence scores and human-readable reasons.
M docs/ml_classifier.md +108 -39
@@ -1,51 +1,112 @@
1 1 # audiofiles: Sample Classification
2 2
3 - audiofiles classifies samples from deterministic DSP features only. No trained model
4 - ships in the binary: the feature extraction is plain signal measurement, and tagging is
5 - driven by the user's own library. This keeps the classifier free of any training-data
6 - copyright surface.
3 + Classification runs on deterministic DSP features. Feature extraction is plain signal
4 + measurement, and the labels come from the user's own library: a rules layer the user
5 + writes, and a k-NN layer whose exemplars are the samples the user has already tagged.
6 + There is no neural network and no training step at build time.
7 7
8 - The system is being built out in phases toward a hybrid tag pipeline: a deterministic
9 - rules layer plus an exemplar k-NN layer learned from the user's library, feeding one
10 - provenance-tracked tag resolution step. The sections below describe what exists today.
8 + Two things run per sample, and they are separate systems:
11 9
12 - ## What exists today
10 + - **The `SampleClass` column.** One coarse label per sample, produced today by a
11 + hardcoded threshold tree. It feeds the filter panel, theme colours, the `{class}`
12 + rename token, and export resolution. The tree is being retired (see below).
13 + - **The tag pipeline.** Multi-label, provenance-tracked, per-library. This is where the
14 + work is going.
15 +
16 + ## Per-sample analysis
13 17
14 18 ```
15 19 Audio file
16 20 -> decode (Symphonia -> mono f32)
17 - -> feature extraction (9 spectral/basic + 26 MFCC = 35 features)
18 - -> classify_full() rule-based threshold tree -> SampleClass
19 - -> persist 35-feature vector (sample_features) + classification (audio_analysis)
21 + -> basic: peak, rms, crest factor, attack time (full signal)
22 + -> loudness: LUFS (full signal)
23 + -> spectral: centroid, flatness, rolloff, zcr, (capped at 30s)
24 + bandwidth, centroid variance
25 + -> mfcc: 13 means + 13 variances (capped at 30s)
26 + -> 35-feature vector -> sample_features (synced)
27 + -> classify_full() -> SampleClass -> audio_analysis (no confidence)
20 28 ```
21 29
22 - ### Rule-based classification (interim)
23 -
24 - `classify_full()` in `crates/audiofiles-core/src/analysis/classify.rs` assigns a
25 - `SampleClass` from a priority-ordered threshold tree over cheap DSP features (duration,
26 - spectral centroid/flatness/rolloff/ZCR/bandwidth, crest factor, attack time). Rules are
27 - evaluated in order; the first match wins. This is an interim bridge: it is scheduled to be
28 - superseded by the user-editable rules layer, at which point the hardcoded thresholds retire.
29 -
30 - The trained Random Forest models that previously refined drum/bass/vocal/synth
31 - sub-classes were removed. The captured thresholds and the full taxonomy are archived in
32 - `_private/docs/audiofiles/design-user-classifier.md` for reference.
33 -
34 - ### Persisted feature vector
35 -
36 - Every analyzed sample's 35-feature vector is stored in the `sample_features` table
37 - (`hash`, `feat_version`, `vector` as a JSON array, `computed_at`). It is the foundation
38 - for the forthcoming rules + k-NN tag pipeline, syncs across a user's own devices, and is
39 - stamped with `FEATURE_VERSION` so a change to the extraction layout invalidates stale
40 - vectors rather than silently mixing incompatible feature spaces.
30 + The 35-feature vector is the durable asset. It is stored per sample in `sample_features`
31 + (`hash`, `feat_version`, `vector` as a JSON array, `computed_at`), syncs across the user's
32 + own devices, and is stamped with `FEATURE_VERSION` so a change to the extraction layout
33 + invalidates stale vectors instead of mixing incompatible feature spaces.
41 34
42 35 ### Smart-skip
43 36
44 - The expensive BPM/key/loop stages are gated on cheap raw features rather than the
45 - classification label: BPM/loop run only for clips long enough to carry tempo; key runs
46 - only for clips that are long enough and not noise-like (high spectral flatness). The gate
47 - skips only the clearly-pointless cases and otherwise runs, so it never wrongly skips on a
48 - misread class.
37 + The expensive BPM/key/loop stages are gated on cheap raw features rather than on the
38 + classification label: BPM and loop detection run only for clips long enough to carry
39 + tempo, and key detection only for clips that are long enough and not noise-like (high
40 + spectral flatness). The gate skips the clearly pointless cases and otherwise runs, so a
41 + misread class cannot cause a wrong skip.
42 +
43 + ## The tag pipeline
44 +
45 + Layered, each layer recording where a tag came from. Provenance is `rule`, `ml`, or
46 + `cluster`; a tag with no provenance row is manual, is sticky, and the engine never
47 + reconciles it away.
48 +
49 + | Layer | Module | What it does |
50 + |-------|--------|--------------|
51 + | A | `rules.rs` | User-authored ordered `IF <conditions> THEN <actions>` over metadata + features. Ships empty. Deterministic and re-runnable. Provenance `rule`. |
52 + | B | `exemplar.rs` | k-NN over the user's own labelled samples. Multi-label soft scores with per-tag review/auto thresholds. Excludes `source = 'ml'` so it never trains on its own output, and carries the driving neighbours for explanation. Provenance `ml`. |
53 + | Trained head | `trained_head.rs` | Optional per-library one-vs-rest logistic regression distilled from the same exemplars, for large libraries: `O(tags x d)` inference instead of `O(N x d)`. A local regenerable cache, not synced. Deterministic training, no RNG. |
54 + | Sharing | `afcl.rs` | Portable `.afcl` layer: feature vectors, labels, rules, and policy thresholds, with no audio. Imported exemplars join the k-NN index below the user's own labels, imported rules arrive disabled for review, and the whole layer is removable in one action. |
55 + | Cold start | `cluster.rs` | k-means over the library's feature vectors so the user can name clusters and seed the first labels before any rules or tags exist. Deterministic seeding, no RNG. Provenance `cluster`. |
56 +
57 + All five are wired into the app today.
58 +
59 + ## The threshold tree is being retired
60 +
61 + `classify_full()` in `crates/audiofiles-core/src/analysis/classify.rs` assigns a
62 + `SampleClass` from a priority-ordered tree of about 40 hand-set thresholds over cheap DSP
63 + features. Rules are evaluated in order and the first match wins.
64 +
65 + Measured against 1,049 labelled drum samples it is 23.6% accurate on exact class match.
66 + Two of the seven drum classes, `Clap` and `Tom`, cannot be emitted by any rule, so a
67 + sample of either is wrong every time. Coarse questions do far better than fine ones on
68 + these features: a single feature and a single threshold separate low drums from bright
69 + drums at 92.4%. Instrument identity is not in the features; family is.
70 +
71 + So the tree is being replaced rather than tuned. Tuning about 40 parameters against a
72 + target a quarter of the corpus cannot reach would fit noise. Retiring the tree does not
73 + retire `SampleClass`: the taxonomy is used in 239 places across 41 files and stays. What
74 + goes is the constants in one function, and the open question is what feeds the column in
75 + its place.
76 +
77 + One consequence is already in effect. `classify_ml()` reports no confidence at all
78 + (`Option::None`), because a threshold tree has no probability behind it. Tag suggestion
79 + used to read that absence as a 0.0 and substitute a fixed 0.7, which presented a
80 + 23.6%-accurate guess as a moderately confident one. It now suggests a classification tag
81 + only when a real confidence is reported and clears 0.5, so the rule tree proposes nothing.
82 + The class still shows in the UI.
83 +
84 + ## What ships in the binary, and under what licence
85 +
86 + Today: no model and no third-party data. Every number the classifier uses is either
87 + computed from the user's audio at analysis time or written by hand as a threshold.
88 +
89 + That is changing, and the change is deliberate. Layer A ships empty and Layer B needs the
90 + user's own labels, so a new library starts with nothing to match against. The intended fix
91 + is a bundled official classifier layer: a `kind = "official"` `.afcl` built from the
92 + Reverb Drum Machines corpus, imported on first run and weighted below anything the user
93 + labels themselves.
94 +
95 + That layer will put data derived from a third-party dataset in the binary. The terms it
96 + rests on:
97 +
98 + - The corpus is CC-BY 4.0, which permits derivatives.
99 + - Attribution travels with the artifact in `AfclManifest.license_note`, which the UI
100 + surfaces.
101 + - An `.afcl` carries no audio. Each exemplar is the 35-number feature vector, 26 of which
102 + are MFCC means and variances, plus its labels. The vector is not reversible to audio.
103 +
104 + The corpus is drum-machine one-shots, so an official layer built from it has no exemplars
105 + for bass, vocal, pad, synth, texture, ambience, or foley. Those categories fall back to
106 + the user's own labels.
107 +
108 + No `.afcl` is in the repo yet. The import side is complete and gated on `feat_version`;
109 + nothing produces an official layer, and `ExportOptions` has no `kind` field to mark one.
49 110
50 111 ## Feature vector
51 112
@@ -79,17 +140,25 @@
79 140
80 141 | Table | Columns | Description |
81 142 |-------|---------|-------------|
82 - | audio_analysis | classification, classification_confidence | `SampleClass` string; confidence 0.0 for the rule-based path |
143 + | audio_analysis | classification, classification_confidence | `SampleClass` string; confidence is NULL for the rule-based path, which reports none |
83 144 | sample_features | hash, feat_version, vector, computed_at | The persisted 35-feature vector (JSON array) |
145 + | classifier_exemplars | id, layer_id, feat_version, vector, tags | Exemplars from an imported `.afcl`, no sample row |
146 + | trained_head | feat_version, exemplar_count, model, trained_at | Single-row local cache of the distilled model, not synced |
84 147
85 148 ## Key files
86 149
87 150 | What | Where |
88 151 |------|-------|
89 - | Rule-based classifier + taxonomy | `crates/audiofiles-core/src/analysis/classify.rs` |
152 + | Threshold tree + `SampleClass` taxonomy | `crates/audiofiles-core/src/analysis/classify.rs` |
153 + | Tag rules (Layer A) | `crates/audiofiles-core/src/rules.rs` |
154 + | Exemplar k-NN (Layer B) | `crates/audiofiles-core/src/analysis/exemplar.rs` |
155 + | Trained head | `crates/audiofiles-core/src/analysis/trained_head.rs` |
156 + | `.afcl` import and export | `crates/audiofiles-core/src/analysis/afcl.rs` |
157 + | Clustering cold start | `crates/audiofiles-core/src/analysis/cluster.rs` |
158 + | Tag suggestion | `crates/audiofiles-core/src/analysis/suggest.rs` |
90 159 | Analysis orchestrator + smart-skip | `crates/audiofiles-core/src/analysis/mod.rs` |
91 160 | Spectral features | `crates/audiofiles-core/src/analysis/spectral.rs` |
92 161 | MFCC computation | `crates/audiofiles-core/src/analysis/mfcc.rs` |
93 162 | Crest factor, attack time | `crates/audiofiles-core/src/analysis/basic.rs` |
94 - | Feature-vector schema (migration 020) | `crates/audiofiles-core/src/db.rs` |
95 - | Full design + retired-heuristics reference | `_private/docs/audiofiles/design-user-classifier.md` |
163 + </content>
164 + </invoke>
M crates/audiofiles-core/src/analysis/classify.rs +6 -4
@@ -2,10 +2,12 @@
2 2 //!
3 3 //! Assigns a `SampleClass` from a priority-ordered threshold tree over cheap DSP
4 4 //! features (`classify_full`). The trained Random Forest models were retired in the
5 - //! Phase 0 classifier rework; the captured thresholds + taxonomy live in
6 - //! `_private/docs/audiofiles/design-user-classifier.md`. The 35-feature vector this
7 - //! module assembles is now persisted (`sample_features`) as the foundation for the
8 - //! rules + k-NN tag pipeline.
5 + //! Phase 0 classifier rework. The 35-feature vector this module assembles is
6 + //! persisted (`sample_features`) as the foundation for the rules + k-NN tag pipeline.
7 + //!
8 + //! The tree itself is an interim bridge and is being retired: measured at 23.6%
9 + //! strict accuracy, with `Clap` and `Tom` unreachable by any rule. It reports no
10 + //! confidence, and nothing downstream should invent one for it. See `docs/ml_classifier.md`.
9 11
10 12 use super::mfcc::MfccFeatures;
11 13 use super::spectral::SpectralFeatures;