Skip to main content

max / audiofiles

Grade the classifier layer at family resolution, not per instrument The eval harness was sound and pointed at the wrong question. It cross-validated seven specific drum instruments; audiofiles settled on 2026-07-29 that classification happens at coarse family resolution, because instrument identity is not recoverable from these features and instrument labels are not perceptually coherent. Corpus labels are now projected onto families as rows are read back. The projection lives on the read side, not in labelled.rs, so the vault stays ground truth at the corpus's own resolution and the .afcl export is untouched. AF_BENCH_EVAL_LABELS reproduces the retired instrument run instead of deleting it out from under two write-ups. Run 3, low and drum-bright: 95.5% macro top-1, 97.4% macro held-out recall at a 95%-confident 95% precision bar, against 43-81% on three of seven instrument classes. The failure the earlier runs reported was mostly the resolution. Settles the open tom split: folded into low it is recovered 97.2% of the time against kick's 99.5%; split out, low drops to 57.6% held-out recall and the confusion runs both ways. Tom is the low register. Percussion is excluded rather than assigned. Its only per-file register signal is spectral centroid, which the layer already scores on, so labelling ground truth with it would grade the classifier against its own input. The verdict now prints scope on pass and fail both: five of seven families have no corpus material, so nothing here is a verdict on the layer.
Co-Authored-By
Claude Opus 5 (1M context) <noreply@anthropic.com>
Author: Max Johnson <me@maxj.phd> · 2026-08-07 03:14 UTC
Signed with PGP, not checked
Commit: 0a3a6f93520e6da2f99b354aae86b460003de8fe
Parent: 50fe541
5 files changed, +659 insertions, -35 deletions
M README.md +2
@@ -25,6 +25,8 @@
25 25 cargo run --release -p audiofiles-bench -- ingest # import and query
26 26 cargo run --release -p audiofiles-bench -- accuracy # bpm/key vs ground truth
27 27 cargo run --release -p audiofiles-bench -- layer-eval # classifier layer, cross-validated
28 + # AF_BENCH_EVAL_LABELS=family (default),
29 + # family-tom-split, instrument
28 30 ```
29 31
30 32 ## Workspace Architecture
M docs/ml_classifier.md +90 -7
@@ -159,15 +159,40 @@
159 159 candidate source for the pitched families; it is 16 kHz mono against the drums' 44.1
160 160 kHz, and five of the 35 features are Nyquist-bounded, so the two cannot simply be mixed
161 161 without the classifier learning sample rate as a proxy for instrument.
162 - 2. **The layer does not clear its own ship gate.** It is measured now:
162 + 2. **What it can claim is not measured yet, only what it gets right.**
163 163 `cargo run --release -p audiofiles-bench -- layer-eval` cross-validates the k-NN
164 - layer over the labelled corpus, five stratified folds, at the runtime `k` and the
165 - runtime thresholds. The gate lives in `layer_eval.rs` and was written before the
166 - first run.
164 + layer over the labelled corpus, five stratified folds, at the runtime `k`, with a
165 + threshold calibrated per class out of fold. The gate lives in `layer_eval.rs` and
166 + was written before the first run. At family resolution both measured families clear
167 + it (below). Accuracy is not the property that decides this, though: the layer's
168 + model is the user's own library, so the question is whether it gives the *same*
169 + answer as that library fills, and no metric for that exists yet.
170 + 3. **The layer would inherit the global 0.85 regardless.** `include_policy` is `false`
171 + in `afcl_gen`, so the calibrated per-class thresholds are not exported and no
172 + measured number survives the trip into a shipped artifact.
167 173
168 - Both are tracked in the audiofiles GoingsOn project.
174 + All three are tracked in the audiofiles GoingsOn project.
169 175
170 - ### What the measurement says
176 + ### Retracted: runs 1 and 2 answer the wrong question
177 +
178 + The two tables below grade **seven specific drum instruments**. Both halves of that
179 + were settled against on 2026-07-29, before either run happened:
180 +
181 + - audiofiles classifies at coarse family resolution. Instrument identity is not
182 + recoverable from these features (33.4% for a 7-way specific split with ~40 tuned
183 + thresholds, against 92.4% for families on one unfitted cut), instrument labels are
184 + not perceptually coherent (the best one lands in its top perceptual bin 54% of the
185 + time), and only 13.5% of real users tag a specific instrument at all.
186 + - Drums-only is the defect the scope work exists to fix, not the scope to measure
187 + within.
188 +
189 + They are kept here because the gate change between them is worth auditing and because
190 + deleting a published number is worse than marking it. They are not superseded results,
191 + they are results for a question audiofiles is not asking. Do not build on them, and do
192 + not compare them against the family numbers further down: different label space,
193 + different corpus subset. `AF_BENCH_EVAL_LABELS=instrument` reproduces them.
194 +
195 + ### What runs 1 and 2 said (instrument resolution, retired)
171 196
172 197 First run, 1049 files across the seven drum classes, features v5:
173 198
@@ -195,7 +220,7 @@
195 220 class and is not comparable across classes. Nor is 0.85 itself measured: it predates
196 221 any evaluation of this layer.
197 222
198 - ### The per-class answer
223 + ### The per-class answer (instrument resolution, retired)
199 224
200 225 So the harness calibrates a threshold per class instead: the most permissive one at
201 226 which the class is 95%-confident (Wilson lower bound) of holding 95% precision. Those
@@ -224,6 +249,64 @@
224 249 clave, maraca, bongo, conga), and the confusion matrix scatters it across every other
225 250 row at 46.7% top-1. No threshold fixes a class that is not one sound.
226 251
252 + ### Run 3, at family resolution
253 +
254 + Same harness, same folds, same calibration. The only change is that corpus labels are
255 + projected onto coarse families as they are read back (`bench/src/families.rs`), so the
256 + index carries family tags and the k-NN aggregates a neighbourhood across every drum in
257 + a family instead of splitting it seven ways.
258 +
259 + Two of the seven families have corpus material. `low` is kick plus tom, `drum-bright`
260 + is snare, clap, hi-hat and cymbal.
261 +
262 + | | n | top-1 recall | calibrated threshold | held-out precision | held-out recall |
263 + |---|---|---|---|---|---|
264 + | low | 451 | 98.2% | 0.652 | 96.7% | 97.6% |
265 + | drum-bright | 431 | 92.8% | 0.194 | 96.8% | 97.2% |
266 + | macro | 882 | 95.5% | | 96.1% | 97.4% |
267 +
268 + Both families clear the gate, and clear it by a distance the instrument runs never
269 + approached: held-out recall at a 95%-confident 95% precision bar goes from 43-81% on
270 + three of seven classes to 97% on both families. The failure the instrument runs
271 + reported was mostly the resolution, not the layer.
272 +
273 + Three things to read carefully rather than celebrate:
274 +
275 + - **`drum-bright` calibrates to 0.194, and the threshold moves 0.13 to 0.34 across
276 + folds.** A class holding 97% precision at a fifth of its neighbourhood is a class the
277 + index rarely confuses, but a threshold that unstable is not yet a property to ship.
278 + - **The shipped global 0.85 costs real recall here and is not the operating point
279 + either family wants.** At 0.85, `drum-bright` recall is 84.0% against 97.2%
280 + calibrated, and 9% of samples get no tag at all. `include_policy` is still `false` in
281 + `afcl_gen`, so a shipped layer today inherits 0.85 and none of the calibrated numbers
282 + hold.
283 + - **`k` still selects on the data it reports.** k=5 looks best at every resolution
284 + (98.1% top-1 macro), and that remains a hyperparameter chosen on the test set.
285 +
286 + #### Percussion is excluded, not scored
287 +
288 + 167 files, 16% of the corpus, dropped from training and testing both. It has no honest
289 + family label: it measures 47 low / 120 bright and 13% coherence across 19 perceptual
290 + bins, and the only per-file register signal available is spectral centroid, which is
291 + one of the 35 features the layer scores on. Labelling ground truth with the
292 + classifier's own input would report a class it cannot miss. `texture` is its likely
293 + home once there is material to check that against.
294 +
295 + #### Tom folds into `low`
296 +
297 + The one open split in `af-coarse-families`, and the corpus answers it. Running tom as
298 + its own class (`AF_BENCH_EVAL_LABELS=family-tom-split`) is worse on every axis:
299 +
300 + | | macro top-1 | macro held-out recall | worst class held-out recall |
301 + |---|---|---|---|
302 + | tom folded into `low` | 95.5% | 97.4% | 97.2% |
303 + | tom held out | 92.4% | 78.6% | 57.6% |
304 +
305 + Split out, `low` (kick alone) drops to 57.6% held-out recall and the confusion is
306 + mutual: 20 toms read as `low`, 10 kicks read as tom. Folded in, toms are recovered as
307 + `low` 97.2% of the time against kick's 99.5%, so the family's members behave alike.
308 + Tom is not a family. It is the low register.
309 +
227 310 ## Feature vector
228 311
229 312 35 features: 9 scalar + 13 MFCC means + 13 MFCC variances.
M crates/audiofiles-bench/src/layer_eval.rs +255 -28
@@ -39,9 +39,31 @@
39 39 //! The output still reports the shipped defaults, because "what happens if this
40 40 //! ships unchanged" remains a real question with a bad answer.
41 41 //!
42 + //! # And then the whole question changed, which needs saying louder
43 + //!
44 + //! Runs 1 and 2 (2026-08-06, `02395cb` and `50fe541`) graded seven **specific drum
45 + //! instruments**. Both halves of that were settled against on 2026-07-29 and this
46 + //! module did it anyway: audiofiles classifies at coarse family resolution because
47 + //! instrument identity is not in these features (33.4% with ~40 tuned thresholds
48 + //! against 92.4% for families on one unfitted cut, and instrument labels are not
49 + //! perceptually coherent), and drums-only is the defect the whole scope effort
50 + //! exists to fix rather than the scope to measure within. The machinery below was
51 + //! never the problem: fold split, index construction, calibration, the Wilson
52 + //! bound and the confusion matrix are all label-agnostic. The label mapping was.
53 + //!
54 + //! So the corpus is now projected onto families as it is read back
55 + //! ([`crate::families`]), and everything is graded at that resolution. Two things
56 + //! follow that a reader should not have to infer:
57 + //!
58 + //! - The drum corpus reaches **two of seven** families, `low` and `drum-bright`.
59 + //! Nothing here is a verdict on the layer. Five families are unmeasured.
60 + //! - Runs 1 and 2 answer a retired question. They are not superseded results, they
61 + //! are results for something else, and `docs/ml_classifier.md` says so.
62 + //!
42 63 //! Usage: `cargo run --release -p audiofiles-bench -- layer-eval`
43 64 //! Env: `AF_BENCH_CORPUS`, `AF_BENCH_VAULT`, `AF_BENCH_EVAL_FOLDS` (default 5),
44 65 //! `AF_BENCH_EVAL_K` (comma-separated sweep, default `5,10,15,25,50`),
66 + //! `AF_BENCH_EVAL_LABELS` (`family` default, `family-tom-split`, `instrument`),
45 67 //! `AF_BENCH_JSON`.
46 68 //!
47 69 //! [`DEFAULT_AUTO_THRESHOLD`]: audiofiles_core::analysis::exemplar::DEFAULT_AUTO_THRESHOLD
@@ -57,7 +79,8 @@
57 79 use audiofiles_core::db::Database;
58 80
59 81 use crate::calibration::{self, Counts, Point};
60 - use crate::labelled::{self, label_for_tag};
82 + use crate::families::{self, LabelSpace};
83 + use crate::labelled;
61 84 use crate::report::Report;
62 85
63 86 /// Default fold count.
@@ -111,12 +134,19 @@
111 134 struct Row {
112 135 hash: String,
113 136 vector: Vec<f64>,
137 + /// Labels in the evaluation's space, so the index under test carries the tags
138 + /// being graded.
114 139 tags: Vec<String>,
115 140 /// The single class this row is ground truth for, or `None` when the corpus
116 141 /// gave it more than one. Content-addressed import collapses a file that
117 142 /// appears in two class folders into one row carrying both tags; its true
118 143 /// class is undecidable, so it trains but is never tested.
119 144 truth: Option<String>,
145 + /// The corpus folder(s) this row came from, before projection. Kept so a
146 + /// family's members can be broken out by where they came from: `low` is kick
147 + /// plus tom, and whether the layer recovers a tom as readily as a kick is the
148 + /// open split `af-coarse-families` asks about.
149 + origin: String,
120 150 }
121 151
122 152 /// What one test sample produced under one `k`.
@@ -129,14 +159,32 @@
129 159 /// The fold this sample was held out of, so a threshold is never calibrated
130 160 /// on the predictions it is graded against.
131 161 fold: usize,
162 + /// Corpus folder(s) behind the truth label. See [`Row::origin`].
163 + origin: String,
132 164 }
133 165
134 166 fn pct(v: Option<f64>) -> String {
135 167 v.map_or_else(|| "-".to_string(), |x| format!("{:.1}%", x * 100.0))
136 168 }
137 169
138 - /// Read the analysed corpus back out of the scratch vault as scoreable rows.
139 - fn load_rows(db: &Database) -> Result<Vec<Row>, String> {
170 + /// What `load_rows` set aside, so no exclusion is silent.
171 + struct Dropped {
172 + /// Rows whose every corpus tag projects nowhere in this label space.
173 + unprojectable: usize,
174 + /// Corpus labels those rows came from.
175 + origins: BTreeSet<String>,
176 + }
177 +
178 + /// Read the analysed corpus back out of the scratch vault as scoreable rows,
179 + /// projected onto `space`.
180 + ///
181 + /// The projection happens here rather than in [`labelled`] on purpose. The vault
182 + /// is ground truth at the finest resolution the corpus carries, which is what
183 + /// `afcl_gen` exports from; grading at a coarser resolution is a property of the
184 + /// evaluation, not of the corpus, so it belongs on the read side. That also keeps
185 + /// the exported layer byte-identical and leaves the retired instrument question
186 + /// runnable instead of deleted.
187 + fn load_rows(db: &Database, space: LabelSpace) -> Result<(Vec<Row>, Dropped), String> {
140 188 let conn = db.conn();
141 189
142 190 let mut tags_by_hash: HashMap<String, Vec<String>> = HashMap::new();
@@ -156,6 +204,10 @@
156 204 }
157 205
158 206 let mut out = Vec::new();
207 + let mut dropped = Dropped {
208 + unprojectable: 0,
209 + origins: BTreeSet::new(),
210 + };
159 211 let mut stmt = conn
160 212 .prepare("SELECT hash, vector FROM sample_features WHERE feat_version = ?1 ORDER BY hash")
161 213 .map_err(|e| format!("features query: {e}"))?;
@@ -166,11 +218,11 @@
166 218 .map_err(|e| format!("features query: {e}"))?;
167 219 for r in rows {
168 220 let (hash, json) = r.map_err(|e| format!("features row: {e}"))?;
169 - let Some(mut tags) = tags_by_hash.remove(&hash) else {
221 + let Some(mut corpus_tags) = tags_by_hash.remove(&hash) else {
170 222 continue;
171 223 };
172 - tags.sort();
173 - tags.dedup();
224 + corpus_tags.sort();
225 + corpus_tags.dedup();
174 226 let vector: Vec<f64> =
175 227 serde_json::from_str(&json).map_err(|e| format!("vector for {hash}: {e}"))?;
176 228 // The index drops these too (`is_usable_vector`), so counting them as
@@ -178,15 +230,43 @@
178 230 if vector.len() != NUM_FEATURES || !vector.iter().all(|x| x.is_finite()) {
179 231 continue;
180 232 }
233 +
234 + // Project, then dedup again: two instrument tags landing in one family is
235 + // not ambiguity, it is the coarser question being easier. A file in both
236 + // the kick and tom folders has no instrument truth and a perfectly good
237 + // family one.
238 + let mut tags: Vec<String> = corpus_tags
239 + .iter()
240 + .filter_map(|t| families::project(space, t))
241 + .map(str::to_string)
242 + .collect();
243 + tags.sort();
244 + tags.dedup();
245 + if tags.is_empty() {
246 + dropped.unprojectable += 1;
247 + dropped.origins.extend(
248 + corpus_tags
249 + .iter()
250 + .map(|t| labelled::label_for_tag(t).to_string()),
251 + );
252 + continue;
253 + }
254 +
181 255 let truth = (tags.len() == 1).then(|| tags[0].clone());
256 + let origin = corpus_tags
257 + .iter()
258 + .map(|t| labelled::label_for_tag(t))
259 + .collect::<Vec<_>>()
260 + .join("+");
182 261 out.push(Row {
183 262 hash,
184 263 vector,
185 264 tags,
186 265 truth,
266 + origin,
187 267 });
188 268 }
189 - Ok(out)
269 + Ok((out, dropped))
190 270 }
191 271
192 272 /// Assign each row a fold, stratified by class.
@@ -266,6 +346,7 @@
266 346 config: &AnalysisConfig,
267 347 folds: usize,
268 348 k_sweep: &[usize],
349 + space: LabelSpace,
269 350 ) {
270 351 println!("━━━ CLASSIFIER LAYER EVALUATION ━━━");
271 352 println!();
@@ -274,6 +355,7 @@
274 355 println!(" features v{FEATURE_VERSION}");
275 356 println!(" k {DEFAULT_K} (runtime default), sweeping {k_sweep:?}");
276 357 println!(" folds {folds}, stratified by class");
358 + println!(" labels {}", space.describe());
277 359 println!();
278 360 println!(" Gate:");
279 361 println!(
@@ -306,13 +388,28 @@
306 388 }
307 389 };
308 390
309 - let rows = match load_rows(&built.db) {
391 + let (rows, dropped) = match load_rows(&built.db, space) {
310 392 Ok(r) => r,
311 393 Err(e) => {
312 394 eprintln!("reading the vault back: {e}");
313 395 std::process::exit(1);
314 396 }
315 397 };
398 + if dropped.unprojectable > 0 {
399 + println!();
400 + println!(
401 + " {} row(s) carry no label in this space ({}) and are excluded from",
402 + dropped.unprojectable,
403 + dropped
404 + .origins
405 + .iter()
406 + .cloned()
407 + .collect::<Vec<_>>()
408 + .join(", ")
409 + );
410 + println!(" training and testing both:");
411 + println!("{}", families::DROPPED_NOTE);
412 + }
316 413 let ambiguous = rows.iter().filter(|r| r.truth.is_none()).count();
317 414 let testable = rows.len() - ambiguous;
318 415 if testable == 0 {
@@ -384,6 +481,7 @@
384 481 top1,
385 482 scores,
386 483 fold,
484 + origin: row.origin.clone(),
387 485 });
388 486 }
389 487 }
@@ -391,6 +489,8 @@
391 489 println!();
392 490
393 491 let mut report = Report::new("layer-eval");
492 + report.set("label_space", format!("{space:?}"));
493 + report.set("dropped_unprojectable", dropped.unprojectable);
394 494 report.set("folds", folds);
395 495 report.set("k", DEFAULT_K);
396 496 report.set("feat_version", FEATURE_VERSION);
@@ -406,14 +506,15 @@
406 506 .expect("the sweep always contains the runtime k");
407 507 report.set("tested", default_k.len());
408 508
409 - let top1 = print_confusion(default_k, &classes, &mut report);
410 - print_shipped_defaults(default_k, &classes, &mut report);
411 - print_threshold_sweep(default_k, &classes);
412 - let calibrated = print_calibration(default_k, &classes, folds, &mut report);
509 + let top1 = print_confusion(default_k, &classes, space, &mut report);
510 + print_origin_breakdown(default_k, &classes, space, &mut report);
511 + print_shipped_defaults(default_k, &classes, space, &mut report);
512 + print_threshold_sweep(default_k, &classes, space);
513 + let calibrated = print_calibration(default_k, &classes, folds, space, &mut report);
413 514 if k_sweep.len() > 1 {
414 515 print_k_sweep(&by_k, &classes, folds, &mut report);
415 516 }
416 - print_verdict(&classes, &top1, &calibrated, &mut report);
517 + print_verdict(&classes, &top1, &calibrated, space, &mut report);
417 518 report.write();
418 519 }
419 520
@@ -421,6 +522,7 @@
421 522 fn print_confusion(
422 523 predictions: &[Prediction],
423 524 classes: &[String],
525 + space: LabelSpace,
424 526 report: &mut Report,
425 527 ) -> BTreeMap<String, Counts> {
426 528 println!("━━━ TOP-1 CONFUSION (k = {DEFAULT_K}) ━━━");
@@ -432,12 +534,12 @@
432 534
433 535 let width = classes
434 536 .iter()
435 - .map(|c| label_for_tag(c).len().max(5))
537 + .map(|c| families::label_for(space, c).len().max(5))
436 538 .collect::<Vec<_>>();
437 539
438 540 print!(" {:<12}", "true \\ pred");
439 541 for (c, w) in classes.iter().zip(&width) {
440 - print!(" {:>w$}", label_for_tag(c), w = w);
542 + print!(" {:>w$}", families::label_for(space, c), w = w);
441 543 }
442 544 println!(" {:>6} {:>8}", "(none)", "recall");
443 545 println!(
@@ -450,7 +552,7 @@
450 552
451 553 for truth in classes {
452 554 let mine: Vec<&Prediction> = predictions.iter().filter(|p| &p.truth == truth).collect();
453 - print!(" {:<12}", label_for_tag(truth));
555 + print!(" {:<12}", families::label_for(space, truth));
454 556 let mut correct = 0usize;
455 557 for (pred, w) in classes.iter().zip(&width) {
456 558 let n = mine
@@ -477,7 +579,7 @@
477 579 .filter(|p| p.top1.as_deref() == Some(truth.as_str()))
478 580 .count();
479 581 if predicted_as == 0 {
480 - never_predicted.push(label_for_tag(truth));
582 + never_predicted.push(families::label_for(space, truth));
481 583 }
482 584 counts.insert(
483 585 truth.clone(),
@@ -511,7 +613,7 @@
511 613 report.set("top1_accuracy", round4(micro));
512 614 report.set("never_predicted", never_predicted.len());
513 615 for (tag, c) in &counts {
514 - let label = label_for_tag(tag);
616 + let label = families::label_for(space, tag);
515 617 if let Some(r) = c.recall() {
516 618 report.set(&format!("top1_{label}_recall"), round4(r));
517 619 }
@@ -522,9 +624,98 @@
522 624 counts
523 625 }
524 626
627 + /// Per-class top-1 recall broken out by the corpus folder each sample came from.
628 + ///
629 + /// A family is only a family if its members behave like one. `low` is kick plus
630 + /// tom, and `af-coarse-families` calls that the one open split worth measuring:
631 + /// tom sits between `low` and `bass` on centroid (p25-p75 904-1996 against kick's
632 + /// 397-885) and is 23% of the drum corpus, so folding it in silently assumes the
633 + /// answer. If toms are recovered as `low` about as often as kicks are, the fold
634 + /// holds; if they are systematically lost, `low` is two things wearing one label.
635 + ///
636 + /// This is the measurement, not a proposal to add a `tom` family. Reading it
637 + /// against `AF_BENCH_EVAL_LABELS=family-tom-split`, which grades tom as its own
638 + /// class, is what separates "tom is hard" from "tom is not low".
639 + fn print_origin_breakdown(
640 + predictions: &[Prediction],
641 + classes: &[String],
642 + space: LabelSpace,
643 + report: &mut Report,
644 + ) {
645 + let origins: BTreeSet<&str> = predictions.iter().map(|p| p.origin.as_str()).collect();
646 + // Nothing to say when every class is one folder: the table would be the
647 + // recall column of the confusion matrix, transposed.
648 + if origins.len() <= classes.len() {
649 + return;
650 + }
651 +
652 + println!("━━━ BY CORPUS ORIGIN (k = {DEFAULT_K}) ━━━");
653 + println!();
654 + println!(" The same top-1 answers, grouped by the folder the sample came from");
655 + println!(" rather than by the class it was projected onto. A family whose");
656 + println!(" members disagree here is not one family.");
657 + println!();
658 + println!(
659 + " {:<14} {:<14} {:>6} {:>9} most common wrong answer",
660 + "origin", "projects to", "n", "recall"
661 + );
662 + println!(" {}", "─".repeat(76));
663 +
664 + for origin in origins {
665 + let mine: Vec<&Prediction> = predictions.iter().filter(|p| p.origin == origin).collect();
666 + let Some(truth) = mine.first().map(|p| p.truth.clone()) else {
667 + continue;
668 + };
669 + let correct = mine
670 + .iter()
671 + .filter(|p| p.top1.as_deref() == Some(truth.as_str()))
672 + .count();
673 + let recall = correct as f64 / mine.len() as f64;
674 +
675 + // Where the misses go, which is the informative half: a tom read as
676 + // drum-bright says something different from a tom the index has no
677 + // answer for at all.
678 + let mut wrong: BTreeMap<&str, usize> = BTreeMap::new();
679 + for p in &mine {
680 + match p.top1.as_deref() {
681 + Some(t) if t != truth => *wrong.entry(t).or_default() += 1,
682 + None => *wrong.entry("(none)").or_default() += 1,
683 + _ => {}
684 + }
685 + }
686 + let worst = wrong.iter().max_by_key(|(_, n)| **n).map_or_else(
687 + || "-".to_string(),
688 + |(t, n)| {
689 + let label = if *t == "(none)" {
690 + "(none)"
691 + } else {
692 + families::label_for(space, t)
693 + };
694 + format!("{label} ({n})")
695 + },
696 + );
697 +
698 + println!(
699 + " {:<14} {:<14} {:>6} {:>9} {worst}",
700 + origin,
701 + families::label_for(space, &truth),
702 + mine.len(),
703 + pct(Some(recall)),
704 + );
705 + report.set(&format!("origin_{origin}_recall"), round4(recall));
706 + report.set(&format!("origin_{origin}_n"), mine.len());
707 + }
708 + println!();
709 + }
710 +
525 711 /// What the layer does today, unchanged: one global auto threshold for every
526 712 /// class. Kept because it is the status quo the ship decision is against.
527 - fn print_shipped_defaults(predictions: &[Prediction], classes: &[String], report: &mut Report) {
713 + fn print_shipped_defaults(
714 + predictions: &[Prediction],
715 + classes: &[String],
716 + space: LabelSpace,
717 + report: &mut Report,
718 + ) {
528 719 println!("━━━ AT THE SHIPPED DEFAULTS (one global threshold) ━━━");
529 720 println!();
530 721
@@ -545,7 +736,7 @@
545 736 let c = counts[class];
546 737 println!(
547 738 " {:<12} {:>7} {:>8} {:>10} {:>9}",
548 - label_for_tag(class),
739 + families::label_for(space, class),
549 740 c.actual(),
550 741 c.fired(),
551 742 pct(c.precision()),
@@ -574,7 +765,7 @@
574 765 println!();
575 766
576 767 for (tag, c) in &counts {
577 - let label = label_for_tag(tag);
768 + let label = families::label_for(space, tag);
578 769 if let Some(p) = c.precision() {
579 770 report.set(&format!("shipped_{label}_precision"), round4(p));
580 771 }
@@ -589,7 +780,7 @@
589 780 ///
590 781 /// This is the evidence that one global threshold cannot serve seven classes: read
591 782 /// down a column and the same number means a different thing in every row.
592 - fn print_threshold_sweep(predictions: &[Prediction], classes: &[String]) {
783 + fn print_threshold_sweep(predictions: &[Prediction], classes: &[String], space: LabelSpace) {
593 784 let points: BTreeMap<&String, Vec<Point>> = classes
594 785 .iter()
595 786 .map(|c| (c, class_points(predictions, c)))
@@ -608,7 +799,7 @@
608 799 println!();
609 800 println!(" {}", "─".repeat(12 + SWEEP_THRESHOLDS.len() * 8));
610 801 for class in classes {
611 - print!(" {:<12}", label_for_tag(class));
802 + print!(" {:<12}", families::label_for(space, class));
612 803 for t in SWEEP_THRESHOLDS {
613 804 let c = calibration::counts_at(&points[class], *t);
614 805 print!(" {:>7}", pct(metric(c)));
@@ -630,6 +821,7 @@
630 821 predictions: &[Prediction],
631 822 classes: &[String],
632 823 folds: usize,
824 + space: LabelSpace,
633 825 report: &mut Report,
634 826 ) -> BTreeMap<String, Counts> {
635 827 println!(
@@ -657,7 +849,7 @@
657 849 calibration::out_of_fold(&points, folds, GATE.target_precision, GATE.min_support);
658 850 out_of_fold.insert(class.clone(), oof);
659 851
660 - let label = label_for_tag(class);
852 + let label = families::label_for(space, class);
661 853 match in_sample {
662 854 Some(op) => println!(
663 855 " {:<12} {:>7} {:>10.3} {:>10} {:>9} {:>18}",
@@ -844,6 +1036,7 @@
844 1036 classes: &[String],
845 1037 top1: &BTreeMap<String, Counts>,
846 1038 calibrated: &BTreeMap<String, Counts>,
1039 + space: LabelSpace,
847 1040 report: &mut Report,
848 1041 ) {
849 1042 println!("━━━ VERDICT ━━━");
@@ -852,7 +1045,7 @@
852 1045 let mut failures: Vec<String> = Vec::new();
853 1046
854 1047 for class in classes {
855 - let label = label_for_tag(class);
1048 + let label = families::label_for(space, class);
856 1049 let c = calibrated[class];
857 1050 if c.fired() == 0 {
858 1051 failures.push(format!(
@@ -900,9 +1093,6 @@
900 1093 println!(" calibrated thresholds with it: set `include_policy` in afcl_gen and");
901 1094 println!(" export the tag_policy rows, or the layer inherits the global 0.85");
902 1095 println!(" and none of the above holds.");
903 - println!();
904 - println!(" This clears the measurement only. The layer is still drums-only, so");
905 - println!(" it answers for a bass with the nearest drum it knows; that is Phase C.");
906 1096 } else {
907 1097 println!(" FAIL on {} criterion/criteria:", failures.len());
908 1098 for f in &failures {
@@ -911,10 +1101,47 @@
911 1101 }
912 1102 println!();
913 1103
1104 + // Printed on pass and on fail both, because the scope caveat is not a
1105 + // consolation for a failure: a pass here is the more dangerous of the two to
1106 + // read as a verdict on the layer.
1107 + print_scope(classes, space);
1108 +
914 1109 report.set("gate_pass", failures.is_empty());
915 1110 report.set("gate_failures", failures.len());
916 1111 }
917 1112
1113 + /// What this corpus can and cannot support a claim about.
1114 + ///
1115 + /// The gate above says whether the classes present are shippable. It cannot say
1116 + /// anything about the classes absent, and the absent ones are the majority: the
1117 + /// corpus is drum one-shots, so it reaches two of the seven families and five have
1118 + /// no material at all. A layer that answers `drum-bright` confidently for a vocal
1119 + /// it has never seen passes every criterion above.
1120 + fn print_scope(classes: &[String], space: LabelSpace) {
1121 + if space == LabelSpace::Instrument {
1122 + println!(" SCOPE: this run grades specific drum instruments, which is the");
1123 + println!(" retired question (wiki af-browse-axes, 2026-07-29). Kept runnable so");
1124 + println!(" the results already written up stay reproducible. Do not extend it.");
1125 + println!();
1126 + return;
1127 + }
1128 +
1129 + let (covered, uncovered) = families::covered_families(classes);
1130 + println!(
1131 + " SCOPE: {} of {} families measured: {}.",
1132 + covered.len(),
1133 + families::FAMILIES.len(),
1134 + covered.join(", ")
1135 + );
1136 + if !uncovered.is_empty() {
1137 + println!(" No corpus material for: {}.", uncovered.join(", "));
1138 + println!(" The layer has never been asked about them and will answer with the");
1139 + println!(" nearest drum it knows. Nothing above is a verdict on the layer;");
1140 + println!(" widening the corpus is the next phase.");
1141 + }
1142 + println!();
1143 + }
1144 +
918 1145 /// Fold count from the environment, or [`DEFAULT_FOLDS`].
919 1146 pub(crate) fn folds_from_env() -> usize {
Lines truncated
M crates/audiofiles-bench/src/main.rs +2
@@ -47,6 +47,7 @@
47 47 mod accuracy;
48 48 mod afcl_gen;
49 49 mod calibration;
50 + mod families;
50 51 mod ingest;
51 52 mod labelled;
52 53 mod layer_eval;
@@ -351,6 +352,7 @@
351 352 &full_pipeline_config(),
352 353 layer_eval::folds_from_env(),
353 354 &layer_eval::k_sweep_from_env(),
355 + families::LabelSpace::from_env(),
354 356 );
355 357 return;
356 358 }
A crates/audiofiles-bench/src/families.rs +310
@@ -1,0 +1,310 @@
1 + //! The coarse family taxonomy, and the projection from corpus labels onto it.
2 + //!
3 + //! Everything the layer is graded on runs through here. The corpus is labelled at
4 + //! instrument resolution because that is what its folder names carry, but
5 + //! instrument resolution is not what audiofiles classifies at and has not been
6 + //! since 2026-07-29: the same 35 features separate coarse families at 92.4% on one
7 + //! unfitted centroid cut and specific instruments at 33.4% with ~40 tuned
8 + //! thresholds, and only 13.5% of real users tag a specific instrument at all.
9 + //! Wiki `af-coarse-families` is the taxonomy, `af-browse-axes` the evidence.
10 + //!
11 + //! So the vault keeps the fine labels (the `.afcl` export is built from the same
12 + //! vault and is unchanged by any of this) and the evaluation projects them onto
13 + //! families as it reads them back. One swap, no second corpus, and the retired
14 + //! instrument-resolution question stays reproducible via [`LabelSpace::Instrument`]
15 + //! rather than being deleted out from under the two write-ups that report it.
16 + //!
17 + //! # What this corpus can and cannot say
18 + //!
19 + //! The labelled corpus is 1,049 drum one-shots. Projected onto families it covers
20 + //! **two of seven**: `low` and `drum-bright`. It says nothing about `bass`,
21 + //! `tonal`, `vocal`, `texture` or `music`, and a number measured here must never
22 + //! be reported as a verdict on the layer. Widening the corpus is Phase 3.
23 +
24 + /// A coarse family: the resolution the classifier is meant to answer at.
25 + pub(crate) struct Family {
26 + /// Short label for report tables.
27 + pub(crate) label: &'static str,
28 + /// The tag carried in the index.
29 + pub(crate) tag: &'static str,
30 + }
31 +
32 + /// The seven families from `af-coarse-families`, in register order.
33 + ///
34 + /// The `family.` prefix is a **placeholder and is not ratified**. The note names
35 + /// these as bare words (`low`, `drum-bright`, `music`) and the shipped vocabulary
36 + /// is `instrument.*`, which `low` and `music` do not fit under; a separate
37 + /// namespace keeps the k-NN's answers from contending with the tags the filename
38 + /// rules emit, and keeps provenance readable. Nothing here writes a tag into a
39 + /// user's library, so the measurement does not wait on the ruling. Anything that
40 + /// does ship tags must.
41 + pub(crate) const FAMILIES: &[Family] = &[
42 + Family {
43 + label: "low",
44 + tag: "family.low",
45 + },
46 + Family {
47 + label: "bass",
48 + tag: "family.bass",
49 + },
50 + Family {
51 + label: "tonal",
52 + tag: "family.tonal",
53 + },
54 + Family {
55 + label: "drum-bright",
56 + tag: "family.drum-bright",
57 + },
58 + Family {
59 + label: "vocal",
60 + tag: "family.vocal",
61 + },
62 + Family {
63 + label: "texture",
64 + tag: "family.texture",
65 + },
66 + Family {
67 + label: "music",
68 + tag: "family.music",
69 + },
70 + ];
71 +
72 + /// `tom` held out as its own class, to test whether it earns a family.
73 + ///
74 + /// Not a member of [`FAMILIES`] and not a shipping candidate. `af-coarse-families`
75 + /// calls tom the one open split worth measuring: it sits between `low` and `bass`
76 + /// on centroid (p25-p75 904-1996 against kick's 397-885) and is 246 files, 23% of
77 + /// the drum corpus, so folding it into `low` silently is a big assumption to leave
78 + /// untested.
79 + pub(crate) const TOM_PROVISIONAL: &str = "family.tom-provisional";
80 +
81 + /// Which resolution an evaluation runs at.
82 + #[derive(Clone, Copy, PartialEq, Eq, Debug)]
83 + pub(crate) enum LabelSpace {
84 + /// Seven specific drum instruments. The retired question, kept runnable so the
85 + /// results already written up stay reproducible.
86 + Instrument,
87 + /// Coarse families, `tom` folded into `low` as the note proposes.
88 + Family,
89 + /// Coarse families, `tom` held out as its own class.
90 + FamilyTomSplit,
91 + }
92 +
93 + impl LabelSpace {
94 + /// Parse `AF_BENCH_EVAL_LABELS`. Unknown values fall back to the default
95 + /// rather than erroring, but say so.
96 + pub(crate) fn from_env() -> Self {
97 + match std::env::var("AF_BENCH_EVAL_LABELS").as_deref() {
98 + Ok("instrument") => Self::Instrument,
99 + Ok("family-tom-split") => Self::FamilyTomSplit,
100 + Ok("family") | Err(_) => Self::Family,
101 + Ok(other) => {
102 + eprintln!(
103 + "AF_BENCH_EVAL_LABELS={other} is not one of instrument, family, \
104 + family-tom-split; using family"
105 + );
106 + Self::Family
107 + }
108 + }
109 + }
110 +
111 + pub(crate) fn describe(self) -> &'static str {
112 + match self {
113 + Self::Instrument => "instrument (retired resolution, kept reproducible)",
114 + Self::Family => "coarse family, tom folded into low",
115 + Self::FamilyTomSplit => "coarse family, tom held out as its own class",
116 + }
117 + }
118 + }
119 +
120 + /// Project one corpus tag onto the evaluation's label space.
121 + ///
122 + /// `None` drops the row from the run entirely, train and test both. There is
123 + /// exactly one such case and it is deliberate: see [`DROPPED_NOTE`].
124 + pub(crate) fn project(space: LabelSpace, corpus_tag: &str) -> Option<&'static str> {
125 + if space == LabelSpace::Instrument {
126 + // Corpus tags are already instrument tags, so this is the identity. The
127 + // table is still walked, for the `'static` copy the caller needs and to
128 + // reject a tag the corpus never produced.
129 + return CORPUS
130 + .iter()
131 + .find(|(t, _, _)| *t == corpus_tag)
132 + .map(|(t, _, _)| *t);
133 + }
134 + let (_, family, split) = CORPUS.iter().find(|(t, _, _)| *t == corpus_tag)?;
135 + match (space, split) {
136 + (LabelSpace::FamilyTomSplit, Some(s)) => Some(s),
137 + _ => *family,
138 + }
139 + }
140 +
141 + /// Corpus tag -> (family, class when tom is split out).
142 + ///
143 + /// A `None` family is a folder with no honest family label. `percussion` is the
144 + /// only one and it is 167 files, 16% of the corpus. `af-coarse-families` measured
145 + /// it 47 low / 120 bright, 13% coherence across 19 perceptual bins, and
146 + /// deliberately gives it no family: its members belong in `low` or `drum-bright`
147 + /// by register. That per-file split cannot be made here. The only register signal
148 + /// available is spectral centroid, which is one of the 35 features the layer
149 + /// scores on, so labelling ground truth with it would grade the classifier against
150 + /// its own input and report a class it cannot miss. Excluded, loudly, rather than
151 + /// guessed.
152 + type CorpusRow = (&'static str, Option<&'static str>, Option<&'static str>);
153 + const CORPUS: &[CorpusRow] = &[
154 + ("instrument.drum.kick", Some("family.low"), None),
155 + (
156 + "instrument.drum.tom",
157 + Some("family.low"),
158 + Some(TOM_PROVISIONAL),
159 + ),
160 + ("instrument.drum.snare", Some("family.drum-bright"), None),
161 + ("instrument.drum.clap", Some("family.drum-bright"), None),
162 + ("instrument.drum.hihat", Some("family.drum-bright"), None),
163 + ("instrument.drum.cymbal", Some("family.drum-bright"), None),
164 + ("instrument.percussion", None, None),
165 + ];
166 +
167 + /// Printed whenever a projection drops rows, so the exclusion is never silent.
168 + pub(crate) const DROPPED_NOTE: &str =
169 + " Percussion has no honest family label: it measures 47 low / 120 bright and 13%
170 + coherence across 19 perceptual bins, and the only per-file register signal
171 + available is a feature the layer already scores on. Splitting it by centroid
172 + would grade the classifier against its own input. Dropped from train and test
173 + both; texture is its likely home once Phase 3 has material to check against.";
174 +
175 + /// Report label for a tag in this space.
176 + pub(crate) fn label_for(space: LabelSpace, tag: &str) -> &str {
177 + if space == LabelSpace::Instrument {
178 + return crate::labelled::label_for_tag(tag);
179 + }
180 + if tag == TOM_PROVISIONAL {
181 + return "tom-prov";
182 + }
183 + FAMILIES
184 + .iter()
185 + .find(|f| f.tag == tag)
186 + .map_or(tag, |f| f.label)
187 + }
188 +
189 + /// Which families the corpus can say anything at all about, for the verdict.
190 + pub(crate) fn covered_families(present: &[String]) -> (Vec<&'static str>, Vec<&'static str>) {
191 + let covered: Vec<&'static str> = FAMILIES
192 + .iter()
193 + .filter(|f| present.iter().any(|p| p == f.tag))
194 + .map(|f| f.label)
195 + .collect();
196 + let uncovered: Vec<&'static str> = FAMILIES
197 + .iter()
198 + .filter(|f| !present.iter().any(|p| p == f.tag))
199 + .map(|f| f.label)
200 + .collect();
201 + (covered, uncovered)
202 + }
203 +
204 + #[cfg(test)]
205 + mod tests {
206 + use super::*;
207 +
208 + #[test]
209 + fn drums_project_onto_two_families() {
210 + assert_eq!(
211 + project(LabelSpace::Family, "instrument.drum.kick"),
212 + Some("family.low")
213 + );
214 + assert_eq!(
215 + project(LabelSpace::Family, "instrument.drum.tom"),
216 + Some("family.low")
217 + );
218 + for bright in [
219 + "instrument.drum.snare",
220 + "instrument.drum.clap",
221 + "instrument.drum.hihat",
222 + "instrument.drum.cymbal",
223 + ] {
224 + assert_eq!(
225 + project(LabelSpace::Family, bright),
226 + Some("family.drum-bright"),
227 + "{bright}"
228 + );
229 + }
230 + }
231 +
232 + #[test]
233 + fn percussion_is_dropped_not_guessed() {
234 + // The class the note calls genuinely both. Any Some() here would be a
235 + // ground-truth label invented from the classifier's own input.
236 + assert_eq!(project(LabelSpace::Family, "instrument.percussion"), None);
237 + assert_eq!(
238 + project(LabelSpace::FamilyTomSplit, "instrument.percussion"),
239 + None
240 + );
241 + assert_eq!(
242 + project(LabelSpace::Instrument, "instrument.percussion"),
243 + Some("instrument.percussion")
244 + );
245 + }
246 +
247 + #[test]
248 + fn the_tom_split_moves_only_tom() {
249 + assert_eq!(
250 + project(LabelSpace::FamilyTomSplit, "instrument.drum.tom"),
251 + Some(TOM_PROVISIONAL)
252 + );
253 + assert_eq!(
254 + project(LabelSpace::FamilyTomSplit, "instrument.drum.kick"),
255 + Some("family.low")
256 + );
257 + }
258 +
259 + #[test]
260 + fn instrument_space_is_the_identity_on_corpus_tags() {
261 + for (tag, _, _) in CORPUS {
262 + assert_eq!(project(LabelSpace::Instrument, tag), Some(*tag));
263 + }
264 + }
265 +
266 + #[test]
267 + fn an_unknown_tag_projects_nowhere() {
268 + assert_eq!(project(LabelSpace::Family, "instrument.bass"), None);
269 + assert_eq!(project(LabelSpace::Instrument, "instrument.bass"), None);
270 + }
271 +
272 + #[test]
273 + fn every_corpus_family_is_a_real_family() {
274 + for (tag, family, _) in CORPUS {
275 + if let Some(f) = family {
276 + assert!(
277 + FAMILIES.iter().any(|x| x.tag == *f),
278 + "{tag} maps to {f}, which is not a family"
279 + );
280 + }
281 + }
282 + }
283 +
284 + #[test]
285 + fn coverage_names_the_five_families_the_corpus_cannot_reach() {
286 + let present = vec!["family.low".to_string(), "family.drum-bright".to_string()];
287 + let (covered, uncovered) = covered_families(&present);
288 + assert_eq!(covered, vec!["low", "drum-bright"]);
289 + assert_eq!(
290 + uncovered,
291 + vec!["bass", "tonal", "vocal", "texture", "music"]
292 + );
293 + }
294 +
295 + #[test]
296 + fn labels_shorten_for_report_tables() {
297 + assert_eq!(
298 + label_for(LabelSpace::Family, "family.drum-bright"),
299 + "drum-bright"
300 + );
301 + assert_eq!(
302 + label_for(LabelSpace::FamilyTomSplit, TOM_PROVISIONAL),
303 + "tom-prov"
304 + );
305 + assert_eq!(
306 + label_for(LabelSpace::Instrument, "instrument.drum.kick"),
307 + "kick"
308 + );
309 + }
310 + }