Skip to main content

max / audiofiles

Calibrate the layer eval per class instead of at one global threshold Run 1 graded all seven classes at the shipped 0.85 auto threshold and read the failure as the layer being weak. A score is a share of the k=15 neighbourhood, so what a class can reach depends on how many of its own members sit inside a fixed k: hi-hat and snare are equally separable by top-1 and differ ninefold in recall at 0.85. The threshold was doing most of the failing, and 0.85 was never measured against anything. The gate now asks what threshold each class needs to hold 95% precision, and what recall that buys. Thresholds are calibrated on the folds a sample is not in, and qualify on a Wilson lower bound rather than an observed ratio: the first calibrated run picked the point that just cleared the bar and every class then undershot it held out, which is selection bias with a known direction, not bad luck. Adds a k sweep, per-threshold precision and recall tables, and keeps the shipped-defaults table since that is the status quo being decided against. Result: kick, snare and tom calibrate to 97-99% held-out precision at 43-81% recall. Clap, cymbal, hi-hat and percussion have no qualifying threshold at any k in 5..50.
Author: Max Johnson <me@maxj.phd> · 2026-08-07 02:44 UTC
Signed with PGP, not checked
Commit: 50fe541cd06d616bf9200e4982f6925b5c3d1236
Parent: 02395cb
4 files changed, +666 insertions, -189 deletions
M docs/ml_classifier.md +38 -11
@@ -182,20 +182,47 @@
182 182 | percussion | 46.7% | never fires | 0.0% | 28.1% |
183 183 | macro | 73.1% | 84.5% | 28.8% | 61.1% |
184 184
185 - The gate fails, and it fails on recall alone. Every class the layer auto-applies it
186 - auto-applies correctly (macro precision 84.5%, four classes at 100%), but only kick,
187 - snare and tom reach the 0.85 auto threshold often enough to be useful, and percussion
188 - never reaches it at all. The k-NN is conservative rather than wrong.
189 -
190 185 Read against the retired threshold tree, this is a different result: 73.1% macro top-1
191 186 against that tree's 33.4% strict, with no unreachable class. Instrument identity is
192 - more present in the 35-feature vector under k-NN than it was under thresholds. It is
193 - still not present enough for a default that writes tags unasked, which is the
194 - distinction the auto and review columns above draw.
187 + more present in the 35-feature vector under k-NN than it was under thresholds.
195 188
196 - Percussion is the class to read first. It is a catch-all (cowbell, clave, maraca,
197 - bongo, conga) rather than one instrument, and the confusion matrix scatters it across
198 - every other row. A taxonomy fix, not a tuning problem.
189 + That table grades every class at one global threshold, which turned out to be the
190 + wrong question. A score is the share of the k=15 neighbourhood's kernel weight
191 + carrying a tag, so 0.85 asks for roughly 13 of 15 neighbours to agree, and whether a
192 + class can reach that depends on how many of its own members sit inside a fixed `k`.
193 + Hi-hat (109 files) and snare (188) are equally separable by top-1, 71.6% against
194 + 76.6%, and differ ninefold in recall at 0.85. The score behaves sensibly inside a
195 + class and is not comparable across classes. Nor is 0.85 itself measured: it predates
196 + any evaluation of this layer.
197 +
198 + ### The per-class answer
199 +
200 + So the harness calibrates a threshold per class instead: the most permissive one at
201 + which the class is 95%-confident (Wilson lower bound) of holding 95% precision. Those
202 + are `tag_policy` rows, which the layer format can already carry. Thresholds are fitted
203 + on the folds a sample is not in, so nothing picks its operating point from the data it
204 + is graded on.
205 +
206 + | | threshold | held-out precision | held-out recall |
207 + |---|---|---|---|
208 + | tom | 0.759 | 97.5% | 80.9% |
209 + | kick | 0.932 | 99.1% | 50.7% |
210 + | snare | 0.883 | 98.9% | 43.1% |
211 + | hi-hat, clap, cymbal, percussion | none exists | | |
212 +
213 + **Four of the seven classes cannot be auto-applied at 95% precision at any threshold,
214 + and sweeping `k` over 5, 10, 15, 25 and 50 does not rescue them.** Smaller `k` does
215 + improve separability (top-1 macro rises to 80.7% at k=5), so it is worth knowing for
216 + ranking and "more like this", but it does not convert into a shippable auto-apply
217 + policy: the count of uncalibratable classes is four at every `k` from 5 to 15, and
218 + worse above.
219 +
220 + What that leaves is a three-class layer. Kick, snare and tom clear the bar with 43% to
221 + 81% recall, which is a real head start on the classes that dominate a drum library.
222 +
223 + Percussion is a taxonomy problem rather than a tuning one. It is a catch-all (cowbell,
224 + clave, maraca, bongo, conga), and the confusion matrix scatters it across every other
225 + row at 46.7% top-1. No threshold fixes a class that is not one sound.
199 226
200 227 ## Feature vector
201 228
M crates/audiofiles-bench/src/layer_eval.rs +181 -177
@@ -5,25 +5,46 @@
5 5 //! had no number behind it and the layer sits behind an off-by-default feature.
6 6 //!
7 7 //! What it measures: stratified k-fold cross-validation of [`exemplar`] over the
8 - //! same corpus the layer is built from, at the same `k` and the same thresholds
9 - //! the app uses at runtime. Each fold builds a real [`ExemplarIndex`] from the
10 - //! other folds and scores this fold's samples against it, so no sample is ever a
11 - //! neighbour of itself and the standardization params are fitted on training data
12 - //! alone.
8 + //! same corpus the layer is built from, at the same `k` the app uses at runtime.
9 + //! Each fold builds a real `ExemplarIndex` from the other folds and scores this
10 + //! fold's samples against it, so no sample is ever a neighbour of itself and the
11 + //! standardization params are fitted on training data alone.
13 12 //!
14 13 //! Why it is not one accuracy figure: the retired threshold classifier scored
15 14 //! 33.4% strict with two of its seven classes unreachable by any rule, and a
16 - //! single number is exactly what hid that. Everything here is per class. A class
17 - //! the layer never predicts shows up as a zero row in the confusion matrix and a
18 - //! zero recall, both of which a headline average would smooth away.
15 + //! single number is exactly what hid that. Everything here is per class.
19 16 //!
20 - //! The ship gate ([`GATE`]) is written down in this file rather than decided
21 - //! after reading a run. That ordering is the whole point: a threshold chosen once
22 - //! the number is on screen is a threshold the number chose.
17 + //! # The gate changed after the first run, and that needs saying out loud
18 + //!
19 + //! Run 1 (2026-08-06, `02395cb`) graded every class on precision and recall at
20 + //! [`DEFAULT_AUTO_THRESHOLD`], the global 0.85 the app ships. It failed: four of
21 + //! seven classes had auto recall under 7%, percussion never fired at all.
22 + //!
23 + //! Reading the run said the threshold was doing most of the failing. A score is
24 + //! the share of the k=15 neighbourhood's kernel weight carrying a tag, so what a
25 + //! class can reach depends on how many of its own members sit inside a fixed `k`.
26 + //! Hi-hat (109 files) and snare (188) are equally separable by top-1 (71.6%
27 + //! against 76.6%) and differ ninefold in recall at 0.85. And 0.85 is itself an
28 + //! unvalidated constant: it predates any measurement of this layer.
29 + //!
30 + //! So the gate below asks a different question, not an easier one. Old: "at the
31 + //! threshold we happen to ship, is each class good enough?" New: "at the
32 + //! precision we actually require, what threshold does each class need, and is the
33 + //! recall there worth shipping?" The precision bar went **up**, 0.80 to 0.95,
34 + //! because that is what auto-applying a tag into someone's library unasked
35 + //! deserves; the per-class threshold is what stops class size being graded as if
36 + //! it were quality. Run 1's numbers stay in `docs/ml_classifier.md` so the change
37 + //! is auditable rather than quietly overwritten.
38 + //!
39 + //! The output still reports the shipped defaults, because "what happens if this
40 + //! ships unchanged" remains a real question with a bad answer.
23 41 //!
24 42 //! Usage: `cargo run --release -p audiofiles-bench -- layer-eval`
25 43 //! Env: `AF_BENCH_CORPUS`, `AF_BENCH_VAULT`, `AF_BENCH_EVAL_FOLDS` (default 5),
44 + //! `AF_BENCH_EVAL_K` (comma-separated sweep, default `5,10,15,25,50`),
26 45 //! `AF_BENCH_JSON`.
46 + //!
47 + //! [`DEFAULT_AUTO_THRESHOLD`]: audiofiles_core::analysis::exemplar::DEFAULT_AUTO_THRESHOLD
27 48
28 49 use std::collections::{BTreeMap, BTreeSet, HashMap};
29 50 use std::path::Path;
@@ -35,6 +56,7 @@
35 56 use audiofiles_core::analysis::features::{FEATURE_VERSION, NUM_FEATURES};
36 57 use audiofiles_core::db::Database;
37 58
59 + use crate::calibration::{self, Counts, Point};
38 60 use crate::labelled::{self, label_for_tag};
39 61 use crate::report::Report;
40 62
@@ -47,33 +69,41 @@
47 69 /// files) with single-digit test sets whose per-class recall moves in 10% steps.
48 70 const DEFAULT_FOLDS: usize = 5;
49 71
50 - /// The ship gate, decided before the first run.
72 + /// Neighbour counts to sweep.
51 73 ///
52 - /// A bundled default that is confidently wrong is worse than no default, so the
53 - /// binding criterion is precision at the auto-apply threshold: that is the score
54 - /// at which the layer writes a tag into someone's library without being asked.
55 - /// Recall is graded far more loosely, because a tag the layer declines to apply
56 - /// costs a user nothing they did not already have.
57 - ///
58 - /// The zero-prediction check is separate from recall on purpose. A class the
59 - /// layer never reaches is a different failure from a class it reaches badly: it
60 - /// means the shipped taxonomy promises something the data cannot deliver, and it
61 - /// is what the retired threshold tree did to clap and tom.
74 + /// `k` is a global constant at runtime and the score is a share of it, so it is
75 + /// the lever that decides whether a small class can reach any threshold at all.
76 + /// Sweeping it costs nothing (the corpus is imported and analysed once, and only
77 + /// the scoring repeats) and it is the difference between "this class is hard" and
78 + /// "this class is outnumbered inside a window we chose".
79 + const DEFAULT_K_SWEEP: &[usize] = &[5, 10, 15, 25, 50];
80 +
81 + /// Score thresholds the sweep tables report, spanning both shipped defaults.
82 + const SWEEP_THRESHOLDS: &[f64] = &[0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9];
83 +
84 + /// The ship gate. See the module header for why it is not run 1's gate.
62 85 struct Gate {
63 - /// Per-class precision at [`DEFAULT_AUTO_THRESHOLD`], over classes the layer
64 - /// predicts at all. A wrong auto-applied tag is the harm this layer can do.
65 - auto_precision: f64,
66 - /// Per-class recall at [`DEFAULT_AUTO_THRESHOLD`]. Deliberately low: a
67 - /// default that fires on a third of a class is still a head start.
68 - auto_recall: f64,
69 - /// Macro-averaged top-1 recall. Below this the feature space is not
70 - /// separating these classes at all, whatever the thresholds do.
86 + /// Precision each class must reach for the layer to auto-apply it, as a 95%
87 + /// lower bound rather than an observed ratio (see [`calibration`]). A tag
88 + /// written into a library unasked should be right 19 times in 20.
89 + target_precision: f64,
90 + /// Recall at that precision, below which a per-class threshold is not worth
91 + /// shipping: the tag fires too rarely to be a head start.
92 + min_recall: f64,
93 + /// Floor under the confidence bound. The bound already rejects a precision
94 + /// claimed off a handful of predictions (run 1 read cymbal as 100% precise
95 + /// on two); this stops a policy shipping off a thin sample regardless.
96 + min_support: usize,
97 + /// Macro-averaged top-1 recall: the threshold-free separability floor. Below
98 + /// this the feature space is not distinguishing these classes at all and no
99 + /// per-class calibration rescues it.
71 100 top1_macro_recall: f64,
72 101 }
73 102
74 103 const GATE: Gate = Gate {
75 - auto_precision: 0.80,
76 - auto_recall: 0.35,
104 + target_precision: 0.95,
105 + min_recall: 0.40,
106 + min_support: 15,
77 107 top1_macro_recall: 0.60,
78 108 };
79 109
@@ -89,27 +119,16 @@
89 119 truth: Option<String>,
90 120 }
91 121
92 - /// Per-class counts at one score threshold, over the whole cross-validation.
93 - #[derive(Default, Clone, Copy)]
94 - struct Counts {
95 - tp: usize,
96 - fp: usize,
97 - fn_: usize,
98 - }
99 -
100 - impl Counts {
101 - fn precision(self) -> Option<f64> {
102 - let predicted = self.tp + self.fp;
103 - (predicted > 0).then(|| self.tp as f64 / predicted as f64)
104 - }
105 - fn recall(self) -> Option<f64> {
106 - let actual = self.tp + self.fn_;
107 - (actual > 0).then(|| self.tp as f64 / actual as f64)
108 - }
109 - fn f1(self) -> Option<f64> {
110 - let (p, r) = (self.precision()?, self.recall()?);
111 - (p + r > 0.0).then(|| 2.0 * p * r / (p + r))
112 - }
122 + /// What one test sample produced under one `k`.
123 + struct Prediction {
124 + truth: String,
125 + /// Highest-scoring tag, or `None` when the index returned nothing.
126 + top1: Option<String>,
127 + /// Score per class, for threshold sweeps.
128 + scores: BTreeMap<String, f64>,
129 + /// The fold this sample was held out of, so a threshold is never calibrated
130 + /// on the predictions it is graded against.
131 + fold: usize,
113 132 }
114 133
115 134 fn pct(v: Option<f64>) -> String {
@@ -228,38 +247,55 @@
228 247 Ok(db)
229 248 }
230 249
231 - /// What one test sample produced.
232 - struct Prediction {
233 - truth: String,
234 - /// Highest-scoring tag, or `None` when the index returned nothing.
235 - top1: Option<String>,
236 - /// Score per class, for threshold sweeps.
237 - scores: BTreeMap<String, f64>,
250 + /// Every test sample's evidence for one class. A class absent from a sample's
251 + /// scores scored zero for it, which is a real observation and not a gap.
252 + fn class_points(predictions: &[Prediction], class: &str) -> Vec<Point> {
253 + predictions
254 + .iter()
255 + .map(|p| Point {
256 + score: p.scores.get(class).copied().unwrap_or(0.0),
257 + actual: p.truth == class,
258 + fold: p.fold,
259 + })
260 + .collect()
238 261 }
239 262
240 - pub(crate) fn run(corpus: &Path, vault: &Path, config: &AnalysisConfig, folds: usize) {
263 + pub(crate) fn run(
264 + corpus: &Path,
265 + vault: &Path,
266 + config: &AnalysisConfig,
267 + folds: usize,
268 + k_sweep: &[usize],
269 + ) {
241 270 println!("━━━ CLASSIFIER LAYER EVALUATION ━━━");
242 271 println!();
243 272 println!(" corpus {}", corpus.display());
244 273 println!(" scratch {}", vault.display());
245 274 println!(" features v{FEATURE_VERSION}");
246 - println!(" k {DEFAULT_K} (runtime default)");
275 + println!(" k {DEFAULT_K} (runtime default), sweeping {k_sweep:?}");
247 276 println!(" folds {folds}, stratified by class");
248 277 println!();
249 - println!(" Gate, fixed before this run:");
278 + println!(" Gate:");
250 279 println!(
251 - " per-class precision at the auto threshold ({DEFAULT_AUTO_THRESHOLD:.2}) >= {:.0}%",
252 - GATE.auto_precision * 100.0
280 + " per-class precision, 95%-confident, at a per-class threshold >= {:.0}%",
281 + GATE.target_precision * 100.0
253 282 );
254 283 println!(
255 - " per-class recall at the auto threshold >= {:.0}%",
256 - GATE.auto_recall * 100.0
284 + " per-class recall at that threshold >= {:.0}%",
285 + GATE.min_recall * 100.0
257 286 );
258 287 println!(
259 - " macro-averaged top-1 recall >= {:.0}%",
288 + " predictions behind that precision >= {}",
289 + GATE.min_support
290 + );
291 + println!(
292 + " macro-averaged top-1 recall >= {:.0}%",
260 293 GATE.top1_macro_recall * 100.0
261 294 );
262 - println!(" no class the layer never predicts");
295 + println!(" every class calibratable, in every fold");
296 + println!();
297 + println!(" Thresholds are calibrated on the folds a sample is NOT in, so no");
298 + println!(" class picks its operating point from the data it is graded on.");
263 299 println!();
264 300
265 301 let built = match labelled::build_vault(corpus, vault, config) {
@@ -298,8 +334,10 @@
298 334 let classes: Vec<String> = classes.into_iter().collect();
299 335 let fold_of = assign_folds(&rows, folds);
300 336
301 - // Cross-validation
302 - let mut predictions: Vec<Prediction> = Vec::with_capacity(testable);
337 + // Cross-validation. The index is built once per fold and scored at every `k`,
338 + // because building it is the expensive half and `k` only enters at scoring.
339 + let mut by_k: BTreeMap<usize, Vec<Prediction>> =
340 + k_sweep.iter().map(|k| (*k, Vec::new())).collect();
303 341 for fold in 0..folds {
304 342 let train: Vec<&Row> = rows
305 343 .iter()
@@ -335,16 +373,19 @@
335 373 );
336 374
337 375 for row in test {
338 - // No `exclude_hash`: the row is not in this index at all, which is
339 - // the property the fold split exists to give.
340 - let scored = index.score(&row.vector, DEFAULT_K, None);
341 - let top1 = scored.first().map(|s| s.tag.clone());
342 - let scores = scored.into_iter().map(|s| (s.tag, s.score)).collect();
343 - predictions.push(Prediction {
344 - truth: row.truth.clone().unwrap_or_default(),
345 - top1,
346 - scores,
347 - });
376 + for &k in k_sweep {
377 + // No `exclude_hash`: the row is not in this index at all, which
378 + // is the property the fold split exists to give.
379 + let scored = index.score(&row.vector, k, None);
380 + let top1 = scored.first().map(|s| s.tag.clone());
381 + let scores = scored.into_iter().map(|s| (s.tag, s.score)).collect();
382 + by_k.entry(k).or_default().push(Prediction {
383 + truth: row.truth.clone().unwrap_or_default(),
384 + top1,
385 + scores,
386 + fold,
387 + });
388 + }
348 389 }
349 390 }
350 391 println!();
@@ -354,28 +395,25 @@
354 395 report.set("k", DEFAULT_K);
355 396 report.set("feat_version", FEATURE_VERSION);
356 397 report.set("exemplars_total", rows.len());
357 - report.set("tested", predictions.len());
358 398 report.set("ambiguous_excluded", ambiguous);
399 + report.set("gate_target_precision", GATE.target_precision);
400 + report.set("gate_min_recall", GATE.min_recall);
401 + report.set("gate_min_support", GATE.min_support);
402 + report.set("gate_top1_macro_recall", GATE.top1_macro_recall);
359 403
360 - let top1 = print_confusion(&predictions, &classes, &mut report);
361 - let auto = print_threshold_table(
362 - &predictions,
363 - &classes,
364 - DEFAULT_AUTO_THRESHOLD,
365 - "auto-apply",
366 - "auto",
367 - &mut report,
368 - );
369 - let _review = print_threshold_table(
370 - &predictions,
371 - &classes,
372 - DEFAULT_REVIEW_THRESHOLD,
373 - "review",
374 - "review",
375 - &mut report,
376 - );
404 + let default_k = by_k
405 + .get(&DEFAULT_K)
406 + .expect("the sweep always contains the runtime k");
407 + report.set("tested", default_k.len());
377 408
378 - print_verdict(&classes, &top1, &auto, &mut report);
409 + let top1 = print_confusion(default_k, &classes, &mut report);
410 + print_shipped_defaults(default_k, &classes, &mut report);
411 + print_threshold_sweep(default_k, &classes);
412 + let calibrated = print_calibration(default_k, &classes, folds, &mut report);
413 + if k_sweep.len() > 1 {
414 + print_k_sweep(&by_k, &classes, folds, &mut report);
415 + }
416 + print_verdict(&classes, &top1, &calibrated, &mut report);
379 417 report.write();
380 418 }
381 419
@@ -385,10 +423,11 @@
385 423 classes: &[String],
386 424 report: &mut Report,
387 425 ) -> BTreeMap<String, Counts> {
388 - println!("━━━ TOP-1 CONFUSION ━━━");
426 + println!("━━━ TOP-1 CONFUSION (k = {DEFAULT_K}) ━━━");
389 427 println!();
390 428 println!(" Rows are the corpus label, columns the highest-scoring tag.");
391 - println!(" `(none)` is a sample the index scored nothing for at all.");
429 + println!(" Threshold-free: this is what the layer would say if forced to pick,");
430 + println!(" so it measures separability rather than any policy over it.");
392 431 println!();
393 432
394 433 let width = classes
@@ -455,10 +494,6 @@
455 494 let micro = counts.values().map(|c| c.tp).sum::<usize>() as f64 / predictions.len() as f64;
456 495 println!(" macro-averaged recall {}", pct(macro_recall));
457 496 println!(" overall top-1 accuracy {}", pct(Some(micro)));
458 - println!();
459 - println!(" The macro figure is the one that matters: it weights the 48-file");
460 - println!(" clap class the same as the 246-file tom class, so a layer that");
461 - println!(" gets the big classes right and ignores a small one cannot hide.");
462 497 if never_predicted.is_empty() {
463 498 println!(" Every class is predicted at least once.");
464 499 } else {
@@ -471,126 +506,318 @@
471 506 println!();
472 507
473 508 if let Some(m) = macro_recall {
474 - report.set("top1_macro_recall", (m * 10000.0).round() / 10000.0);
509 + report.set("top1_macro_recall", round4(m));
475 510 }
476 - report.set("top1_accuracy", (micro * 10000.0).round() / 10000.0);
511 + report.set("top1_accuracy", round4(micro));
477 512 report.set("never_predicted", never_predicted.len());
478 513 for (tag, c) in &counts {
479 514 let label = label_for_tag(tag);
480 515 if let Some(r) = c.recall() {
481 - report.set(
482 - &format!("top1_{label}_recall"),
483 - (r * 10000.0).round() / 10000.0,
484 - );
516 + report.set(&format!("top1_{label}_recall"), round4(r));
485 517 }
486 518 if let Some(p) = c.precision() {
487 - report.set(
488 - &format!("top1_{label}_precision"),
489 - (p * 10000.0).round() / 10000.0,
490 - );
519 + report.set(&format!("top1_{label}_precision"), round4(p));
491 520 }
492 521 }
493 522 counts
494 523 }
495 524
496 - /// Per-class precision/recall/F1 at one score threshold, scored multi-label:
497 - /// every tag at or above the threshold is a prediction, so a sample can be right
498 - /// about one class and wrong about another in the same breath. That is how the
499 - /// layer behaves at runtime, and a top-1 view would not show a layer that applies
500 - /// three tags to everything.
501 - fn print_threshold_table(
502 - predictions: &[Prediction],
503 - classes: &[String],
504 - threshold: f64,
505 - title: &str,
506 - key: &str,
507 - report: &mut Report,
508 - ) -> BTreeMap<String, Counts> {
509 - println!(
510 - "━━━ AT THE {} THRESHOLD ({threshold:.2}) ━━━",
511 - title.to_uppercase()
512 - );
525 + /// What the layer does today, unchanged: one global auto threshold for every
526 + /// class. Kept because it is the status quo the ship decision is against.
527 + fn print_shipped_defaults(predictions: &[Prediction], classes: &[String], report: &mut Report) {
528 + println!("━━━ AT THE SHIPPED DEFAULTS (one global threshold) ━━━");
513 529 println!();
514 530
515 531 let mut counts: BTreeMap<String, Counts> = BTreeMap::new();
516 532 for class in classes {
517 - let mut c = Counts::default();
518 - for p in predictions {
519 - let predicted = p.scores.get(class).is_some_and(|s| *s >= threshold);
520 - let actual = &p.truth == class;
521 - match (predicted, actual) {
522 - (true, true) => c.tp += 1,
523 - (true, false) => c.fp += 1,
524 - (false, true) => c.fn_ += 1,
525 - (false, false) => {}
526 - }
527 - }
528 - counts.insert(class.clone(), c);
533 + counts.insert(
534 + class.clone(),
535 + calibration::counts_at(&class_points(predictions, class), DEFAULT_AUTO_THRESHOLD),
536 + );
529 537 }
530 538
531 539 println!(
532 - " {:<12} {:>7} {:>8} {:>10} {:>9} {:>8}",
533 - "class", "n", "fired", "precision", "recall", "F1"
540 + " {:<12} {:>7} {:>8} {:>10} {:>9} auto {DEFAULT_AUTO_THRESHOLD:.2} / review {DEFAULT_REVIEW_THRESHOLD:.2}",
541 + "class", "n", "fired", "precision", "recall"
534 542 );
535 - println!(" {}", "─".repeat(58));
543 + println!(" {}", "─".repeat(50));
536 544 for class in classes {
537 545 let c = counts[class];
538 546 println!(
539 - " {:<12} {:>7} {:>8} {:>10} {:>9} {:>8}",
547 + " {:<12} {:>7} {:>8} {:>10} {:>9}",
540 548 label_for_tag(class),
541 - c.tp + c.fn_,
542 - c.tp + c.fp,
549 + c.actual(),
550 + c.fired(),
543 551 pct(c.precision()),
544 552 pct(c.recall()),
545 - pct(c.f1()),
546 553 );
547 554 }
548 - println!(" {}", "─".repeat(58));
555 + println!(" {}", "─".repeat(50));
549 556 println!(
550 - " {:<12} {:>7} {:>8} {:>10} {:>9} {:>8}",
557 + " {:<12} {:>7} {:>8} {:>10} {:>9}",
551 558 "macro",
552 559 predictions.len(),
553 - counts.values().map(|c| c.tp + c.fp).sum::<usize>(),
560 + counts.values().map(|c| c.fired()).sum::<usize>(),
554 561 pct(macro_average(classes, &counts, Counts::precision)),
555 562 pct(macro_average(classes, &counts, Counts::recall)),
556 - pct(macro_average(classes, &counts, Counts::f1)),
557 563 );
558 564 println!();
559 -
560 - // Tags per sample, which is what a user sees. A layer with fine per-class
561 - // numbers that puts four tags on every one-shot is still not shippable.
562 - let fired: usize = counts.values().map(|c| c.tp + c.fp).sum();
565 + let silent = predictions
566 + .iter()
567 + .filter(|p| !p.scores.values().any(|s| *s >= DEFAULT_AUTO_THRESHOLD))
568 + .count();
563 569 println!(
564 - " {:.2} tag(s) per sample on average; {} sample(s) got nothing.",
565 - fired as f64 / predictions.len() as f64,
566 - predictions
567 - .iter()
568 - .filter(|p| !p.scores.values().any(|s| *s >= threshold))
569 - .count()
570 + " {silent} of {} samples ({:.0}%) get no tag at all.",
571 + predictions.len(),
572 + silent as f64 / predictions.len() as f64 * 100.0
570 573 );
571 574 println!();
572 575
573 - report.set(
574 - &format!("{key}_tags_per_sample"),
575 - (fired as f64 / predictions.len() as f64 * 100.0).round() / 100.0,
576 - );
577 576 for (tag, c) in &counts {
578 577 let label = label_for_tag(tag);
579 578 if let Some(p) = c.precision() {
580 - report.set(
581 - &format!("{key}_{label}_precision"),
582 - (p * 10000.0).round() / 10000.0,
583 - );
579 + report.set(&format!("shipped_{label}_precision"), round4(p));
584 580 }
585 581 if let Some(r) = c.recall() {
586 - report.set(
587 - &format!("{key}_{label}_recall"),
588 - (r * 10000.0).round() / 10000.0,
582 + report.set(&format!("shipped_{label}_recall"), round4(r));
583 + }
584 + }
585 + report.set("shipped_silent_samples", silent);
586 + }
587 +
588 + /// Precision and recall for every class across a range of thresholds.
589 + ///
590 + /// This is the evidence that one global threshold cannot serve seven classes: read
591 + /// down a column and the same number means a different thing in every row.
592 + fn print_threshold_sweep(predictions: &[Prediction], classes: &[String]) {
Lines truncated
M crates/audiofiles-bench/src/main.rs +4 -1
@@ -29,7 +29,8 @@
29 29 //! (FSL10K root for accuracy), `AF_BENCH_BATCH`, `AF_BENCH_LIMIT`,
30 30 //! `AF_BENCH_ANALYZE`, `AF_BENCH_LAYOUT_N`, `AF_BENCH_JSON` (machine-readable
31 31 //! output path), `AF_AFCL_OUT` (where `afcl` writes the layer),
32 - //! `AF_BENCH_EVAL_FOLDS` (cross-validation folds for `layer-eval`),
32 + //! `AF_BENCH_EVAL_FOLDS` and `AF_BENCH_EVAL_K` (folds and the neighbour-count
33 + //! sweep for `layer-eval`),
33 34 //! `AF_BENCH_STAGES` (files per per-stage probe during `ingest`, 0 = off).
34 35 //!
35 36 //! Section 1 times the analysis stages per file, against the corpus and no
@@ -45,6 +46,7 @@
45 46
46 47 mod accuracy;
47 48 mod afcl_gen;
49 + mod calibration;
48 50 mod ingest;
49 51 mod labelled;
50 52 mod layer_eval;
@@ -348,6 +350,7 @@
348 350 &vault,
349 351 &full_pipeline_config(),
350 352 layer_eval::folds_from_env(),
353 + &layer_eval::k_sweep_from_env(),
351 354 );
352 355 return;
353 356 }
A crates/audiofiles-bench/src/calibration.rs +443
@@ -1,0 +1,443 @@
1 + //! Per-class threshold calibration: what operating point does a class support?
2 + //!
3 + //! The first run of `layer-eval` graded all seven classes against one global
4 + //! threshold ([`DEFAULT_AUTO_THRESHOLD`], 0.85) and read the failure as the layer
5 + //! being weak. Most of it was the threshold being wrong for six of the seven
6 + //! classes.
7 + //!
8 + //! A score is the share of the k-nearest neighbourhood's kernel weight carrying a
9 + //! tag, so 0.85 asks for roughly 13 of 15 neighbours to agree. Whether a class
10 + //! can reach that is bounded by how many of its own members fall inside a fixed
11 + //! `k`, which is a property of class size and local density rather than of how
12 + //! distinguishable the sound is. Measured: hi-hat (109 files) and snare (188) sit
13 + //! in the same top-1 band, 71.6% against 76.6%, and their recall at 0.85 differs
14 + //! by a factor of nine. The score is well behaved inside a class and not
15 + //! comparable across classes.
16 + //!
17 + //! So the question a ship gate should ask is not "what does this class do at
18 + //! 0.85". It is "what is the most permissive threshold at which this class still
19 + //! meets the precision we require, and what recall does it buy there". That
20 + //! threshold is exactly a `tag_policy` row, which the layer format can already
21 + //! carry and the export currently declines to ship.
22 + //!
23 + //! [`DEFAULT_AUTO_THRESHOLD`]: audiofiles_core::analysis::exemplar::DEFAULT_AUTO_THRESHOLD
24 +
25 + /// Per-class counts at one score threshold.
26 + #[derive(Default, Clone, Copy)]
27 + pub(crate) struct Counts {
28 + pub(crate) tp: usize,
29 + pub(crate) fp: usize,
30 + pub(crate) fn_: usize,
31 + }
32 +
33 + impl Counts {
34 + /// Of what fired, how much was right. `None` when nothing fired, which is a
35 + /// different fact from firing and being wrong, and the two must not print
36 + /// the same.
37 + pub(crate) fn precision(self) -> Option<f64> {
38 + let predicted = self.tp + self.fp;
39 + (predicted > 0).then(|| self.tp as f64 / predicted as f64)
40 + }
41 +
42 + /// Of what should have fired, how much did.
43 + pub(crate) fn recall(self) -> Option<f64> {
44 + let actual = self.tp + self.fn_;
45 + (actual > 0).then(|| self.tp as f64 / actual as f64)
46 + }
47 +
48 + pub(crate) fn fired(self) -> usize {
49 + self.tp + self.fp
50 + }
51 +
52 + pub(crate) fn actual(self) -> usize {
53 + self.tp + self.fn_
54 + }
55 +
56 + pub(crate) fn add(&mut self, other: Self) {
57 + self.tp += other.tp;
58 + self.fp += other.fp;
59 + self.fn_ += other.fn_;
60 + }
61 + }
62 +
63 + /// One test sample's evidence for one class: what the layer scored it, whether it
64 + /// really is that class, and which fold produced it.
65 + ///
66 + /// The fold travels with the point because calibrating a threshold on the same
67 + /// predictions it is then scored against is threshold-fitting on the test set.
68 + /// See [`out_of_fold`].
69 + #[derive(Clone, Copy)]
70 + pub(crate) struct Point {
71 + pub(crate) score: f64,
72 + pub(crate) actual: bool,
73 + pub(crate) fold: usize,
74 + }
75 +
76 + /// A threshold and what shipping it would deliver.
77 + #[derive(Clone, Copy)]
78 + pub(crate) struct OperatingPoint {
79 + pub(crate) threshold: f64,
80 + pub(crate) counts: Counts,
81 + }
82 +
83 + /// z for a one-sided 95% lower bound.
84 + const WILSON_Z: f64 = 1.645;
85 +
86 + /// Lower bound of the Wilson score interval for `successes / trials`.
87 + ///
88 + /// Why a bound rather than the ratio: the first calibrated run picked, for each
89 + /// class, the threshold whose *observed* precision just cleared 95%, and held-out
90 + /// precision then came in at 78.6% to 94.9%. Every class undershot. That is not
91 + /// bad luck, it is what selecting the most permissive point that clears a bar
92 + /// does: the point that clears it by the narrowest margin is the one most likely
93 + /// to have cleared it by chance, and picking the extreme of a noisy set is
94 + /// selection bias with a known direction.
95 + ///
96 + /// The bound removes the ad-hoc part of the fix. Instead of "meet 95% and also
97 + /// have at least N predictions", a class must be 95%-confident of being above the
98 + /// bar, which asks for more evidence from a small sample and less from a large
99 + /// one, on the same scale. 19 of 20 correct reads as 76% here; 190 of 200 reads
100 + /// as 92%.
101 + fn wilson_lower_bound(successes: usize, trials: usize) -> f64 {
102 + if trials == 0 {
103 + return 0.0;
104 + }
105 + let n = trials as f64;
106 + let p = successes as f64 / n;
107 + let z2 = WILSON_Z * WILSON_Z;
108 + let denom = 1.0 + z2 / n;
109 + let center = p + z2 / (2.0 * n);
110 + let margin = WILSON_Z * (p * (1.0 - p) / n + z2 / (4.0 * n * n)).sqrt();
111 + ((center - margin) / denom).max(0.0)
112 + }
113 +
114 + /// Counts at a fixed threshold. Mirrors the runtime rule: a tag applies when its
115 + /// score is at or above the threshold.
116 + pub(crate) fn counts_at(points: &[Point], threshold: f64) -> Counts {
117 + let mut c = Counts::default();
118 + for p in points {
119 + match (p.score >= threshold, p.actual) {
120 + (true, true) => c.tp += 1,
121 + (true, false) => c.fp += 1,
122 + (false, true) => c.fn_ += 1,
123 + (false, false) => {}
124 + }
125 + }
126 + c
127 + }
128 +
129 + /// The most permissive threshold at which this class is 95%-confident of holding
130 + /// `target_precision`, with at least `min_support` predictions behind it.
131 + ///
132 + /// Most permissive means lowest, because a lower threshold is more recall, and
133 + /// recall is what we are buying once precision is fixed. Walks the score-sorted
134 + /// points once, which visits every threshold that can produce a distinct split.
135 + ///
136 + /// The bar is [`wilson_lower_bound`], not the observed ratio: see there for why
137 + /// the observed ratio systematically overstates what a chosen threshold delivers
138 + /// on data it did not see.
139 + ///
140 + /// Two further guards:
141 + ///
142 + /// - `min_support`. A floor under the bound, so a class cannot ship a policy off
143 + /// a handful of predictions even when the arithmetic allows it.
144 + /// - A threshold of zero is refused. It would fire on every sample including the
145 + /// ones the index scored nothing for, which is not a classifier.
146 + ///
147 + /// Precision is not monotone in the threshold, so a lower qualifying threshold
148 + /// may sit below a stretch that does not qualify. That is fine and deliberate:
149 + /// what ships is a single threshold, and the numbers reported are the ones
150 + /// measured at it.
151 + pub(crate) fn operating_point(
152 + points: &[Point],
153 + target_precision: f64,
154 + min_support: usize,
155 + ) -> Option<OperatingPoint> {
156 + let positives = points.iter().filter(|p| p.actual).count();
157 + if positives == 0 {
158 + return None;
159 + }
160 +
161 + let mut sorted: Vec<&Point> = points.iter().collect();
162 + sorted.sort_by(|a, b| b.score.total_cmp(&a.score));
163 +
164 + let mut tp = 0usize;
165 + let mut fp = 0usize;
166 + let mut best: Option<OperatingPoint> = None;
167 +
168 + let mut i = 0;
169 + while i < sorted.len() {
170 + // Every point sharing this score has to be taken with it: no threshold
171 + // can separate two samples that scored the same.
172 + let score = sorted[i].score;
173 + #[allow(
174 + clippy::float_cmp,
175 + reason = "exact equality is the point: a threshold cannot split two \
176 + samples that scored bit-identically, so the group boundary \
177 + has to be exact rather than within a tolerance"
178 + )]
179 + while i < sorted.len() && sorted[i].score == score {
180 + if sorted[i].actual {
181 + tp += 1;
182 + } else {
183 + fp += 1;
184 + }
185 + i += 1;
186 + }
187 + if score <= 0.0 {
188 + break;
189 + }
190 + let fired = tp + fp;
191 + if fired < min_support {
192 + continue;
193 + }
194 + if wilson_lower_bound(tp, fired) >= target_precision {
195 + // Descending walk, so each qualifying point is more permissive than
196 + // the last. Keep overwriting and the survivor is the lowest.
197 + best = Some(OperatingPoint {
198 + threshold: score,
199 + counts: Counts {
200 + tp,
201 + fp,
202 + fn_: positives - tp,
203 + },
204 + });
205 + }
206 + }
207 + best
208 + }
209 +
210 + /// Calibrate and evaluate without letting a class pick its threshold from the
211 + /// samples it is then graded on.
212 + ///
213 + /// For each fold: choose the threshold from every *other* fold's points, then
214 + /// count this fold's points at it. Summing across folds gives per-class numbers
215 + /// that no threshold was fitted to. A fold whose calibration finds no qualifying
216 + /// threshold contributes its positives as misses, because a policy that cannot be
217 + /// derived is a policy that does not ship and a tag that never fires.
218 + ///
219 + /// Returns the summed counts and the thresholds chosen, one per fold that found
220 + /// one. The spread of those thresholds is worth reading: a class whose threshold
221 + /// swings between folds is one whose operating point is not a stable property of
222 + /// the class.
223 + pub(crate) fn out_of_fold(
224 + points: &[Point],
225 + folds: usize,
226 + target_precision: f64,
227 + min_support: usize,
228 + ) -> (Counts, Vec<f64>) {
229 + let mut total = Counts::default();
230 + let mut thresholds = Vec::new();
231 +
232 + for fold in 0..folds {
233 + let calibration: Vec<Point> = points.iter().filter(|p| p.fold != fold).copied().collect();
234 + let held_out: Vec<Point> = points.iter().filter(|p| p.fold == fold).copied().collect();
235 + if held_out.is_empty() {
236 + continue;
237 + }
238 +
239 + // The calibration set is smaller than the whole, so scale the support
240 + // floor with it. Otherwise a class that clears `min_support` overall
241 + // fails to calibrate on 4/5 of the data for arithmetic reasons.
242 + let scaled_support = (min_support * (folds - 1)).div_ceil(folds).max(1);
243 + match operating_point(&calibration, target_precision, scaled_support) {
244 + Some(op) => {
245 + total.add(counts_at(&held_out, op.threshold));
246 + thresholds.push(op.threshold);
247 + }
248 + None => {
249 + // No threshold: nothing fires, so every positive here is a miss.
250 + total.fn_ += held_out.iter().filter(|p| p.actual).count();
251 + }
252 + }
253 + }
254 + (total, thresholds)
255 + }
256 +
257 + /// Mean, and the spread, of the per-fold thresholds.
258 + pub(crate) fn threshold_spread(thresholds: &[f64]) -> Option<(f64, f64, f64)> {
259 + if thresholds.is_empty() {
260 + return None;
261 + }
262 + let mean = thresholds.iter().sum::<f64>() / thresholds.len() as f64;
263 + let min = thresholds.iter().copied().fold(f64::INFINITY, f64::min);
264 + let max = thresholds.iter().copied().fold(f64::NEG_INFINITY, f64::max);
265 + Some((mean, min, max))
266 + }
267 +
268 + #[cfg(test)]
269 + mod tests {
270 + use super::*;
271 +
272 + fn pts(spec: &[(f64, bool)]) -> Vec<Point> {
273 + spec.iter()
274 + .enumerate()
275 + .map(|(i, &(score, actual))| Point {
276 + score,
277 + actual,
278 + fold: i % 5,
279 + })
280 + .collect()
281 + }
282 +
283 + #[test]
284 + fn counts_at_matches_the_runtime_at_or_above_rule() {
285 + let p = pts(&[(0.9, true), (0.5, true), (0.5, false), (0.1, true)]);
286 + let c = counts_at(&p, 0.5);
287 + assert_eq!((c.tp, c.fp, c.fn_), (2, 1, 1), "0.5 fires at exactly 0.5");
288 + }
289 +
290 + #[test]
291 + fn operating_point_takes_the_most_permissive_qualifying_threshold() {
292 + // Perfectly ordered: every positive above every negative. Asserted as a
293 + // property rather than a magic threshold, because what "most permissive"
294 + // resolves to depends on the bound, the target and the sample size, and
295 + // a hardcoded number would be testing this fixture's arithmetic.
296 + let spec: Vec<(f64, bool)> = (0..100)
297 + .map(|i| (0.9 - f64::from(i) * 0.001, true))
298 + .chain((0..100).map(|i| (0.2 - f64::from(i) * 0.001, false)))
299 + .collect();
300 + let p = pts(&spec);
301 + let op = operating_point(&p, 0.95, 1).unwrap();
302 +
303 + // What it reports is what that threshold actually does.
304 + let at = counts_at(&p, op.threshold);
305 + assert_eq!((at.tp, at.fp), (op.counts.tp, op.counts.fp));
306 + assert!(wilson_lower_bound(at.tp, at.fired()) >= 0.95);
307 +
308 + // And nothing lower qualifies, which is what makes it the most permissive.
309 + for lower in p
310 + .iter()
311 + .map(|x| x.score)
312 + .filter(|s| *s < op.threshold && *s > 0.0)
313 + {
314 + let c = counts_at(&p, lower);
315 + assert!(
316 + wilson_lower_bound(c.tp, c.fired()) < 0.95,
317 + "threshold {lower} also qualifies, so {} was not the lowest",
318 + op.threshold
319 + );
320 + }
321 + }
322 +
323 + #[test]
324 + fn operating_point_refuses_a_threshold_backed_by_too_few_predictions() {
325 + // Two perfect predictions then a mess. The bound alone rejects this:
326 + // 2 of 2 is not evidence of 95%, and the first run read cymbal that way.
327 + let p = pts(&[
328 + (0.99, true),
329 + (0.98, true),
330 + (0.5, false),
331 + (0.5, false),
332 + (0.5, false),
333 + (0.4, true),
334 + ]);
335 + assert!(
336 + operating_point(&p, 0.95, 1).is_none(),
337 + "two perfect predictions are not 95% confidence of 95% precision"
338 + );
339 + }
340 +
341 + #[test]
342 + fn the_bound_demands_more_evidence_from_a_smaller_sample() {
343 + // The property the whole calibration rests on: the same observed ratio
344 + // qualifies at scale and does not qualify on a handful.
345 + assert!(wilson_lower_bound(19, 20) < 0.85);
346 + assert!(wilson_lower_bound(190, 200) > 0.9);
347 + // Monotone in sample size at a fixed ratio.
348 + assert!(wilson_lower_bound(9, 10) < wilson_lower_bound(90, 100));
349 + // Degenerate inputs stay in range rather than producing a NaN that
350 + // would silently compare false against every target.
351 + assert!(wilson_lower_bound(0, 0).abs() < f64::EPSILON);
352 + assert!(wilson_lower_bound(0, 5) >= 0.0);
353 + assert!(wilson_lower_bound(5, 5) <= 1.0);
354 + }
355 +
356 + #[test]
357 + fn operating_point_never_returns_a_zero_threshold() {
358 + // Every sample scores zero: the index had nothing to say. A zero
359 + // threshold would "apply" the tag to all of them.
360 + let spec: Vec<(f64, bool)> = (0..40).map(|i| (0.0, i % 2 == 0)).collect();
361 + assert!(operating_point(&pts(&spec), 0.4, 1).is_none());
362 + }
363 +
364 + #[test]
365 + fn operating_point_is_none_when_precision_is_unreachable() {
366 + let p = pts(&[(0.9, false), (0.8, false), (0.7, true)]);
367 + assert!(operating_point(&p, 0.95, 1).is_none());
368 + }
369 +
370 + #[test]
371 + fn tied_scores_are_taken_together() {
372 + // A threshold cannot split two equal scores, so the walk must not report
373 + // a precision that only holds if it does. A clean run of positives, then
374 + // a tied group of one positive and five negatives: taking the group
375 + // whole fails the bar, and splitting it would pass.
376 + let mut spec: Vec<(f64, bool)> = (0..100)
377 + .map(|i| (0.9 - f64::from(i) * 0.001, true))
378 + .collect();
379 + spec.push((0.5, true));
380 + spec.extend((0..5).map(|_| (0.5, false)));
381 +
382 + let op = operating_point(&pts(&spec), 0.95, 1).unwrap();
383 + assert!(
384 + op.threshold > 0.5,
385 + "the tie was split; got {}",
386 + op.threshold
387 + );
388 + }
389 +
390 + #[test]
391 + fn out_of_fold_counts_an_uncalibratable_fold_as_misses() {
392 + // Nothing separates these, so no fold finds a threshold and every
393 + // positive must be a miss rather than silently vanishing.
394 + let p = pts(&[
395 + (0.5, true),
396 + (0.5, false),
397 + (0.5, true),
398 + (0.5, false),
399 + (0.5, true),
400 + (0.5, false),
401 + (0.5, true),
402 + (0.5, false),
403 + (0.5, true),
404 + (0.5, false),
405 + ]);
406 + let (counts, thresholds) = out_of_fold(&p, 5, 0.95, 1);
407 + assert!(thresholds.is_empty());
408 + assert_eq!(counts.tp, 0);
409 + assert_eq!(counts.actual(), 5, "all five positives are accounted for");
410 + }
411 +
412 + #[test]
413 + fn out_of_fold_is_not_more_optimistic_than_the_data_supports() {
414 + // Separable data: out-of-fold should recover it, since the threshold
415 + // learned on four folds transfers to the fifth.
416 + let mut spec: Vec<(f64, bool)> = (0..100)
417 + .map(|i| (0.9 - f64::from(i) * 0.001, true))
418 + .collect();
419 + spec.extend((0..100).map(|i| (0.3 - f64::from(i) * 0.001, false)));
420 + let p = pts(&spec);
421 + let (counts, thresholds) = out_of_fold(&p, 5, 0.95, 4);
422 + assert_eq!(thresholds.len(), 5, "every fold calibrates");
423 + // Precision holds on folds the threshold never saw, which is the whole
424 + // contract. Not 100%: the target is 0.95, so the most permissive
425 + // qualifying threshold deliberately admits a few negatives, and demanding
426 + // perfection here would be asserting a stricter bar than was asked for.
427 + assert!(
428 + counts.precision().unwrap() >= 0.95,
429 + "{:?}",
430 + counts.precision()
431 + );
432 + assert!(counts.recall().unwrap() > 0.9, "{:?}", counts.recall());
433 + }
434 +
435 + #[test]
436 + fn threshold_spread_reports_min_and_max() {
437 + let (mean, min, max) = threshold_spread(&[0.4, 0.6, 0.5]).unwrap();
438 + assert!((mean - 0.5).abs() < 1e-9);
439 + assert!((min - 0.4).abs() < 1e-9);
440 + assert!((max - 0.6).abs() < 1e-9);
441 + assert!(threshold_spread(&[]).is_none());
442 + }
443 + }