Skip to main content

max / audiofiles

50.0 KB · 1319 lines History Blame Raw
1 //! Cross-validated evaluation of the exemplar k-NN layer over the labelled corpus.
2 //!
3 //! This is the meter the bundled `.afcl` never had, and the reason it was
4 //! eventually retired rather than shipped (2026-08-08, wiki `af-likeness-web`).
5 //! The generator is gone; this outlived it because what it measures is the
6 //! exemplar k-NN itself, which the app still runs on.
7 //!
8 //! What it measures: stratified k-fold cross-validation of [`exemplar`] over the
9 //! same corpus the layer is built from, at the same `k` the app uses at runtime.
10 //! Each fold builds a real `ExemplarIndex` from the other folds and scores this
11 //! fold's samples against it, so no sample is ever a neighbour of itself and the
12 //! standardization params are fitted on training data alone.
13 //!
14 //! Why it is not one accuracy figure: the retired threshold classifier scored
15 //! 33.4% strict with two of its seven classes unreachable by any rule, and a
16 //! single number is exactly what hid that. Everything here is per class.
17 //!
18 //! # The gate changed after the first run, and that needs saying out loud
19 //!
20 //! Run 1 (2026-08-06, `02395cb`) graded every class on precision and recall at
21 //! [`DEFAULT_AUTO_THRESHOLD`], the global 0.85 the app ships. It failed: four of
22 //! seven classes had auto recall under 7%, percussion never fired at all.
23 //!
24 //! Reading the run said the threshold was doing most of the failing. A score is
25 //! the share of the k=15 neighbourhood's kernel weight carrying a tag, so what a
26 //! class can reach depends on how many of its own members sit inside a fixed `k`.
27 //! Hi-hat (109 files) and snare (188) are equally separable by top-1 (71.6%
28 //! against 76.6%) and differ ninefold in recall at 0.85. And 0.85 is itself an
29 //! unvalidated constant: it predates any measurement of this layer.
30 //!
31 //! So the gate below asks a different question, not an easier one. Old: "at the
32 //! threshold we happen to ship, is each class good enough?" New: "at the
33 //! precision we actually require, what threshold does each class need, and is the
34 //! recall there worth shipping?" The precision bar went **up**, 0.80 to 0.95,
35 //! because that is what auto-applying a tag into someone's library unasked
36 //! deserves; the per-class threshold is what stops class size being graded as if
37 //! it were quality. Run 1's numbers stay in `docs/ml_classifier.md` so the change
38 //! is auditable rather than quietly overwritten.
39 //!
40 //! The output still reports the shipped defaults, because "what happens if this
41 //! ships unchanged" remains a real question with a bad answer.
42 //!
43 //! # And then the whole question changed, which needs saying louder
44 //!
45 //! Runs 1 and 2 (2026-08-06, `02395cb` and `50fe541`) graded seven **specific drum
46 //! instruments**. Both halves of that were settled against on 2026-07-29 and this
47 //! module did it anyway: audiofiles classifies at coarse family resolution because
48 //! instrument identity is not in these features (33.4% with ~40 tuned thresholds
49 //! against 92.4% for families on one unfitted cut, and instrument labels are not
50 //! perceptually coherent), and drums-only is the defect the whole scope effort
51 //! exists to fix rather than the scope to measure within. The machinery below was
52 //! never the problem: fold split, index construction, calibration, the Wilson
53 //! bound and the confusion matrix are all label-agnostic. The label mapping was.
54 //!
55 //! So the corpus is now projected onto families as it is read back
56 //! ([`crate::families`]), and everything is graded at that resolution. Two things
57 //! follow that a reader should not have to infer:
58 //!
59 //! - The drum corpus reaches **two of seven** families, `low` and `drum-bright`.
60 //! Nothing here is a verdict on the layer. Five families are unmeasured.
61 //! - Runs 1 and 2 answer a retired question. They are not superseded results, they
62 //! are results for something else, and `docs/ml_classifier.md` says so.
63 //!
64 //! Usage: `cargo run --release -p audiofiles-bench -- layer-eval`
65 //! Env: `AF_BENCH_CORPUS`, `AF_BENCH_VAULT`, `AF_BENCH_EVAL_FOLDS` (default 5),
66 //! `AF_BENCH_EVAL_K` (comma-separated sweep, default `5,10,15,25,50`),
67 //! `AF_BENCH_EVAL_LABELS` (`family` default, `family-tom-split`, `instrument`),
68 //! `AF_BENCH_JSON`.
69 //!
70 //! [`DEFAULT_AUTO_THRESHOLD`]: audiofiles_core::analysis::exemplar::DEFAULT_AUTO_THRESHOLD
71
72 use std::collections::{BTreeMap, BTreeSet};
73 use std::path::Path;
74
75 use audiofiles_core::analysis::config::AnalysisConfig;
76 use audiofiles_core::analysis::exemplar::{
77 self, DEFAULT_AUTO_THRESHOLD, DEFAULT_K, DEFAULT_REVIEW_THRESHOLD,
78 };
79 use audiofiles_core::analysis::features::FEATURE_VERSION;
80
81 use crate::calibration::{self, Counts, Point};
82 use crate::families::{self, LabelSpace};
83 use crate::labelled;
84 use crate::report::Report;
85 use crate::rows::{self, Row};
86
87 /// Default fold count.
88 ///
89 /// Five over ~1000 files leaves every training fold within a fifth of the size of
90 /// the layer that actually ships, so the measured neighbourhood density is close
91 /// to the real one. Fewer folds would train on visibly less data than ships and
92 /// understate the layer; many more would leave the smallest class (clap, 48
93 /// files) with single-digit test sets whose per-class recall moves in 10% steps.
94 const DEFAULT_FOLDS: usize = 5;
95
96 /// Neighbour counts to sweep.
97 ///
98 /// `k` is a global constant at runtime and the score is a share of it, so it is
99 /// the lever that decides whether a small class can reach any threshold at all.
100 /// Sweeping it costs nothing (the corpus is imported and analysed once, and only
101 /// the scoring repeats) and it is the difference between "this class is hard" and
102 /// "this class is outnumbered inside a window we chose".
103 const DEFAULT_K_SWEEP: &[usize] = &[5, 10, 15, 25, 50];
104
105 /// Score thresholds the sweep tables report, spanning both shipped defaults.
106 const SWEEP_THRESHOLDS: &[f64] = &[0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.85, 0.9];
107
108 /// The ship gate. See the module header for why it is not run 1's gate.
109 struct Gate {
110 /// Precision each class must reach for the layer to auto-apply it, as a 95%
111 /// lower bound rather than an observed ratio (see [`calibration`]). A tag
112 /// written into a library unasked should be right 19 times in 20.
113 target_precision: f64,
114 /// Recall at that precision, below which a per-class threshold is not worth
115 /// shipping: the tag fires too rarely to be a head start.
116 min_recall: f64,
117 /// Floor under the confidence bound. The bound already rejects a precision
118 /// claimed off a handful of predictions (run 1 read cymbal as 100% precise
119 /// on two); this stops a policy shipping off a thin sample regardless.
120 min_support: usize,
121 /// Macro-averaged top-1 recall: the threshold-free separability floor. Below
122 /// this the feature space is not distinguishing these classes at all and no
123 /// per-class calibration rescues it.
124 top1_macro_recall: f64,
125 }
126
127 const GATE: Gate = Gate {
128 target_precision: 0.95,
129 min_recall: 0.40,
130 min_support: 15,
131 top1_macro_recall: 0.60,
132 };
133
134 /// What one test sample produced under one `k`.
135 struct Prediction {
136 truth: String,
137 /// Highest-scoring tag, or `None` when the index returned nothing.
138 top1: Option<String>,
139 /// Score per class, for threshold sweeps.
140 scores: BTreeMap<String, f64>,
141 /// The fold this sample was held out of, so a threshold is never calibrated
142 /// on the predictions it is graded against.
143 fold: usize,
144 /// Corpus folder(s) behind the truth label. See [`Row::origin`].
145 origin: String,
146 }
147
148 fn pct(v: Option<f64>) -> String {
149 v.map_or_else(|| "-".to_string(), |x| format!("{:.1}%", x * 100.0))
150 }
151
152 /// Every test sample's evidence for one class. A class absent from a sample's
153 /// scores scored zero for it, which is a real observation and not a gap.
154 fn class_points(predictions: &[Prediction], class: &str) -> Vec<Point> {
155 predictions
156 .iter()
157 .map(|p| Point {
158 score: p.scores.get(class).copied().unwrap_or(0.0),
159 actual: p.truth == class,
160 fold: p.fold,
161 })
162 .collect()
163 }
164
165 pub(crate) fn run(
166 corpus: &Path,
167 vault: &Path,
168 config: &AnalysisConfig,
169 folds: usize,
170 k_sweep: &[usize],
171 space: LabelSpace,
172 ) {
173 println!("━━━ CLASSIFIER LAYER EVALUATION ━━━");
174 println!();
175 println!(" corpus {}", corpus.display());
176 println!(" scratch {}", vault.display());
177 println!(" features v{FEATURE_VERSION}");
178 println!(" k {DEFAULT_K} (runtime default), sweeping {k_sweep:?}");
179 println!(" folds {folds}, stratified by class");
180 println!(" labels {}", space.describe());
181 println!();
182 println!(" Gate:");
183 println!(
184 " per-class precision, 95%-confident, at a per-class threshold >= {:.0}%",
185 GATE.target_precision * 100.0
186 );
187 println!(
188 " per-class recall at that threshold >= {:.0}%",
189 GATE.min_recall * 100.0
190 );
191 println!(
192 " predictions behind that precision >= {}",
193 GATE.min_support
194 );
195 println!(
196 " macro-averaged top-1 recall >= {:.0}%",
197 GATE.top1_macro_recall * 100.0
198 );
199 println!(" every class calibratable, in every fold");
200 println!();
201 println!(" Thresholds are calibrated on the folds a sample is NOT in, so no");
202 println!(" class picks its operating point from the data it is graded on.");
203 println!();
204
205 let built = match labelled::build_vault(corpus, vault, config) {
206 Ok(v) => v,
207 Err(e) => {
208 eprintln!("corpus: {e}");
209 std::process::exit(1);
210 }
211 };
212
213 let (rows, dropped) = match rows::load_rows(&built.db, space) {
214 Ok(r) => r,
215 Err(e) => {
216 eprintln!("reading the vault back: {e}");
217 std::process::exit(1);
218 }
219 };
220 if dropped.unprojectable > 0 {
221 println!();
222 println!(
223 " {} row(s) carry no label in this space ({}) and are excluded from",
224 dropped.unprojectable,
225 dropped
226 .origins
227 .iter()
228 .cloned()
229 .collect::<Vec<_>>()
230 .join(", ")
231 );
232 println!(" training and testing both:");
233 println!("{}", families::DROPPED_NOTE);
234 }
235 let ambiguous = rows.iter().filter(|r| r.truth.is_none()).count();
236 let testable = rows.len() - ambiguous;
237 if testable == 0 {
238 eprintln!("no single-labelled samples to test");
239 std::process::exit(1);
240 }
241 println!();
242 println!(" {} scoreable row(s) in the vault", rows.len());
243 if ambiguous > 0 {
244 // Not a silent drop: the same file under two class folders is a corpus
245 // problem, and the count is how anyone notices it grew.
246 println!(
247 " {ambiguous} carry more than one class tag (duplicate audio across folders);\n \
248 they train in every fold and are never tested"
249 );
250 }
251
252 let classes: BTreeSet<String> = rows.iter().filter_map(|r| r.truth.clone()).collect();
253 let classes: Vec<String> = classes.into_iter().collect();
254 let fold_of = rows::assign_folds(&rows, folds);
255
256 // Cross-validation. The index is built once per fold and scored at every `k`,
257 // because building it is the expensive half and `k` only enters at scoring.
258 let mut by_k: BTreeMap<usize, Vec<Prediction>> =
259 k_sweep.iter().map(|k| (*k, Vec::new())).collect();
260 for fold in 0..folds {
261 let train: Vec<&Row> = rows
262 .iter()
263 .zip(&fold_of)
264 .filter(|(_, f)| **f != Some(fold))
265 .map(|(r, _)| r)
266 .collect();
267 let test: Vec<&Row> = rows
268 .iter()
269 .zip(&fold_of)
270 .filter(|(_, f)| **f == Some(fold))
271 .map(|(r, _)| r)
272 .collect();
273
274 let db = match rows::local_db(&train) {
275 Ok(db) => db,
276 Err(e) => {
277 eprintln!("fold {fold}: {e}");
278 std::process::exit(1);
279 }
280 };
281 let index = match exemplar::build_index(&db) {
282 Ok(i) => i,
283 Err(e) => {
284 eprintln!("fold {fold}: build_index: {e}");
285 std::process::exit(1);
286 }
287 };
288 println!(
289 " fold {fold}: {} exemplars, {} held out",
290 index.len(),
291 test.len()
292 );
293
294 for row in test {
295 for &k in k_sweep {
296 // No `exclude_hash`: the row is not in this index at all, which
297 // is the property the fold split exists to give.
298 let scored = index.score(&row.vector, k, None);
299 let top1 = scored.first().map(|s| s.tag.clone());
300 let scores = scored.into_iter().map(|s| (s.tag, s.score)).collect();
301 by_k.entry(k).or_default().push(Prediction {
302 truth: row.truth.clone().unwrap_or_default(),
303 top1,
304 scores,
305 fold,
306 origin: row.origin.clone(),
307 });
308 }
309 }
310 }
311 println!();
312
313 let mut report = Report::new("layer-eval");
314 report.set("label_space", format!("{space:?}"));
315 report.set("dropped_unprojectable", dropped.unprojectable);
316 report.set("folds", folds);
317 report.set("k", DEFAULT_K);
318 report.set("feat_version", FEATURE_VERSION);
319 report.set("exemplars_total", rows.len());
320 report.set("ambiguous_excluded", ambiguous);
321 report.set("gate_target_precision", GATE.target_precision);
322 report.set("gate_min_recall", GATE.min_recall);
323 report.set("gate_min_support", GATE.min_support);
324 report.set("gate_top1_macro_recall", GATE.top1_macro_recall);
325
326 let default_k = by_k
327 .get(&DEFAULT_K)
328 .expect("the sweep always contains the runtime k");
329 report.set("tested", default_k.len());
330
331 let top1 = print_confusion(default_k, &classes, space, &mut report);
332 print_origin_breakdown(default_k, &classes, space, &mut report);
333 print_shipped_defaults(default_k, &classes, space, &mut report);
334 print_threshold_sweep(default_k, &classes, space);
335 let calibrated = print_calibration(default_k, &classes, folds, space, &mut report);
336 if k_sweep.len() > 1 {
337 print_k_sweep(&by_k, &classes, folds, &mut report);
338 print_nested_k(&rows, &fold_of, &classes, k_sweep, folds, &mut report);
339 }
340 print_verdict(&classes, &top1, &calibrated, space, &mut report);
341 report.write();
342 }
343
344 /// Top-1 confusion matrix, and per-class top-1 recall. Returns the per-class counts.
345 fn print_confusion(
346 predictions: &[Prediction],
347 classes: &[String],
348 space: LabelSpace,
349 report: &mut Report,
350 ) -> BTreeMap<String, Counts> {
351 println!("━━━ TOP-1 CONFUSION (k = {DEFAULT_K}) ━━━");
352 println!();
353 println!(" Rows are the corpus label, columns the highest-scoring tag.");
354 println!(" Threshold-free: this is what the layer would say if forced to pick,");
355 println!(" so it measures separability rather than any policy over it.");
356 println!();
357
358 let width = classes
359 .iter()
360 .map(|c| families::label_for(space, c).len().max(5))
361 .collect::<Vec<_>>();
362
363 print!(" {:<12}", "true \\ pred");
364 for (c, w) in classes.iter().zip(&width) {
365 print!(" {:>w$}", families::label_for(space, c), w = w);
366 }
367 println!(" {:>6} {:>8}", "(none)", "recall");
368 println!(
369 " {}",
370 "─".repeat(12 + width.iter().map(|w| w + 1).sum::<usize>() + 16)
371 );
372
373 let mut counts: BTreeMap<String, Counts> = BTreeMap::new();
374 let mut never_predicted: Vec<&str> = Vec::new();
375
376 for truth in classes {
377 let mine: Vec<&Prediction> = predictions.iter().filter(|p| &p.truth == truth).collect();
378 print!(" {:<12}", families::label_for(space, truth));
379 let mut correct = 0usize;
380 for (pred, w) in classes.iter().zip(&width) {
381 let n = mine
382 .iter()
383 .filter(|p| p.top1.as_deref() == Some(pred.as_str()))
384 .count();
385 if pred == truth {
386 correct = n;
387 }
388 print!(" {n:>w$}");
389 }
390 let none = mine.iter().filter(|p| p.top1.is_none()).count();
391 let recall = if mine.is_empty() {
392 None
393 } else {
394 Some(correct as f64 / mine.len() as f64)
395 };
396 println!(" {:>6} {:>8}", none, pct(recall));
397
398 // Precision needs the whole column, so it is counted here rather than
399 // inside the row loop.
400 let predicted_as = predictions
401 .iter()
402 .filter(|p| p.top1.as_deref() == Some(truth.as_str()))
403 .count();
404 if predicted_as == 0 {
405 never_predicted.push(families::label_for(space, truth));
406 }
407 counts.insert(
408 truth.clone(),
409 Counts {
410 tp: correct,
411 fp: predicted_as - correct,
412 fn_: mine.len() - correct,
413 },
414 );
415 }
416 println!();
417
418 let macro_recall = macro_average(classes, &counts, Counts::recall);
419 let micro = counts.values().map(|c| c.tp).sum::<usize>() as f64 / predictions.len() as f64;
420 println!(" macro-averaged recall {}", pct(macro_recall));
421 println!(" overall top-1 accuracy {}", pct(Some(micro)));
422 if never_predicted.is_empty() {
423 println!(" Every class is predicted at least once.");
424 } else {
425 println!();
426 println!(
427 " NEVER PREDICTED: {}. This class is unreachable, not merely weak.",
428 never_predicted.join(", ")
429 );
430 }
431 println!();
432
433 if let Some(m) = macro_recall {
434 report.set("top1_macro_recall", round4(m));
435 }
436 report.set("top1_accuracy", round4(micro));
437 report.set("never_predicted", never_predicted.len());
438 for (tag, c) in &counts {
439 let label = families::label_for(space, tag);
440 if let Some(r) = c.recall() {
441 report.set(&format!("top1_{label}_recall"), round4(r));
442 }
443 if let Some(p) = c.precision() {
444 report.set(&format!("top1_{label}_precision"), round4(p));
445 }
446 }
447 counts
448 }
449
450 /// Per-class top-1 recall broken out by the corpus folder each sample came from.
451 ///
452 /// A family is only a family if its members behave like one. `low` is kick plus
453 /// tom, and `af-coarse-families` calls that the one open split worth measuring:
454 /// tom sits between `low` and `bass` on centroid (p25-p75 904-1996 against kick's
455 /// 397-885) and is 23% of the drum corpus, so folding it in silently assumes the
456 /// answer. If toms are recovered as `low` about as often as kicks are, the fold
457 /// holds; if they are systematically lost, `low` is two things wearing one label.
458 ///
459 /// This is the measurement, not a proposal to add a `tom` family. Reading it
460 /// against `AF_BENCH_EVAL_LABELS=family-tom-split`, which grades tom as its own
461 /// class, is what separates "tom is hard" from "tom is not low".
462 fn print_origin_breakdown(
463 predictions: &[Prediction],
464 classes: &[String],
465 space: LabelSpace,
466 report: &mut Report,
467 ) {
468 let origins: BTreeSet<&str> = predictions.iter().map(|p| p.origin.as_str()).collect();
469 // Nothing to say when every class is one folder: the table would be the
470 // recall column of the confusion matrix, transposed.
471 if origins.len() <= classes.len() {
472 return;
473 }
474
475 println!("━━━ BY CORPUS ORIGIN (k = {DEFAULT_K}) ━━━");
476 println!();
477 println!(" The same top-1 answers, grouped by the folder the sample came from");
478 println!(" rather than by the class it was projected onto. A family whose");
479 println!(" members disagree here is not one family.");
480 println!();
481 println!(
482 " {:<14} {:<14} {:>6} {:>9} most common wrong answer",
483 "origin", "projects to", "n", "recall"
484 );
485 println!(" {}", "─".repeat(76));
486
487 for origin in origins {
488 let mine: Vec<&Prediction> = predictions.iter().filter(|p| p.origin == origin).collect();
489 let Some(truth) = mine.first().map(|p| p.truth.clone()) else {
490 continue;
491 };
492 let correct = mine
493 .iter()
494 .filter(|p| p.top1.as_deref() == Some(truth.as_str()))
495 .count();
496 let recall = correct as f64 / mine.len() as f64;
497
498 // Where the misses go, which is the informative half: a tom read as
499 // drum-bright says something different from a tom the index has no
500 // answer for at all.
501 let mut wrong: BTreeMap<&str, usize> = BTreeMap::new();
502 for p in &mine {
503 match p.top1.as_deref() {
504 Some(t) if t != truth => *wrong.entry(t).or_default() += 1,
505 None => *wrong.entry("(none)").or_default() += 1,
506 _ => {}
507 }
508 }
509 let worst = wrong.iter().max_by_key(|(_, n)| **n).map_or_else(
510 || "-".to_string(),
511 |(t, n)| {
512 let label = if *t == "(none)" {
513 "(none)"
514 } else {
515 families::label_for(space, t)
516 };
517 format!("{label} ({n})")
518 },
519 );
520
521 println!(
522 " {:<14} {:<14} {:>6} {:>9} {worst}",
523 origin,
524 families::label_for(space, &truth),
525 mine.len(),
526 pct(Some(recall)),
527 );
528 report.set(&format!("origin_{origin}_recall"), round4(recall));
529 report.set(&format!("origin_{origin}_n"), mine.len());
530 }
531 println!();
532 }
533
534 /// What the layer does today, unchanged: one global auto threshold for every
535 /// class. Kept because it is the status quo the ship decision is against.
536 fn print_shipped_defaults(
537 predictions: &[Prediction],
538 classes: &[String],
539 space: LabelSpace,
540 report: &mut Report,
541 ) {
542 println!("━━━ AT THE SHIPPED DEFAULTS (one global threshold) ━━━");
543 println!();
544
545 let mut counts: BTreeMap<String, Counts> = BTreeMap::new();
546 for class in classes {
547 counts.insert(
548 class.clone(),
549 calibration::counts_at(&class_points(predictions, class), DEFAULT_AUTO_THRESHOLD),
550 );
551 }
552
553 println!(
554 " {:<12} {:>7} {:>8} {:>10} {:>9} auto {DEFAULT_AUTO_THRESHOLD:.2} / review {DEFAULT_REVIEW_THRESHOLD:.2}",
555 "class", "n", "fired", "precision", "recall"
556 );
557 println!(" {}", "─".repeat(50));
558 for class in classes {
559 let c = counts[class];
560 println!(
561 " {:<12} {:>7} {:>8} {:>10} {:>9}",
562 families::label_for(space, class),
563 c.actual(),
564 c.fired(),
565 pct(c.precision()),
566 pct(c.recall()),
567 );
568 }
569 println!(" {}", "─".repeat(50));
570 println!(
571 " {:<12} {:>7} {:>8} {:>10} {:>9}",
572 "macro",
573 predictions.len(),
574 counts.values().map(|c| c.fired()).sum::<usize>(),
575 pct(macro_average(classes, &counts, Counts::precision)),
576 pct(macro_average(classes, &counts, Counts::recall)),
577 );
578 println!();
579 let silent = predictions
580 .iter()
581 .filter(|p| !p.scores.values().any(|s| *s >= DEFAULT_AUTO_THRESHOLD))
582 .count();
583 println!(
584 " {silent} of {} samples ({:.0}%) get no tag at all.",
585 predictions.len(),
586 silent as f64 / predictions.len() as f64 * 100.0
587 );
588 println!();
589
590 for (tag, c) in &counts {
591 let label = families::label_for(space, tag);
592 if let Some(p) = c.precision() {
593 report.set(&format!("shipped_{label}_precision"), round4(p));
594 }
595 if let Some(r) = c.recall() {
596 report.set(&format!("shipped_{label}_recall"), round4(r));
597 }
598 }
599 report.set("shipped_silent_samples", silent);
600 }
601
602 /// Precision and recall for every class across a range of thresholds.
603 ///
604 /// This is the evidence that one global threshold cannot serve seven classes: read
605 /// down a column and the same number means a different thing in every row.
606 fn print_threshold_sweep(predictions: &[Prediction], classes: &[String], space: LabelSpace) {
607 let points: BTreeMap<&String, Vec<Point>> = classes
608 .iter()
609 .map(|c| (c, class_points(predictions, c)))
610 .collect();
611
612 for (title, metric) in [
613 ("PRECISION", Counts::precision as fn(Counts) -> Option<f64>),
614 ("RECALL", Counts::recall as fn(Counts) -> Option<f64>),
615 ] {
616 println!("━━━ {title} BY THRESHOLD (k = {DEFAULT_K}) ━━━");
617 println!();
618 print!(" {:<12}", "class");
619 for t in SWEEP_THRESHOLDS {
620 print!(" {t:>7.2}");
621 }
622 println!();
623 println!(" {}", "─".repeat(12 + SWEEP_THRESHOLDS.len() * 8));
624 for class in classes {
625 print!(" {:<12}", families::label_for(space, class));
626 for t in SWEEP_THRESHOLDS {
627 let c = calibration::counts_at(&points[class], *t);
628 print!(" {:>7}", pct(metric(c)));
629 }
630 println!();
631 }
632 println!();
633 }
634 println!(" A dash is a class that fires nothing at that threshold, which is not");
635 println!(" the same as firing and being wrong. Read the two tables together: a");
636 println!(" class holding high precision far down the range has headroom the");
637 println!(" global 0.85 is not spending.");
638 println!();
639 }
640
641 /// The per-class operating points the data supports, and what they deliver on
642 /// folds they were not calibrated on.
643 fn print_calibration(
644 predictions: &[Prediction],
645 classes: &[String],
646 folds: usize,
647 space: LabelSpace,
648 report: &mut Report,
649 ) -> BTreeMap<String, Counts> {
650 println!(
651 "━━━ CALIBRATED OPERATING POINTS (target precision {:.0}%) ━━━",
652 GATE.target_precision * 100.0
653 );
654 println!();
655 println!(" The most permissive threshold at which each class still meets the");
656 println!(" precision bar, and what it buys. These are `tag_policy` rows: what a");
657 println!(" layer would ship if it carried its own thresholds instead of");
658 println!(" inheriting one global pair.");
659 println!();
660 println!(
661 " {:<12} {:>7} {:>10} {:>10} {:>9} {:>18}",
662 "class", "n", "threshold", "precision", "recall", "held-out recall"
663 );
664 println!(" {}", "─".repeat(70));
665
666 let mut out_of_fold: BTreeMap<String, Counts> = BTreeMap::new();
667 for class in classes {
668 let points = class_points(predictions, class);
669 let in_sample =
670 calibration::operating_point(&points, GATE.target_precision, GATE.min_support);
671 let (oof, thresholds) =
672 calibration::out_of_fold(&points, folds, GATE.target_precision, GATE.min_support);
673 out_of_fold.insert(class.clone(), oof);
674
675 let label = families::label_for(space, class);
676 match in_sample {
677 Some(op) => println!(
678 " {:<12} {:>7} {:>10.3} {:>10} {:>9} {:>18}",
679 label,
680 op.counts.actual(),
681 op.threshold,
682 pct(op.counts.precision()),
683 pct(op.counts.recall()),
684 pct(oof.recall()),
685 ),
686 None => println!(
687 " {:<12} {:>7} {:>10} {:>10} {:>9} {:>18}",
688 label,
689 points.iter().filter(|p| p.actual).count(),
690 "none",
691 "-",
692 "-",
693 pct(oof.recall()),
694 ),
695 }
696
697 if let Some((mean, min, max)) = calibration::threshold_spread(&thresholds) {
698 report.set(&format!("calibrated_{label}_threshold"), round4(mean));
699 if (max - min) > 0.15 {
700 // A threshold that moves this much between folds is not a stable
701 // property of the class, and shipping the mean would be fiction.
702 println!(
703 " {:<12} threshold unstable across folds: {min:.2} to {max:.2}",
704 ""
705 );
706 }
707 }
708 if thresholds.len() < folds {
709 println!(
710 " {:<12} {} of {folds} folds found no qualifying threshold",
711 "",
712 folds - thresholds.len()
713 );
714 }
715 if let Some(p) = oof.precision() {
716 report.set(&format!("calibrated_{label}_precision"), round4(p));
717 }
718 if let Some(r) = oof.recall() {
719 report.set(&format!("calibrated_{label}_recall"), round4(r));
720 }
721 report.set(&format!("calibrated_{label}_fired"), oof.fired());
722 }
723 println!(" {}", "─".repeat(70));
724 println!(
725 " {:<12} {:>7} {:>10} {:>10} {:>9} {:>18}",
726 "macro",
727 predictions.len(),
728 "",
729 pct(macro_average(classes, &out_of_fold, Counts::precision)),
730 "",
731 pct(macro_average(classes, &out_of_fold, Counts::recall)),
732 );
733 println!();
734 println!(" The last column is the honest one: thresholds chosen on four folds,");
735 println!(" measured on the fifth. The gap between it and the recall column is");
736 println!(" how much of the calibration was fitting noise.");
737 println!();
738
739 if let Some(p) = macro_average(classes, &out_of_fold, Counts::precision) {
740 report.set("calibrated_macro_precision", round4(p));
741 }
742 if let Some(r) = macro_average(classes, &out_of_fold, Counts::recall) {
743 report.set("calibrated_macro_recall", round4(r));
744 }
745 out_of_fold
746 }
747
748 /// Does `k` move the classes a fixed window was starving?
749 fn print_k_sweep(
750 by_k: &BTreeMap<usize, Vec<Prediction>>,
751 classes: &[String],
752 folds: usize,
753 report: &mut Report,
754 ) {
755 println!("━━━ k SWEEP ━━━");
756 println!();
757 println!(" A score is a share of k neighbours, so k decides whether a small");
758 println!(" class can reach any threshold at all. Calibrated columns are");
759 println!(" held-out, per class, at the target precision.");
760 println!();
761 println!(
762 " {:>5} {:>12} {:>14} {:>16} {:>14} {:>12}",
763 "k", "top-1 macro", "calib. recall", "calib. precision", "worst class", "uncalib."
764 );
765 println!(" {}", "─".repeat(78));
766
767 for (k, predictions) in by_k {
768 let mut top1: BTreeMap<String, Counts> = BTreeMap::new();
769 let mut calibrated: BTreeMap<String, Counts> = BTreeMap::new();
770 let mut uncalibratable = 0usize;
771 for class in classes {
772 let mine = predictions.iter().filter(|p| &p.truth == class).count();
773 let correct = predictions
774 .iter()
775 .filter(|p| &p.truth == class && p.top1.as_deref() == Some(class.as_str()))
776 .count();
777 let predicted_as = predictions
778 .iter()
779 .filter(|p| p.top1.as_deref() == Some(class.as_str()))
780 .count();
781 top1.insert(
782 class.clone(),
783 Counts {
784 tp: correct,
785 fp: predicted_as - correct,
786 fn_: mine - correct,
787 },
788 );
789
790 let points = class_points(predictions, class);
791 let (oof, thresholds) =
792 calibration::out_of_fold(&points, folds, GATE.target_precision, GATE.min_support);
793 if thresholds.is_empty() {
794 uncalibratable += 1;
795 }
796 calibrated.insert(class.clone(), oof);
797 }
798
799 let worst = classes
800 .iter()
801 .map(|c| calibrated[c].recall().unwrap_or(0.0))
802 .fold(f64::INFINITY, f64::min);
803 let marker = if *k == DEFAULT_K { " <- runtime" } else { "" };
804 println!(
805 " {:>5} {:>12} {:>14} {:>16} {:>14} {:>12}{marker}",
806 k,
807 pct(macro_average(classes, &top1, Counts::recall)),
808 pct(macro_average(classes, &calibrated, Counts::recall)),
809 pct(macro_average(classes, &calibrated, Counts::precision)),
810 pct(Some(worst)),
811 uncalibratable,
812 );
813
814 report.set(
815 &format!("k{k}_top1_macro_recall"),
816 round4(macro_average(classes, &top1, Counts::recall).unwrap_or(0.0)),
817 );
818 report.set(
819 &format!("k{k}_calibrated_macro_recall"),
820 round4(macro_average(classes, &calibrated, Counts::recall).unwrap_or(0.0)),
821 );
822 report.set(&format!("k{k}_uncalibratable_classes"), uncalibratable);
823 }
824 println!();
825 println!(" `uncalib.` counts classes for which no fold found a threshold meeting");
826 println!(" the precision bar. Those are the classes a shipped layer cannot apply");
827 println!(" at all, whatever the global default is set to.");
828 println!();
829 println!(" CAVEAT: this table selects k on the data it reports. Thresholds are");
830 println!(" calibrated out of fold, k is not, so reading the best row here and");
831 println!(" shipping that k would be choosing a hyperparameter on the test set.");
832 println!(" It is evidence about the mechanism. The outer fold below is the");
833 println!(" number to quote instead.");
834 println!();
835 }
836
837 /// Select `k` under an outer fold, so the selection is never scored on the data
838 /// it was made from.
839 ///
840 /// The sweep above is the standard trap: it fits thresholds out of fold and then
841 /// picks `k` by reading every row of the result. That is choosing a
842 /// hyperparameter on the test set, and the honest correction is the same shape as
843 /// the one that fixed the thresholds — hold the selection out too.
844 ///
845 /// Procedure. For each outer fold: take the other folds as a training corpus,
846 /// split THOSE by an inner fold, select the `k` with the best inner macro
847 /// calibrated recall, then build one index over the whole training corpus and
848 /// score the outer fold at the selected `k`. Nothing about the outer fold is
849 /// visible to the selection, so the pooled result is what "select k this way"
850 /// generalises to.
851 ///
852 /// Indexes are rebuilt rather than reused from `by_k`: a prediction there was
853 /// made by an index containing the outer test fold, so recycling them would leak
854 /// exactly what this exists to stop. That costs an extra `folds * folds` index
855 /// builds and no extra analysis, which is the cheap half.
856 ///
857 /// What to read: if the outer number matches the best row of the sweep, the
858 /// sweep was not being flattered by its own selection and the mechanism finding
859 /// stands. If it comes in below, the difference is the selection bias, and the
860 /// outer number is the one that would survive contact with a user's library.
861 fn print_nested_k(
862 rows: &[Row],
863 outer_of: &[Option<usize>],
864 classes: &[String],
865 k_sweep: &[usize],
866 folds: usize,
867 report: &mut Report,
868 ) {
869 println!("━━━ k UNDER AN OUTER FOLD ━━━");
870 println!();
871 println!(" k selected inside each outer fold's training corpus, then scored on the");
872 println!(" outer fold it never saw. This is the k-sweep number with the selection");
873 println!(" bias removed.");
874 println!();
875 println!(
876 " {:>7} {:>10} {:>10} {:>16} {:>14}",
877 "outer", "train", "k chosen", "inner recall", "outer recall"
878 );
879 println!(" {}", "─".repeat(64));
880
881 let mut pooled: Vec<Prediction> = Vec::new();
882 let mut chosen: Vec<usize> = Vec::new();
883
884 for outer in 0..folds {
885 let train: Vec<&Row> = rows
886 .iter()
887 .zip(outer_of)
888 .filter(|(_, f)| **f != Some(outer))
889 .map(|(r, _)| r)
890 .collect();
891 let test: Vec<&Row> = rows
892 .iter()
893 .zip(outer_of)
894 .filter(|(_, f)| **f == Some(outer))
895 .map(|(r, _)| r)
896 .collect();
897 if test.is_empty() {
898 continue;
899 }
900
901 // Inner CV over the training corpus only.
902 let train_rows: Vec<Row> = train.iter().map(|r| clone_row(r)).collect();
903 let inner_of = rows::assign_folds(&train_rows, folds);
904 let mut inner: BTreeMap<usize, Vec<Prediction>> =
905 k_sweep.iter().map(|k| (*k, Vec::new())).collect();
906 for i in 0..folds {
907 let itrain: Vec<&Row> = train_rows
908 .iter()
909 .zip(&inner_of)
910 .filter(|(_, f)| **f != Some(i))
911 .map(|(r, _)| r)
912 .collect();
913 let itest: Vec<&Row> = train_rows
914 .iter()
915 .zip(&inner_of)
916 .filter(|(_, f)| **f == Some(i))
917 .map(|(r, _)| r)
918 .collect();
919 let Ok(db) = rows::local_db(&itrain) else {
920 continue;
921 };
922 let Ok(index) = exemplar::build_index(&db) else {
923 continue;
924 };
925 for row in itest {
926 for &k in k_sweep {
927 push_prediction(inner.entry(k).or_default(), &index, row, k, i);
928 }
929 }
930 }
931
932 // Selection criterion: macro calibrated recall, the same quantity the
933 // sweep table ranks on, so the two are comparable.
934 let score_of = |preds: &Vec<Prediction>| {
935 let mut calibrated: BTreeMap<String, Counts> = BTreeMap::new();
936 for class in classes {
937 let points = class_points(preds, class);
938 let (oof, _) = calibration::out_of_fold(
939 &points,
940 folds,
941 GATE.target_precision,
942 GATE.min_support,
943 );
944 calibrated.insert(class.clone(), oof);
945 }
946 macro_average(classes, &calibrated, Counts::recall).unwrap_or(0.0)
947 };
948 let Some((best_k, inner_recall)) = k_sweep
949 .iter()
950 .map(|k| (*k, score_of(&inner[k])))
951 // total_cmp then the smaller k, so a tie ships the tighter
952 // neighbourhood rather than whichever the map iterated first.
953 .max_by(|a, b| a.1.total_cmp(&b.1).then(b.0.cmp(&a.0)))
954 else {
955 continue;
956 };
957 chosen.push(best_k);
958
959 // Score the outer fold at the selected k, from an index over the whole
960 // training corpus.
961 let Ok(db) = rows::local_db(&train) else {
962 continue;
963 };
964 let Ok(index) = exemplar::build_index(&db) else {
965 continue;
966 };
967 let mut outer_preds: Vec<Prediction> = Vec::new();
968 for row in &test {
969 push_prediction(&mut outer_preds, &index, row, best_k, outer);
970 }
971 let outer_recall = {
972 let mut top1: BTreeMap<String, Counts> = BTreeMap::new();
973 for class in classes {
974 let mine = outer_preds.iter().filter(|p| &p.truth == class).count();
975 let correct = outer_preds
976 .iter()
977 .filter(|p| &p.truth == class && p.top1.as_deref() == Some(class.as_str()))
978 .count();
979 let predicted_as = outer_preds
980 .iter()
981 .filter(|p| p.top1.as_deref() == Some(class.as_str()))
982 .count();
983 top1.insert(
984 class.clone(),
985 Counts {
986 tp: correct,
987 fp: predicted_as - correct,
988 fn_: mine - correct,
989 },
990 );
991 }
992 macro_average(classes, &top1, Counts::recall)
993 };
994
995 println!(
996 " {:>7} {:>10} {:>10} {:>16} {:>14}",
997 outer,
998 train.len(),
999 best_k,
1000 pct(Some(inner_recall)),
1001 pct(outer_recall),
1002 );
1003 pooled.extend(outer_preds);
1004 }
1005 println!();
1006
1007 if chosen.is_empty() {
1008 println!(" no outer fold completed");
1009 println!();
1010 return;
1011 }
1012
1013 let mut top1: BTreeMap<String, Counts> = BTreeMap::new();
1014 for class in classes {
1015 let mine = pooled.iter().filter(|p| &p.truth == class).count();
1016 let correct = pooled
1017 .iter()
1018 .filter(|p| &p.truth == class && p.top1.as_deref() == Some(class.as_str()))
1019 .count();
1020 let predicted_as = pooled
1021 .iter()
1022 .filter(|p| p.top1.as_deref() == Some(class.as_str()))
1023 .count();
1024 top1.insert(
1025 class.clone(),
1026 Counts {
1027 tp: correct,
1028 fp: predicted_as - correct,
1029 fn_: mine - correct,
1030 },
1031 );
1032 }
1033 let pooled_recall = macro_average(classes, &top1, Counts::recall);
1034 let agreed: BTreeSet<usize> = chosen.iter().copied().collect();
1035
1036 println!(
1037 " Pooled outer macro top-1 recall {} over {} predictions.",
1038 pct(pooled_recall),
1039 pooled.len()
1040 );
1041 if agreed.len() == 1 {
1042 let k = *agreed.iter().next().unwrap_or(&DEFAULT_K);
1043 println!(" Every outer fold selected k = {k}. A selection that does not move with");
1044 println!(" the training data is one the corpus supports, not one it happened onto.");
1045 if k != DEFAULT_K {
1046 println!(
1047 " It is NOT the shipped k ({DEFAULT_K}). That is a real finding, not a rounding:"
1048 );
1049 println!(" the runtime constant predates every measurement of this layer.");
1050 }
1051 report.set("nested_k_selected", k);
1052 } else {
1053 let spread: Vec<String> = agreed.iter().map(usize::to_string).collect();
1054 println!(" Outer folds disagreed on k: {}.", spread.join(", "));
1055 println!(" A selection that moves with the training data is not a property of the");
1056 println!(" corpus, and shipping any single one of these is a coin toss dressed as a");
1057 println!(" measurement. Read the sweep as a mechanism finding and leave k alone.");
1058 report.set("nested_k_disagreed", agreed.len());
1059 }
1060 if let Some(r) = pooled_recall {
1061 report.set("nested_k_outer_macro_recall", round4(r));
1062 }
1063 println!();
1064 }
1065
1066 /// Score one row against one index and push the prediction.
1067 fn push_prediction(
1068 into: &mut Vec<Prediction>,
1069 index: &exemplar::ExemplarIndex,
1070 row: &Row,
1071 k: usize,
1072 fold: usize,
1073 ) {
1074 // No `exclude_hash`: the row is not in this index at all, which is the
1075 // property the fold split exists to give.
1076 let scored = index.score(&row.vector, k, None);
1077 into.push(Prediction {
1078 truth: row.truth.clone().unwrap_or_default(),
1079 top1: scored.first().map(|s| s.tag.clone()),
1080 scores: scored.into_iter().map(|s| (s.tag, s.score)).collect(),
1081 fold,
1082 origin: row.origin.clone(),
1083 });
1084 }
1085
1086 /// A `Row` copy, so the inner split can own its training corpus.
1087 ///
1088 /// `assign_folds` takes `&[Row]` rather than `&[&Row]`, and the inner fold is
1089 /// assigned over a subset that only exists as borrows. Copying ~800 short vectors
1090 /// once per outer fold is cheaper than threading a second lifetime through the
1091 /// split.
1092 fn clone_row(r: &Row) -> Row {
1093 Row {
1094 hash: r.hash.clone(),
1095 vector: r.vector.clone(),
1096 tags: r.tags.clone(),
1097 truth: r.truth.clone(),
1098 origin: r.origin.clone(),
1099 name: r.name.clone(),
1100 }
1101 }
1102
1103 fn macro_average(
1104 classes: &[String],
1105 counts: &BTreeMap<String, Counts>,
1106 metric: fn(Counts) -> Option<f64>,
1107 ) -> Option<f64> {
1108 // Classes with no metric (nothing predicted, so no precision) count as zero
1109 // rather than being skipped. Dropping them would let a layer raise its macro
1110 // average by predicting a class less often.
1111 let vals: Vec<f64> = classes
1112 .iter()
1113 .map(|c| counts.get(c).and_then(|c| metric(*c)).unwrap_or(0.0))
1114 .collect();
1115 (!vals.is_empty()).then(|| vals.iter().sum::<f64>() / vals.len() as f64)
1116 }
1117
1118 fn round4(v: f64) -> f64 {
1119 (v * 10000.0).round() / 10000.0
1120 }
1121
1122 /// Score the run against [`GATE`] and say plainly whether it passes.
1123 fn print_verdict(
1124 classes: &[String],
1125 top1: &BTreeMap<String, Counts>,
1126 calibrated: &BTreeMap<String, Counts>,
1127 space: LabelSpace,
1128 report: &mut Report,
1129 ) {
1130 println!("━━━ VERDICT ━━━");
1131 println!();
1132
1133 let mut failures: Vec<String> = Vec::new();
1134
1135 for class in classes {
1136 let label = families::label_for(space, class);
1137 let c = calibrated[class];
1138 if c.fired() == 0 {
1139 failures.push(format!(
1140 "{label}: no threshold reaches {:.0}% precision, so the layer cannot apply it at all",
1141 GATE.target_precision * 100.0
1142 ));
1143 continue;
1144 }
1145 if let Some(r) = c.recall()
1146 && r < GATE.min_recall
1147 {
1148 failures.push(format!(
1149 "{label}: held-out recall {} at the precision bar, below the {:.0}% gate",
1150 pct(Some(r)),
1151 GATE.min_recall * 100.0
1152 ));
1153 }
1154 if let Some(p) = c.precision()
1155 && p < GATE.target_precision
1156 {
1157 // The threshold met the bar in calibration and missed it held out,
1158 // which means the operating point does not generalise.
1159 failures.push(format!(
1160 "{label}: held-out precision {} below the {:.0}% bar its threshold was calibrated to",
1161 pct(Some(p)),
1162 GATE.target_precision * 100.0
1163 ));
1164 }
1165 if top1.get(class).is_some_and(|t| t.fired() == 0) {
1166 failures.push(format!("{label}: never the top-1 answer for any sample"));
1167 }
1168 }
1169
1170 let macro_recall = macro_average(classes, top1, Counts::recall).unwrap_or(0.0);
1171 if macro_recall < GATE.top1_macro_recall {
1172 failures.push(format!(
1173 "macro top-1 recall {} below the {:.0}% gate",
1174 pct(Some(macro_recall)),
1175 GATE.top1_macro_recall * 100.0
1176 ));
1177 }
1178
1179 if failures.is_empty() {
1180 println!(" PASS, on a per-class policy. Shipping this means shipping the");
1181 println!(" calibrated thresholds with it: set `include_policy` in afcl_gen and");
1182 println!(" export the tag_policy rows, or the layer inherits the global 0.85");
1183 println!(" and none of the above holds.");
1184 } else {
1185 println!(" FAIL on {} criterion/criteria:", failures.len());
1186 for f in &failures {
1187 println!(" - {f}");
1188 }
1189 }
1190 println!();
1191
1192 // Printed on pass and on fail both, because the scope caveat is not a
1193 // consolation for a failure: a pass here is the more dangerous of the two to
1194 // read as a verdict on the layer.
1195 print_scope(classes, space);
1196
1197 report.set("gate_pass", failures.is_empty());
1198 report.set("gate_failures", failures.len());
1199 }
1200
1201 /// What this corpus can and cannot support a claim about.
1202 ///
1203 /// The gate above says whether the classes present are shippable. It cannot say
1204 /// anything about the classes absent, and the absent ones are the majority: the
1205 /// corpus is drum one-shots, so it reaches two of the seven families and five have
1206 /// no material at all. A layer that answers `drum-bright` confidently for a vocal
1207 /// it has never seen passes every criterion above.
1208 fn print_scope(classes: &[String], space: LabelSpace) {
1209 if space == LabelSpace::Instrument {
1210 println!(" SCOPE: this run grades specific drum instruments, which is the");
1211 println!(" retired question (wiki af-browse-axes, 2026-07-29). Kept runnable so");
1212 println!(" the results already written up stay reproducible. Do not extend it.");
1213 println!();
1214 return;
1215 }
1216
1217 let (covered, uncovered) = families::covered_families(classes);
1218 println!(
1219 " SCOPE: {} of {} families measured: {}.",
1220 covered.len(),
1221 families::FAMILIES.len(),
1222 covered.join(", ")
1223 );
1224 if !uncovered.is_empty() {
1225 println!(" No corpus material for: {}.", uncovered.join(", "));
1226 println!(" The layer has never been asked about them and will answer with the");
1227 println!(" nearest drum it knows. Nothing above is a verdict on the layer;");
1228 println!(" widening the corpus is the next phase.");
1229 }
1230 println!();
1231 }
1232
1233 /// Fold count from the environment, or [`DEFAULT_FOLDS`].
1234 pub(crate) fn folds_from_env() -> usize {
1235 std::env::var("AF_BENCH_EVAL_FOLDS")
1236 .ok()
1237 .and_then(|v| v.parse().ok())
1238 .filter(|n| *n >= 2)
1239 .unwrap_or(DEFAULT_FOLDS)
1240 }
1241
1242 /// Neighbour counts to sweep, always including the runtime default so the
1243 /// detailed sections have something to report.
1244 pub(crate) fn k_sweep_from_env() -> Vec<usize> {
1245 let mut ks: Vec<usize> = std::env::var("AF_BENCH_EVAL_K").map_or_else(
1246 |_| DEFAULT_K_SWEEP.to_vec(),
1247 |v| {
1248 v.split(',')
1249 .filter_map(|s| s.trim().parse().ok())
1250 .filter(|k| *k > 0)
1251 .collect()
1252 },
1253 );
1254 ks.push(DEFAULT_K);
1255 ks.sort_unstable();
1256 ks.dedup();
1257 ks
1258 }
1259
1260 #[cfg(test)]
1261 mod tests {
1262 use super::*;
1263
1264 #[test]
1265 fn macro_average_counts_an_unpredicted_class_as_zero() {
1266 // Skipping it would let a layer raise its macro precision by predicting
1267 // a hard class less often, which is backwards.
1268 let classes = vec!["a".to_string(), "b".to_string()];
1269 let mut counts = BTreeMap::new();
1270 counts.insert(
1271 "a".to_string(),
1272 Counts {
1273 tp: 10,
1274 fp: 0,
1275 fn_: 0,
1276 },
1277 );
1278 counts.insert(
1279 "b".to_string(),
1280 Counts {
1281 tp: 0,
1282 fp: 0,
1283 fn_: 10,
1284 },
1285 );
1286 assert_eq!(
1287 macro_average(&classes, &counts, Counts::precision),
1288 Some(0.5)
1289 );
1290 }
1291
1292 #[test]
1293 fn class_points_score_an_absent_class_as_zero() {
1294 // A class missing from a sample's scores is a real zero, not a gap: the
1295 // index considered it and gave it no neighbourhood weight. Treating it
1296 // as missing would drop true negatives and inflate precision.
1297 let p = vec![Prediction {
1298 truth: "instrument.drum.kick".into(),
1299 top1: Some("instrument.drum.kick".into()),
1300 scores: BTreeMap::from([("instrument.drum.kick".to_string(), 0.9)]),
1301 fold: 0,
1302 origin: "kick".into(),
1303 }];
1304 let pts = class_points(&p, "instrument.drum.snare");
1305 assert_eq!(pts.len(), 1);
1306 assert!((pts[0].score - 0.0).abs() < f64::EPSILON);
1307 assert!(!pts[0].actual);
1308 }
1309
1310 #[test]
1311 fn k_sweep_always_contains_the_runtime_k() {
1312 // Safe to set: this test does not read the env, it checks the invariant
1313 // the parser must hold whatever the env said.
1314 let ks = k_sweep_from_env();
1315 assert!(ks.contains(&DEFAULT_K));
1316 assert!(ks.windows(2).all(|w| w[0] < w[1]), "sorted and deduped");
1317 }
1318 }
1319