Skip to main content

max / audiofiles

Add a cross-validated evaluation harness for the classifier layer layer-eval builds the labelled corpus into a scratch vault, then runs stratified k-fold cross-validation of the exemplar k-NN at the runtime k and thresholds, reporting per-class precision and recall plus a top-1 confusion matrix. The ship gate is a constant in the module rather than a number chosen after reading a run. The corpus import that afcl already did moves to labelled.rs, shared by both modes, and is now parallel (order-preserving, so the exported layer stays byte-identical). First run: 73.1% macro top-1, 84.5% macro precision at the auto threshold, and a gate failure on recall in four of seven classes.
Author: Max Johnson <me@maxj.phd> · 2026-08-07 02:27 UTC
Signed with PGP, not checked
Commit: 02395cb14b4d8c519d4741967921b0ff68284eb3
Parent: fec1d22
6 files changed, +871 insertions, -253 deletions
M README.md +2 -1
@@ -24,6 +24,7 @@
24 24 cargo run --release -p audiofiles-bench # analysis pipeline
25 25 cargo run --release -p audiofiles-bench -- ingest # import and query
26 26 cargo run --release -p audiofiles-bench -- accuracy # bpm/key vs ground truth
27 + cargo run --release -p audiofiles-bench -- layer-eval # classifier layer, cross-validated
27 28 ```
28 29
29 30 ## Workspace Architecture
@@ -37,7 +38,7 @@
37 38 | `audiofiles-app` | `crates/audiofiles-app/` | Standalone desktop app via eframe. System audio (cpal), drag-and-drop import, native drag-out to Finder/DAWs, system tray, CLI import, OTA updates. |
38 39 | `audiofiles-sync` | `crates/audiofiles-sync/` | Cloud sync via SyncKit. Pushes/pulls sample metadata, tags, and VFS structure across devices. E2E encrypted. |
39 40 | `audiofiles-rhai` | `crates/audiofiles-rhai/` | Rhai scripting engine for device export profiles. Transforms sample metadata and file layout for hardware samplers. |
40 - | `audiofiles-bench` | `crates/audiofiles-bench/` | Benchmark binary. Analysis-pipeline timing, vault ingest and query latency, and BPM/key accuracy against labeled corpora. Not shipped in the app. |
41 + | `audiofiles-bench` | `crates/audiofiles-bench/` | Benchmark binary. Analysis-pipeline timing, vault ingest and query latency, BPM/key accuracy against labeled corpora, and cross-validated per-class accuracy for the classifier layer. Not shipped in the app. |
41 42
42 43 Dependency flow: `audiofiles-core` is the leaf -> `audiofiles-rhai` and `audiofiles-sync` depend on core -> `audiofiles-browser` depends on core, sync, and rhai -> `audiofiles-app` depends on browser and core. `audiofiles-bench` depends on core only.
43 44
M docs/ml_classifier.md +35 -4
@@ -159,13 +159,44 @@
159 159 candidate source for the pitched families; it is 16 kHz mono against the drums' 44.1
160 160 kHz, and five of the 35 features are Nyquist-bounded, so the two cannot simply be mixed
161 161 without the classifier learning sample rate as a proxy for instrument.
162 - 2. **Nothing measures whether the layer is any good.** The accuracy bench scores BPM and
163 - key only; class accuracy went away with the retired sample-class label. A default
164 - classifier that is confidently wrong is worse than no default, so shipping needs a
165 - held-out measurement and a threshold agreed before the number is read.
162 + 2. **The layer does not clear its own ship gate.** It is measured now:
163 + `cargo run --release -p audiofiles-bench -- layer-eval` cross-validates the k-NN
164 + layer over the labelled corpus, five stratified folds, at the runtime `k` and the
165 + runtime thresholds. The gate lives in `layer_eval.rs` and was written before the
166 + first run.
166 167
167 168 Both are tracked in the audiofiles GoingsOn project.
168 169
170 + ### What the measurement says
171 +
172 + First run, 1049 files across the seven drum classes, features v5:
173 +
174 + | | top-1 recall | precision @ 0.85 | recall @ 0.85 | recall @ 0.50 |
175 + |---|---|---|---|---|
176 + | kick | 93.7% | 93.7% | 65.4% | 92.7% |
177 + | tom | 89.8% | 100% | 74.4% | 89.0% |
178 + | snare | 76.6% | 97.8% | 47.9% | 69.7% |
179 + | hi-hat | 71.6% | 100% | 5.5% | 56.0% |
180 + | clap | 70.8% | 100% | 6.2% | 47.9% |
181 + | cymbal | 62.8% | 100% | 2.3% | 44.2% |
182 + | percussion | 46.7% | never fires | 0.0% | 28.1% |
183 + | macro | 73.1% | 84.5% | 28.8% | 61.1% |
184 +
185 + The gate fails, and it fails on recall alone. Every class the layer auto-applies it
186 + auto-applies correctly (macro precision 84.5%, four classes at 100%), but only kick,
187 + snare and tom reach the 0.85 auto threshold often enough to be useful, and percussion
188 + never reaches it at all. The k-NN is conservative rather than wrong.
189 +
190 + Read against the retired threshold tree, this is a different result: 73.1% macro top-1
191 + against that tree's 33.4% strict, with no unreachable class. Instrument identity is
192 + more present in the 35-feature vector under k-NN than it was under thresholds. It is
193 + still not present enough for a default that writes tags unasked, which is the
194 + distinction the auto and review columns above draw.
195 +
196 + Percussion is the class to read first. It is a catch-all (cowbell, clave, maraca,
197 + bongo, conga) rather than one instrument, and the confusion matrix scatters it across
198 + every other row. A taxonomy fix, not a tuning problem.
199 +
169 200 ## Feature vector
170 201
171 202 35 features: 9 scalar + 13 MFCC means + 13 MFCC variances.
M crates/audiofiles-bench/src/afcl_gen.rs +10 -248
@@ -5,10 +5,9 @@
5 5 //! duplicating both in a second binary to keep the crate's name honest would cost
6 6 //! more than the misnomer does.
7 7 //!
8 - //! What it does: import every labelled file into a throwaway vault, analyse it so
9 - //! `sample_features` holds a vector, tag it from the folder it came in, then call
10 - //! [`build_export`] with `kind = "official"`. The exemplars in the resulting layer
11 - //! are (vector, tags) pairs and carry no audio and no hashes, which is the whole
8 + //! What it does: build the labelled scratch vault ([`crate::labelled`]) and export
9 + //! it with `kind = "official"`. The exemplars in the resulting layer are
10 + //! (vector, tags) pairs and carry no audio and no hashes, which is the whole
12 11 //! reason a `.afcl` can ship at all.
13 12 //!
14 13 //! The output is a build artifact, not a test fixture. Generate it deliberately,
@@ -16,44 +15,17 @@
16 15 //! on `feat_version` and a layer built under an old one is rejected outright, so a
17 16 //! stale layer is worse than no layer.
18 17
19 - use std::collections::BTreeMap;
20 18 use std::path::{Path, PathBuf};
21 - use std::time::Instant;
22 19
23 20 use audiofiles_core::analysis::afcl::{ExportOptions, LayerKind, export_to_path};
21 + use audiofiles_core::analysis::config::AnalysisConfig;
24 22 use audiofiles_core::analysis::features::FEATURE_VERSION;
25 - use audiofiles_core::analysis::{self, config::AnalysisConfig};
26 - use audiofiles_core::db::Database;
27 - use audiofiles_core::starter_rules::DRUM_CLASSES;
28 - use audiofiles_core::store::SampleStore;
29 - use audiofiles_core::tags;
30 23
31 - /// Corpus subdirectory holding the labelled one-shots, one folder per class.
32 - const TRAINING_SUBDIR: &str = "training";
24 + use crate::labelled;
33 25
34 26 /// Corpus manifest naming the source datasets and their licences.
35 27 const MANIFEST: &str = "MANIFEST.json";
36 28
37 - /// Map a corpus folder name to its canonical tag.
38 - ///
39 - /// The folder names and [`DRUM_CLASSES`] are the same taxonomy, so this resolves
40 - /// against that table rather than carrying a second copy of it. The only mismatch
41 - /// is punctuation: the table labels the class `hi-hat` and the corpus folder is
42 - /// `hihat`, so both sides are compared with `-` and spaces removed.
43 - fn tag_for_folder(folder: &str) -> Option<&'static str> {
44 - fn squash(s: &str) -> String {
45 - s.chars()
46 - .filter(|c| !matches!(c, '-' | ' ' | '_'))
47 - .flat_map(char::to_lowercase)
48 - .collect()
49 - }
50 - let want = squash(folder);
51 - DRUM_CLASSES
52 - .iter()
53 - .find(|c| squash(c.label) == want)
54 - .map(|c| c.tag)
55 - }
56 -
57 29 /// Read the corpus manifest into an attribution line for `license_note`.
58 30 ///
59 31 /// This is the CC-BY attribution, and it is the only place it travels: an `.afcl`
@@ -153,54 +125,6 @@
153 125 ))
154 126 }
155 127
156 - /// Collect the labelled files as (path, tag) pairs, grouped for a stable report.
157 - fn collect_labelled(training: &Path) -> Result<BTreeMap<&'static str, Vec<PathBuf>>, String> {
158 - let entries = std::fs::read_dir(training)
159 - .map_err(|e| format!("cannot read {}: {e}", training.display()))?;
160 -
161 - let mut by_tag: BTreeMap<&'static str, Vec<PathBuf>> = BTreeMap::new();
162 - let mut unmapped = Vec::new();
163 - for entry in entries.flatten() {
164 - if !entry.path().is_dir() {
165 - continue;
166 - }
167 - let folder = entry.file_name().to_string_lossy().to_string();
168 - let Some(tag) = tag_for_folder(&folder) else {
169 - unmapped.push(folder);
170 - continue;
171 - };
172 - let mut files: Vec<PathBuf> = std::fs::read_dir(entry.path())
173 - .map_err(|e| format!("cannot read {}: {e}", entry.path().display()))?
174 - .flatten()
175 - .map(|f| f.path())
176 - .filter(|p| p.is_file())
177 - .collect();
178 - // Deterministic order so two runs over the same corpus produce the same
179 - // layer, which is what makes the checked-in artifact reviewable.
180 - files.sort();
181 - by_tag.entry(tag).or_default().extend(files);
182 - }
183 -
184 - if !unmapped.is_empty() {
185 - // Loud rather than silent: a folder nobody mapped is a class silently
186 - // missing from the shipped layer, which reads downstream as the
187 - // classifier being bad at that class rather than never having seen it.
188 - unmapped.sort();
189 - return Err(format!(
190 - "no tag mapping for corpus folder(s): {}. Add them to DRUM_CLASSES or move them out of {}",
191 - unmapped.join(", "),
192 - training.display()
193 - ));
194 - }
195 - if by_tag.is_empty() {
196 - return Err(format!(
197 - "no labelled class folders under {}",
198 - training.display()
199 - ));
200 - }
201 - Ok(by_tag)
202 - }
203 -
204 128 /// Generate the official layer from `corpus` into `out`, using `vault` as scratch.
205 129 pub(crate) fn run(corpus: &Path, vault: &Path, out: &Path, config: &AnalysisConfig) {
206 130 println!("━━━ OFFICIAL .afcl GENERATION ━━━");
@@ -219,111 +143,14 @@
219 143 }
220 144 };
221 145
222 - let by_tag = match collect_labelled(&corpus.join(TRAINING_SUBDIR)) {
223 - Ok(m) => m,
146 + let vault = match labelled::build_vault(corpus, vault, config) {
147 + Ok(v) => v,
224 148 Err(e) => {
225 149 eprintln!("corpus: {e}");
226 150 std::process::exit(1);
227 151 }
228 152 };
229 -
230 - let total: usize = by_tag.values().map(Vec::len).sum();
231 - println!(
232 - " {} labelled file(s) across {} class(es):",
233 - total,
234 - by_tag.len()
235 - );
236 - for (tag, files) in &by_tag {
237 - println!(" {:<28} {:>5}", tag, files.len());
238 - }
239 - println!();
240 -
241 - // A fresh vault every run. The layer is a pure function of the corpus, and
242 - // leftovers from a previous run would silently widen it.
243 - if vault.exists()
244 - && let Err(e) = std::fs::remove_dir_all(vault)
245 - {
246 - eprintln!("could not clear scratch vault {}: {e}", vault.display());
247 - std::process::exit(1);
248 - }
249 - let samples_dir = vault.join("samples");
250 - if let Err(e) = std::fs::create_dir_all(&samples_dir) {
251 - eprintln!("could not create {}: {e}", samples_dir.display());
252 - std::process::exit(1);
253 - }
254 -
255 - let db = match Database::open(vault.join("audiofiles.db")) {
256 - Ok(db) => db,
257 - Err(e) => {
258 - eprintln!("Database::open failed: {e}");
259 - std::process::exit(1);
260 - }
261 - };
262 - let store = match SampleStore::new(&samples_dir) {
263 - Ok(s) => s,
264 - Err(e) => {
265 - eprintln!("SampleStore::new failed: {e}");
266 - std::process::exit(1);
267 - }
268 - };
269 -
270 - // Import and tag. No VFS nodes: nothing here runs a UI query, and the export
271 - // reads `sample_features` joined to `tags`, neither of which needs one.
272 - let start = Instant::now();
273 - let mut to_analyze: Vec<(String, PathBuf)> = Vec::with_capacity(total);
274 - let mut import_failures = 0usize;
275 - for (tag, files) in &by_tag {
276 - for path in files {
277 - match store.import(path, &db) {
278 - Ok(hash) => {
279 - if let Err(e) = tags::add_tag(&db, &hash, tag) {
280 - eprintln!(" tag {tag} on {}: {e}", path.display());
281 - import_failures += 1;
282 - continue;
283 - }
284 - to_analyze.push((hash, path.clone()));
285 - }
286 - Err(e) => {
287 - eprintln!(" import {}: {e}", path.display());
288 - import_failures += 1;
289 - }
290 - }
291 - }
292 - }
293 - println!(
294 - " imported {} file(s) in {:.1}s{}",
295 - to_analyze.len(),
296 - start.elapsed().as_secs_f64(),
297 - if import_failures > 0 {
298 - format!(", {import_failures} failed")
299 - } else {
300 - String::new()
301 - }
302 - );
303 -
304 - // Analyse everything. The ingest benchmark caps this because analysis is the
305 - // expensive stage and it only needs a sample; here every vector is the
306 - // payload, so there is no budget to apply.
307 - let start = Instant::now();
308 - let results: Vec<_> = to_analyze
309 - .iter()
310 - .filter_map(|(hash, path)| analysis::analyze_sample(hash, path, config).ok())
311 - .collect();
312 - let analyzed = results.len();
313 - if let Err(e) = analysis::save_analysis_batch(&db, &results) {
314 - eprintln!("save_analysis_batch failed: {e}");
315 - std::process::exit(1);
316 - }
317 - println!(
318 - " analysed {} file(s) in {:.1}s{}",
319 - analyzed,
320 - start.elapsed().as_secs_f64(),
321 - if analyzed < to_analyze.len() {
322 - format!(", {} failed to analyse", to_analyze.len() - analyzed)
323 - } else {
324 - String::new()
325 - }
326 - );
153 + let analyzed = vault.analyzed;
327 154
328 155 // Exemplars only. Rules and policy are deliberately excluded: the app already
329 156 // ships `starter_rules` in the binary, so exporting them here would put a
@@ -332,7 +159,7 @@
332 159 name: "Official drum classifier".to_string(),
333 160 description: format!(
334 161 "Bundled default layer. {analyzed} exemplars across {} classes, built from the labelled corpus.",
335 - by_tag.len()
162 + vault.by_tag.len()
336 163 ),
337 164 license_note: note,
338 165 kind: LayerKind::Official,
@@ -347,7 +174,7 @@
347 174 eprintln!("could not create {}: {e}", parent.display());
348 175 std::process::exit(1);
349 176 }
350 - if let Err(e) = export_to_path(&db, out, &opts) {
177 + if let Err(e) = export_to_path(&vault.db, out, &opts) {
351 178 eprintln!("export failed: {e}");
352 179 std::process::exit(1);
353 180 }
@@ -374,44 +201,6 @@
374 201 mod tests {
375 202 use super::*;
376 203
377 - #[test]
378 - fn folder_names_map_to_canonical_tags() {
379 - assert_eq!(tag_for_folder("kick"), Some("instrument.drum.kick"));
380 - assert_eq!(tag_for_folder("snare"), Some("instrument.drum.snare"));
381 - assert_eq!(tag_for_folder("cymbal"), Some("instrument.drum.cymbal"));
382 - assert_eq!(tag_for_folder("clap"), Some("instrument.drum.clap"));
383 - assert_eq!(tag_for_folder("tom"), Some("instrument.drum.tom"));
384 - assert_eq!(tag_for_folder("percussion"), Some("instrument.percussion"));
385 - }
386 -
387 - #[test]
388 - fn hihat_folder_matches_the_hi_hat_label() {
389 - // The one place the corpus and DRUM_CLASSES disagree on spelling.
390 - assert_eq!(tag_for_folder("hihat"), Some("instrument.drum.hihat"));
391 - assert_eq!(tag_for_folder("hi-hat"), Some("instrument.drum.hihat"));
392 - assert_eq!(tag_for_folder("Hi Hat"), Some("instrument.drum.hihat"));
393 - }
394 -
395 - #[test]
396 - fn unknown_folder_has_no_tag() {
397 - assert_eq!(tag_for_folder("bass"), None);
398 - assert_eq!(tag_for_folder(""), None);
399 - }
400 -
401 - #[test]
402 - fn every_drum_class_is_reachable_from_some_folder_name() {
403 - // Guards the mapping against a DRUM_CLASSES entry whose label stops
404 - // resolving; without this, a renamed label silently drops a class.
405 - for class in DRUM_CLASSES {
406 - assert_eq!(
407 - tag_for_folder(class.label),
408 - Some(class.tag),
409 - "class {} no longer resolves from its own label",
410 - class.label
411 - );
412 - }
413 - }
414 -
415 204 #[test]
416 205 fn license_note_requires_a_manifest() {
417 206 let dir = tempfile::tempdir().unwrap();
@@ -512,31 +301,4 @@
512 301 let err = license_note(dir.path()).unwrap_err();
513 302 assert!(err.contains("training_source"), "{err}");
514 303 }
515 -
516 - #[test]
517 - fn collect_labelled_rejects_an_unmapped_folder() {
518 - let dir = tempfile::tempdir().unwrap();
519 - let training = dir.path().join(TRAINING_SUBDIR);
520 - std::fs::create_dir_all(training.join("kick")).unwrap();
521 - std::fs::create_dir_all(training.join("didgeridoo")).unwrap();
522 - let err = collect_labelled(&training).unwrap_err();
523 - assert!(err.contains("didgeridoo"), "{err}");
524 - }
525 -
526 - #[test]
527 - fn collect_labelled_groups_files_under_their_tag() {
528 - let dir = tempfile::tempdir().unwrap();
529 - let training = dir.path().join(TRAINING_SUBDIR);
530 - std::fs::create_dir_all(training.join("kick")).unwrap();
531 - std::fs::create_dir_all(training.join("snare")).unwrap();
532 - std::fs::write(training.join("kick/b.wav"), b"x").unwrap();
533 - std::fs::write(training.join("kick/a.wav"), b"x").unwrap();
534 - std::fs::write(training.join("snare/c.wav"), b"x").unwrap();
535 -
536 - let got = collect_labelled(&training).unwrap();
537 - assert_eq!(got["instrument.drum.kick"].len(), 2);
538 - assert_eq!(got["instrument.drum.snare"].len(), 1);
539 - // Sorted, so the artifact is reproducible across runs.
540 - assert!(got["instrument.drum.kick"][0].ends_with("a.wav"));
541 - }
542 304 }
M crates/audiofiles-bench/src/main.rs +26
@@ -12,17 +12,24 @@
12 12 //! `cargo run --release -p audiofiles-bench -- accuracy` bpm/key vs ground truth
13 13 //! `cargo run --release -p audiofiles-bench -- layout` blob layout migration
14 14 //! `cargo run --release -p audiofiles-bench -- afcl` build the official layer
15 + //! `cargo run --release -p audiofiles-bench -- layer-eval` cross-validate that layer
15 16 //!
16 17 //! Two modes are not measurements. `layout` is a checker: it fabricates flat
17 18 //! vaults, sweeps them, and exits non-zero if any scenario fails. `afcl` is a
18 19 //! generator: it turns the labelled corpus into the bundled official `.afcl`.
19 20 //! Both live here because corpus walking and scratch-vault fabrication do.
20 21 //!
22 + //! `layer-eval` is the meter for what `afcl` generates: stratified k-fold
23 + //! cross-validation of the k-NN layer over the same corpus, per class, against a
24 + //! ship gate written down before the first run. It shares `afcl`'s corpus import
25 + //! (see `labelled`), which is why it lives beside it.
26 + //!
21 27 //! Env: `AF_BENCH_CORPUS` (corpus root, default `<repo>/samples`),
22 28 //! `AF_BENCH_VAULT` (scratch vault for ingest and layout), `AF_BENCH_FSL10K`
23 29 //! (FSL10K root for accuracy), `AF_BENCH_BATCH`, `AF_BENCH_LIMIT`,
24 30 //! `AF_BENCH_ANALYZE`, `AF_BENCH_LAYOUT_N`, `AF_BENCH_JSON` (machine-readable
25 31 //! output path), `AF_AFCL_OUT` (where `afcl` writes the layer),
32 + //! `AF_BENCH_EVAL_FOLDS` (cross-validation folds for `layer-eval`),
26 33 //! `AF_BENCH_STAGES` (files per per-stage probe during `ingest`, 0 = off).
27 34 //!
28 35 //! Section 1 times the analysis stages per file, against the corpus and no
@@ -39,6 +46,8 @@
39 46 mod accuracy;
40 47 mod afcl_gen;
41 48 mod ingest;
49 + mod labelled;
50 + mod layer_eval;
42 51 mod layout;
43 52 mod report;
44 53 mod storage;
@@ -326,6 +335,23 @@
326 335 return;
327 336 }
328 337
338 + if args.first().map(String::as_str) == Some("layer-eval") {
339 + // Its own scratch vault, distinct from `afcl`'s: the two modes rebuild
340 + // the same corpus and running one must not delete the other's vault out
341 + // from under a comparison.
342 + let vault = std::env::var("AF_BENCH_VAULT").map_or_else(
343 + |_| std::env::temp_dir().join("af-layer-eval"),
344 + PathBuf::from,
345 + );
346 + layer_eval::run(
347 + &samples_dir,
348 + &vault,
349 + &full_pipeline_config(),
350 + layer_eval::folds_from_env(),
351 + );
352 + return;
353 + }
354 +
329 355 if args.first().map(String::as_str) == Some("accuracy") {
330 356 let Ok(root) = std::env::var("AF_BENCH_FSL10K") else {
331 357 eprintln!("set AF_BENCH_FSL10K to the extracted FSL10K root");
A crates/audiofiles-bench/src/labelled.rs +298
@@ -1,0 +1,298 @@
1 + //! The labelled corpus: folder-to-tag mapping, and the scratch vault built from it.
2 + //!
3 + //! Two modes need the same thing before they can do anything: every file under
4 + //! `training/` imported, analysed, and tagged from the folder it came in, so
5 + //! `sample_features` holds a vector next to a label. `afcl` exports that state as
6 + //! a layer; `layer-eval` cross-validates over it. The step is expensive enough
7 + //! (decode plus full analysis per file) that it is worth sharing, and subtle
8 + //! enough (a fresh vault every run, a fatal unmapped folder) that two copies
9 + //! would drift.
10 +
11 + use std::collections::BTreeMap;
12 + use std::path::{Path, PathBuf};
13 + use std::time::Instant;
14 +
15 + use audiofiles_core::analysis::{self, config::AnalysisConfig};
16 + use audiofiles_core::db::Database;
17 + use audiofiles_core::starter_rules::DRUM_CLASSES;
18 + use audiofiles_core::store::SampleStore;
19 + use audiofiles_core::tags;
20 + use rayon::prelude::*;
21 +
22 + /// Corpus subdirectory holding the labelled one-shots, one folder per class.
23 + pub(crate) const TRAINING_SUBDIR: &str = "training";
24 +
25 + /// Map a corpus folder name to its canonical tag.
26 + ///
27 + /// The folder names and [`DRUM_CLASSES`] are the same taxonomy, so this resolves
28 + /// against that table rather than carrying a second copy of it. The only mismatch
29 + /// is punctuation: the table labels the class `hi-hat` and the corpus folder is
30 + /// `hihat`, so both sides are compared with `-` and spaces removed.
31 + pub(crate) fn tag_for_folder(folder: &str) -> Option<&'static str> {
32 + fn squash(s: &str) -> String {
33 + s.chars()
34 + .filter(|c| !matches!(c, '-' | ' ' | '_'))
35 + .flat_map(char::to_lowercase)
36 + .collect()
37 + }
38 + let want = squash(folder);
39 + DRUM_CLASSES
40 + .iter()
41 + .find(|c| squash(c.label) == want)
42 + .map(|c| c.tag)
43 + }
44 +
45 + /// The short class label for a canonical tag (`instrument.drum.kick` -> `kick`).
46 + ///
47 + /// Report tables are unreadable at full tag width once there is a column per
48 + /// class, and the confusion matrix is square in the number of classes.
49 + pub(crate) fn label_for_tag(tag: &str) -> &str {
50 + DRUM_CLASSES
51 + .iter()
52 + .find(|c| c.tag == tag)
53 + .map_or(tag, |c| c.label)
54 + }
55 +
56 + /// Collect the labelled files as (path, tag) pairs, grouped for a stable report.
57 + pub(crate) fn collect_labelled(
58 + training: &Path,
59 + ) -> Result<BTreeMap<&'static str, Vec<PathBuf>>, String> {
60 + let entries = std::fs::read_dir(training)
61 + .map_err(|e| format!("cannot read {}: {e}", training.display()))?;
62 +
63 + let mut by_tag: BTreeMap<&'static str, Vec<PathBuf>> = BTreeMap::new();
64 + let mut unmapped = Vec::new();
65 + for entry in entries.flatten() {
66 + if !entry.path().is_dir() {
67 + continue;
68 + }
69 + let folder = entry.file_name().to_string_lossy().to_string();
70 + let Some(tag) = tag_for_folder(&folder) else {
71 + unmapped.push(folder);
72 + continue;
73 + };
74 + let mut files: Vec<PathBuf> = std::fs::read_dir(entry.path())
75 + .map_err(|e| format!("cannot read {}: {e}", entry.path().display()))?
76 + .flatten()
77 + .map(|f| f.path())
78 + .filter(|p| p.is_file())
79 + .collect();
80 + // Deterministic order so two runs over the same corpus produce the same
81 + // layer, which is what makes the checked-in artifact reviewable.
82 + files.sort();
83 + by_tag.entry(tag).or_default().extend(files);
84 + }
85 +
86 + if !unmapped.is_empty() {
87 + // Loud rather than silent: a folder nobody mapped is a class silently
88 + // missing from the shipped layer, which reads downstream as the
89 + // classifier being bad at that class rather than never having seen it.
90 + unmapped.sort();
91 + return Err(format!(
92 + "no tag mapping for corpus folder(s): {}. Add them to DRUM_CLASSES or move them out of {}",
93 + unmapped.join(", "),
94 + training.display()
95 + ));
96 + }
97 + if by_tag.is_empty() {
98 + return Err(format!(
99 + "no labelled class folders under {}",
100 + training.display()
101 + ));
102 + }
103 + Ok(by_tag)
104 + }
105 +
106 + /// A scratch vault holding the analysed, tagged corpus.
107 + pub(crate) struct LabelledVault {
108 + pub(crate) db: Database,
109 + /// Files per class, as found on disk.
110 + pub(crate) by_tag: BTreeMap<&'static str, Vec<PathBuf>>,
111 + /// Files that produced a feature vector.
112 + pub(crate) analyzed: usize,
113 + }
114 +
115 + /// Import, analyse and tag every labelled file into a fresh vault at `vault`.
116 + ///
117 + /// Prints its own progress: both callers are long-running terminal modes and
118 + /// this is the slow part of each.
119 + pub(crate) fn build_vault(
120 + corpus: &Path,
121 + vault: &Path,
122 + config: &AnalysisConfig,
123 + ) -> Result<LabelledVault, String> {
124 + let by_tag = collect_labelled(&corpus.join(TRAINING_SUBDIR))?;
125 +
126 + let total: usize = by_tag.values().map(Vec::len).sum();
127 + println!(
128 + " {} labelled file(s) across {} class(es):",
129 + total,
130 + by_tag.len()
131 + );
132 + for (tag, files) in &by_tag {
133 + println!(" {:<28} {:>5}", tag, files.len());
134 + }
135 + println!();
136 +
137 + // A fresh vault every run. What is built from it is a pure function of the
138 + // corpus, and leftovers from a previous run would silently widen it.
139 + if vault.exists() {
140 + std::fs::remove_dir_all(vault)
141 + .map_err(|e| format!("could not clear scratch vault {}: {e}", vault.display()))?;
142 + }
143 + let samples_dir = vault.join("samples");
144 + std::fs::create_dir_all(&samples_dir)
145 + .map_err(|e| format!("could not create {}: {e}", samples_dir.display()))?;
146 +
147 + let db = Database::open(vault.join("audiofiles.db"))
148 + .map_err(|e| format!("Database::open failed: {e}"))?;
149 + let store =
150 + SampleStore::new(&samples_dir).map_err(|e| format!("SampleStore::new failed: {e}"))?;
151 +
152 + // Import and tag. No VFS nodes: nothing here runs a UI query, and neither
153 + // the export nor the eval needs one; both read `sample_features` joined to
154 + // `tags`.
155 + let start = Instant::now();
156 + let mut to_analyze: Vec<(String, PathBuf)> = Vec::with_capacity(total);
157 + let mut import_failures = 0usize;
158 + for (tag, files) in &by_tag {
159 + for path in files {
160 + match store.import(path, &db) {
161 + Ok(hash) => {
162 + if let Err(e) = tags::add_tag(&db, &hash, tag) {
163 + eprintln!(" tag {tag} on {}: {e}", path.display());
164 + import_failures += 1;
165 + continue;
166 + }
167 + to_analyze.push((hash, path.clone()));
168 + }
169 + Err(e) => {
170 + eprintln!(" import {}: {e}", path.display());
171 + import_failures += 1;
172 + }
173 + }
174 + }
175 + }
176 + println!(
177 + " imported {} file(s) in {:.1}s{}",
178 + to_analyze.len(),
179 + start.elapsed().as_secs_f64(),
180 + if import_failures > 0 {
181 + format!(", {import_failures} failed")
182 + } else {
183 + String::new()
184 + }
185 + );
186 +
187 + // Analyse everything. The ingest benchmark caps this because analysis is the
188 + // expensive stage and it only needs a sample; here every vector is the
189 + // payload, so there is no budget to apply.
190 + //
191 + // Parallel, but still deterministic: `analyze_sample` is a pure function of
192 + // the file, and a rayon `collect` into a Vec restores input order, so the
193 + // batch written to the DB is the same sequence a serial run would write.
194 + // That matters because the exported layer is a checked-in artifact.
195 + let start = Instant::now();
196 + let results: Vec<_> = to_analyze
197 + .par_iter()
198 + .filter_map(|(hash, path)| analysis::analyze_sample(hash, path, config).ok())
199 + .collect();
200 + let analyzed = results.len();
201 + analysis::save_analysis_batch(&db, &results)
202 + .map_err(|e| format!("save_analysis_batch failed: {e}"))?;
203 + println!(
204 + " analysed {} file(s) in {:.1}s{}",
205 + analyzed,
206 + start.elapsed().as_secs_f64(),
207 + if analyzed < to_analyze.len() {
208 + format!(", {} failed to analyse", to_analyze.len() - analyzed)
209 + } else {
210 + String::new()
211 + }
212 + );
213 +
214 + Ok(LabelledVault {
215 + db,
216 + by_tag,
217 + analyzed,
218 + })
219 + }
220 +
221 + #[cfg(test)]
222 + mod tests {
223 + use super::*;
224 +
225 + #[test]
226 + fn folder_names_map_to_canonical_tags() {
227 + assert_eq!(tag_for_folder("kick"), Some("instrument.drum.kick"));
228 + assert_eq!(tag_for_folder("snare"), Some("instrument.drum.snare"));
229 + assert_eq!(tag_for_folder("cymbal"), Some("instrument.drum.cymbal"));
230 + assert_eq!(tag_for_folder("clap"), Some("instrument.drum.clap"));
231 + assert_eq!(tag_for_folder("tom"), Some("instrument.drum.tom"));
232 + assert_eq!(tag_for_folder("percussion"), Some("instrument.percussion"));
233 + }
234 +
235 + #[test]
236 + fn hihat_folder_matches_the_hi_hat_label() {
237 + // The one place the corpus and DRUM_CLASSES disagree on spelling.
238 + assert_eq!(tag_for_folder("hihat"), Some("instrument.drum.hihat"));
239 + assert_eq!(tag_for_folder("hi-hat"), Some("instrument.drum.hihat"));
240 + assert_eq!(tag_for_folder("Hi Hat"), Some("instrument.drum.hihat"));
241 + }
242 +
243 + #[test]
244 + fn unknown_folder_has_no_tag() {
245 + assert_eq!(tag_for_folder("bass"), None);
246 + assert_eq!(tag_for_folder(""), None);
247 + }
248 +
249 + #[test]
250 + fn every_drum_class_is_reachable_from_some_folder_name() {
251 + // Guards the mapping against a DRUM_CLASSES entry whose label stops
252 + // resolving; without this, a renamed label silently drops a class.
253 + for class in DRUM_CLASSES {
254 + assert_eq!(
255 + tag_for_folder(class.label),
256 + Some(class.tag),
257 + "class {} no longer resolves from its own label",
258 + class.label
259 + );
260 + }
261 + }
262 +
263 + #[test]
264 + fn labels_round_trip_from_their_tag() {
265 + for class in DRUM_CLASSES {
266 + assert_eq!(label_for_tag(class.tag), class.label);
267 + }
268 + // An unknown tag prints as itself rather than vanishing.
269 + assert_eq!(label_for_tag("instrument.bass"), "instrument.bass");
270 + }
271 +
272 + #[test]
273 + fn collect_labelled_rejects_an_unmapped_folder() {
274 + let dir = tempfile::tempdir().unwrap();
275 + let training = dir.path().join(TRAINING_SUBDIR);
276 + std::fs::create_dir_all(training.join("kick")).unwrap();
277 + std::fs::create_dir_all(training.join("didgeridoo")).unwrap();
278 + let err = collect_labelled(&training).unwrap_err();
279 + assert!(err.contains("didgeridoo"), "{err}");
280 + }
281 +
282 + #[test]
283 + fn collect_labelled_groups_files_under_their_tag() {
284 + let dir = tempfile::tempdir().unwrap();
285 + let training = dir.path().join(TRAINING_SUBDIR);
286 + std::fs::create_dir_all(training.join("kick")).unwrap();
287 + std::fs::create_dir_all(training.join("snare")).unwrap();
288 + std::fs::write(training.join("kick/b.wav"), b"x").unwrap();
289 + std::fs::write(training.join("kick/a.wav"), b"x").unwrap();
290 + std::fs::write(training.join("snare/c.wav"), b"x").unwrap();
291 +
292 + let got = collect_labelled(&training).unwrap();
293 + assert_eq!(got["instrument.drum.kick"].len(), 2);
294 + assert_eq!(got["instrument.drum.snare"].len(), 1);
295 + // Sorted, so the artifact is reproducible across runs.
296 + assert!(got["instrument.drum.kick"][0].ends_with("a.wav"));
297 + }
298 + }
A crates/audiofiles-bench/src/layer_eval.rs +500
@@ -1,0 +1,855 @@
1 + //! Cross-validated evaluation of the exemplar k-NN layer over the labelled corpus.
2 + //!
3 + //! This is the meter the bundled `.afcl` never had. `afcl_gen` builds a layer;
4 + //! nothing said whether that layer answers correctly, so "tuned enough to ship"
5 + //! had no number behind it and the layer sits behind an off-by-default feature.
6 + //!
7 + //! What it measures: stratified k-fold cross-validation of [`exemplar`] over the
8 + //! same corpus the layer is built from, at the same `k` and the same thresholds
9 + //! the app uses at runtime. Each fold builds a real [`ExemplarIndex`] from the
10 + //! other folds and scores this fold's samples against it, so no sample is ever a
11 + //! neighbour of itself and the standardization params are fitted on training data
12 + //! alone.
13 + //!
14 + //! Why it is not one accuracy figure: the retired threshold classifier scored
15 + //! 33.4% strict with two of its seven classes unreachable by any rule, and a
16 + //! single number is exactly what hid that. Everything here is per class. A class
17 + //! the layer never predicts shows up as a zero row in the confusion matrix and a
18 + //! zero recall, both of which a headline average would smooth away.
19 + //!
20 + //! The ship gate ([`GATE`]) is written down in this file rather than decided
21 + //! after reading a run. That ordering is the whole point: a threshold chosen once
22 + //! the number is on screen is a threshold the number chose.
23 + //!
24 + //! Usage: `cargo run --release -p audiofiles-bench -- layer-eval`
25 + //! Env: `AF_BENCH_CORPUS`, `AF_BENCH_VAULT`, `AF_BENCH_EVAL_FOLDS` (default 5),
26 + //! `AF_BENCH_JSON`.
27 +
28 + use std::collections::{BTreeMap, BTreeSet, HashMap};
29 + use std::path::Path;
30 +
31 + use audiofiles_core::analysis::config::AnalysisConfig;
32 + use audiofiles_core::analysis::exemplar::{
33 + self, DEFAULT_AUTO_THRESHOLD, DEFAULT_K, DEFAULT_REVIEW_THRESHOLD,
34 + };
35 + use audiofiles_core::analysis::features::{FEATURE_VERSION, NUM_FEATURES};
36 + use audiofiles_core::db::Database;
37 +
38 + use crate::labelled::{self, label_for_tag};
39 + use crate::report::Report;
40 +
41 + /// Default fold count.
42 + ///
43 + /// Five over ~1000 files leaves every training fold within a fifth of the size of
44 + /// the layer that actually ships, so the measured neighbourhood density is close
45 + /// to the real one. Fewer folds would train on visibly less data than ships and
46 + /// understate the layer; many more would leave the smallest class (clap, 48
47 + /// files) with single-digit test sets whose per-class recall moves in 10% steps.
48 + const DEFAULT_FOLDS: usize = 5;
49 +
50 + /// The ship gate, decided before the first run.
51 + ///
52 + /// A bundled default that is confidently wrong is worse than no default, so the
53 + /// binding criterion is precision at the auto-apply threshold: that is the score
54 + /// at which the layer writes a tag into someone's library without being asked.
55 + /// Recall is graded far more loosely, because a tag the layer declines to apply
56 + /// costs a user nothing they did not already have.
57 + ///
58 + /// The zero-prediction check is separate from recall on purpose. A class the
59 + /// layer never reaches is a different failure from a class it reaches badly: it
60 + /// means the shipped taxonomy promises something the data cannot deliver, and it
61 + /// is what the retired threshold tree did to clap and tom.
62 + struct Gate {
63 + /// Per-class precision at [`DEFAULT_AUTO_THRESHOLD`], over classes the layer
64 + /// predicts at all. A wrong auto-applied tag is the harm this layer can do.
65 + auto_precision: f64,
66 + /// Per-class recall at [`DEFAULT_AUTO_THRESHOLD`]. Deliberately low: a
67 + /// default that fires on a third of a class is still a head start.
68 + auto_recall: f64,
69 + /// Macro-averaged top-1 recall. Below this the feature space is not
70 + /// separating these classes at all, whatever the thresholds do.
71 + top1_macro_recall: f64,
72 + }
73 +
74 + const GATE: Gate = Gate {
75 + auto_precision: 0.80,
76 + auto_recall: 0.35,
77 + top1_macro_recall: 0.60,
78 + };
79 +
80 + /// One corpus sample: its vector, its labels, and the class it is scored against.
81 + struct Row {
82 + hash: String,
83 + vector: Vec<f64>,
84 + tags: Vec<String>,
85 + /// The single class this row is ground truth for, or `None` when the corpus
86 + /// gave it more than one. Content-addressed import collapses a file that
87 + /// appears in two class folders into one row carrying both tags; its true
88 + /// class is undecidable, so it trains but is never tested.
89 + truth: Option<String>,
90 + }
91 +
92 + /// Per-class counts at one score threshold, over the whole cross-validation.
93 + #[derive(Default, Clone, Copy)]
94 + struct Counts {
95 + tp: usize,
96 + fp: usize,
97 + fn_: usize,
98 + }
99 +
100 + impl Counts {
101 + fn precision(self) -> Option<f64> {
102 + let predicted = self.tp + self.fp;
103 + (predicted > 0).then(|| self.tp as f64 / predicted as f64)
104 + }
105 + fn recall(self) -> Option<f64> {
106 + let actual = self.tp + self.fn_;
107 + (actual > 0).then(|| self.tp as f64 / actual as f64)
108 + }
109 + fn f1(self) -> Option<f64> {
110 + let (p, r) = (self.precision()?, self.recall()?);
111 + (p + r > 0.0).then(|| 2.0 * p * r / (p + r))
112 + }
113 + }
114 +
115 + fn pct(v: Option<f64>) -> String {
116 + v.map_or_else(|| "-".to_string(), |x| format!("{:.1}%", x * 100.0))
117 + }
118 +
119 + /// Read the analysed corpus back out of the scratch vault as scoreable rows.
120 + fn load_rows(db: &Database) -> Result<Vec<Row>, String> {
121 + let conn = db.conn();
122 +
123 + let mut tags_by_hash: HashMap<String, Vec<String>> = HashMap::new();
124 + {
125 + let mut stmt = conn
126 + .prepare("SELECT sample_hash, tag FROM tags")
127 + .map_err(|e| format!("tags query: {e}"))?;
128 + let rows = stmt
129 + .query_map([], |row| {
130 + Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))
131 + })
132 + .map_err(|e| format!("tags query: {e}"))?;
133 + for r in rows {
134 + let (hash, tag) = r.map_err(|e| format!("tags row: {e}"))?;
135 + tags_by_hash.entry(hash).or_default().push(tag);
136 + }
137 + }
138 +
139 + let mut out = Vec::new();
140 + let mut stmt = conn
141 + .prepare("SELECT hash, vector FROM sample_features WHERE feat_version = ?1 ORDER BY hash")
142 + .map_err(|e| format!("features query: {e}"))?;
143 + let rows = stmt
144 + .query_map([FEATURE_VERSION], |row| {
145 + Ok((row.get::<_, String>(0)?, row.get::<_, String>(1)?))
146 + })
147 + .map_err(|e| format!("features query: {e}"))?;
148 + for r in rows {
149 + let (hash, json) = r.map_err(|e| format!("features row: {e}"))?;
150 + let Some(mut tags) = tags_by_hash.remove(&hash) else {
151 + continue;
152 + };
153 + tags.sort();
154 + tags.dedup();
155 + let vector: Vec<f64> =
156 + serde_json::from_str(&json).map_err(|e| format!("vector for {hash}: {e}"))?;
157 + // The index drops these too (`is_usable_vector`), so counting them as
158 + // testable would score the layer on samples it never sees.
159 + if vector.len() != NUM_FEATURES || !vector.iter().all(|x| x.is_finite()) {
160 + continue;
161 + }
162 + let truth = (tags.len() == 1).then(|| tags[0].clone());
163 + out.push(Row {
164 + hash,
165 + vector,
166 + tags,
167 + truth,
168 + });
169 + }
170 + Ok(out)
171 + }
172 +
173 + /// Assign each row a fold, stratified by class.
174 + ///
175 + /// Round-robin down each class's hash-sorted list rather than a shuffle: the
176 + /// content hash is already an arbitrary order with respect to the source pack a
177 + /// file came from, so this needs no RNG and two runs over the same corpus produce
178 + /// the same folds. Ambiguous rows get no fold; they train everywhere.
179 + fn assign_folds(rows: &[Row], folds: usize) -> Vec<Option<usize>> {
180 + let mut seen_per_class: HashMap<&str, usize> = HashMap::new();
181 + rows.iter()
182 + .map(|r| {
183 + let truth = r.truth.as_deref()?;
184 + let n = seen_per_class.entry(truth).or_default();
185 + let fold = *n % folds;
186 + *n += 1;
187 + Some(fold)
188 + })
189 + .collect()
190 + }
191 +
192 + /// Build an in-memory vault holding just the training rows, so the index under
193 + /// test is built by the same `build_index` the app calls.
194 + ///
195 + /// Every exemplar here is `local` (weight 1.0) where the shipped ones arrive from
196 + /// an imported layer at a lower weight. That does not change a score: the weight
197 + /// is a constant multiplier inside a sum that is then divided by its own total,
198 + /// so a uniform weight cancels. It would matter only against a mixed index of
199 + /// user labels plus the layer, which is a different measurement (how the layer
200 + /// behaves beside a user's own data) and not what a ship gate turns on.
201 + fn training_db(rows: &[&Row]) -> Result<Database, String> {
202 + let db = Database::open_in_memory().map_err(|e| format!("open_in_memory: {e}"))?;
203 + db.transaction(|_tx| {
204 + for row in rows {
205 + db.conn().execute(
206 + "INSERT INTO samples (hash, original_name, file_extension, file_size, import_date, last_modified)
207 + VALUES (?1, ?1, 'wav', 1, 0, 0)",
208 + [&row.hash],
209 + )?;
210 + let json = serde_json::to_string(&row.vector).unwrap_or_default();
211 + db.conn().execute(
212 + "INSERT INTO sample_features (hash, feat_version, vector, computed_at) VALUES (?1, ?2, ?3, 0)",
213 + rusqlite::params![&row.hash, FEATURE_VERSION, json],
214 + )?;
215 + }
216 + Ok(())
217 + })
218 + .map_err(|e| format!("seeding the fold index: {e}"))?;
219 + // Tags outside the transaction: `add_tag` is the public path and manages its
220 + // own writes, and this is an in-memory DB where the fsync cost it avoids does
221 + // not exist.
222 + for row in rows {
223 + for tag in &row.tags {
224 + audiofiles_core::tags::add_tag(&db, &row.hash, tag)
225 + .map_err(|e| format!("tagging the fold index: {e}"))?;
226 + }
227 + }
228 + Ok(db)
229 + }
230 +
231 + /// What one test sample produced.
232 + struct Prediction {
233 + truth: String,
234 + /// Highest-scoring tag, or `None` when the index returned nothing.
235 + top1: Option<String>,
236 + /// Score per class, for threshold sweeps.
237 + scores: BTreeMap<String, f64>,
238 + }
239 +
240 + pub(crate) fn run(corpus: &Path, vault: &Path, config: &AnalysisConfig, folds: usize) {
241 + println!("━━━ CLASSIFIER LAYER EVALUATION ━━━");
242 + println!();
243 + println!(" corpus {}", corpus.display());
244 + println!(" scratch {}", vault.display());
245 + println!(" features v{FEATURE_VERSION}");
246 + println!(" k {DEFAULT_K} (runtime default)");
247 + println!(" folds {folds}, stratified by class");
248 + println!();
249 + println!(" Gate, fixed before this run:");
250 + println!(
251 + " per-class precision at the auto threshold ({DEFAULT_AUTO_THRESHOLD:.2}) >= {:.0}%",
252 + GATE.auto_precision * 100.0
253 + );
254 + println!(
255 + " per-class recall at the auto threshold >= {:.0}%",
256 + GATE.auto_recall * 100.0
257 + );
258 + println!(
259 + " macro-averaged top-1 recall >= {:.0}%",
260 + GATE.top1_macro_recall * 100.0
261 + );
262 + println!(" no class the layer never predicts");
263 + println!();
264 +
265 + let built = match labelled::build_vault(corpus, vault, config) {
266 + Ok(v) => v,
267 + Err(e) => {
268 + eprintln!("corpus: {e}");
269 + std::process::exit(1);
270 + }
271 + };
272 +
273 + let rows = match load_rows(&built.db) {
274 + Ok(r) => r,
275 + Err(e) => {
276 + eprintln!("reading the vault back: {e}");
277 + std::process::exit(1);
278 + }
279 + };
280 + let ambiguous = rows.iter().filter(|r| r.truth.is_none()).count();
281 + let testable = rows.len() - ambiguous;
282 + if testable == 0 {
283 + eprintln!("no single-labelled samples to test");
284 + std::process::exit(1);
285 + }
286 + println!();
287 + println!(" {} scoreable row(s) in the vault", rows.len());
288 + if ambiguous > 0 {
289 + // Not a silent drop: the same file under two class folders is a corpus
290 + // problem, and the count is how anyone notices it grew.
291 + println!(
292 + " {ambiguous} carry more than one class tag (duplicate audio across folders);\n \
293 + they train in every fold and are never tested"
294 + );
295 + }
296 +
297 + let classes: BTreeSet<String> = rows.iter().filter_map(|r| r.truth.clone()).collect();
298 + let classes: Vec<String> = classes.into_iter().collect();
299 + let fold_of = assign_folds(&rows, folds);
300 +
301 + // Cross-validation
302 + let mut predictions: Vec<Prediction> = Vec::with_capacity(testable);
303 + for fold in 0..folds {
304 + let train: Vec<&Row> = rows
305 + .iter()
306 + .zip(&fold_of)
307 + .filter(|(_, f)| **f != Some(fold))
308 + .map(|(r, _)| r)
309 + .collect();
310 + let test: Vec<&Row> = rows
311 + .iter()
312 + .zip(&fold_of)
313 + .filter(|(_, f)| **f == Some(fold))
314 + .map(|(r, _)| r)
315 + .collect();
316 +
317 + let db = match training_db(&train) {
318 + Ok(db) => db,
319 + Err(e) => {
320 + eprintln!("fold {fold}: {e}");
321 + std::process::exit(1);
322 + }
323 + };
324 + let index = match exemplar::build_index(&db) {
325 + Ok(i) => i,
326 + Err(e) => {
327 + eprintln!("fold {fold}: build_index: {e}");
328 + std::process::exit(1);
329 + }
330 + };
331 + println!(
332 + " fold {fold}: {} exemplars, {} held out",
333 + index.len(),
334 + test.len()
335 + );
336 +
337 + for row in test {
338 + // No `exclude_hash`: the row is not in this index at all, which is
339 + // the property the fold split exists to give.
340 + let scored = index.score(&row.vector, DEFAULT_K, None);
341 + let top1 = scored.first().map(|s| s.tag.clone());
342 + let scores = scored.into_iter().map(|s| (s.tag, s.score)).collect();
343 + predictions.push(Prediction {
344 + truth: row.truth.clone().unwrap_or_default(),
345 + top1,
346 + scores,
347 + });
348 + }
349 + }
350 + println!();
351 +
352 + let mut report = Report::new("layer-eval");
353 + report.set("folds", folds);
354 + report.set("k", DEFAULT_K);
355 + report.set("feat_version", FEATURE_VERSION);
356 + report.set("exemplars_total", rows.len());
357 + report.set("tested", predictions.len());
358 + report.set("ambiguous_excluded", ambiguous);
359 +
360 + let top1 = print_confusion(&predictions, &classes, &mut report);
361 + let auto = print_threshold_table(
362 + &predictions,
363 + &classes,
364 + DEFAULT_AUTO_THRESHOLD,
365 + "auto-apply",
366 + "auto",
367 + &mut report,
368 + );
369 + let _review = print_threshold_table(
370 + &predictions,
371 + &classes,
372 + DEFAULT_REVIEW_THRESHOLD,
373 + "review",
374 + "review",
375 + &mut report,
376 + );
377 +
378 + print_verdict(&classes, &top1, &auto, &mut report);
379 + report.write();
380 + }
381 +
382 + /// Top-1 confusion matrix, and per-class top-1 recall. Returns the per-class counts.
383 + fn print_confusion(
384 + predictions: &[Prediction],
385 + classes: &[String],
386 + report: &mut Report,
387 + ) -> BTreeMap<String, Counts> {
388 + println!("━━━ TOP-1 CONFUSION ━━━");
389 + println!();
390 + println!(" Rows are the corpus label, columns the highest-scoring tag.");
391 + println!(" `(none)` is a sample the index scored nothing for at all.");
392 + println!();
393 +
394 + let width = classes
395 + .iter()
396 + .map(|c| label_for_tag(c).len().max(5))
397 + .collect::<Vec<_>>();
398 +
399 + print!(" {:<12}", "true \\ pred");
400 + for (c, w) in classes.iter().zip(&width) {
401 + print!(" {:>w$}", label_for_tag(c), w = w);
402 + }
403 + println!(" {:>6} {:>8}", "(none)", "recall");
404 + println!(
405 + " {}",
406 + "─".repeat(12 + width.iter().map(|w| w + 1).sum::<usize>() + 16)
407 + );
408 +
409 + let mut counts: BTreeMap<String, Counts> = BTreeMap::new();
410 + let mut never_predicted: Vec<&str> = Vec::new();
411 +
412 + for truth in classes {
413 + let mine: Vec<&Prediction> = predictions.iter().filter(|p| &p.truth == truth).collect();
414 + print!(" {:<12}", label_for_tag(truth));
415 + let mut correct = 0usize;
416 + for (pred, w) in classes.iter().zip(&width) {
417 + let n = mine
418 + .iter()
419 + .filter(|p| p.top1.as_deref() == Some(pred.as_str()))
420 + .count();
421 + if pred == truth {
422 + correct = n;
423 + }
424 + print!(" {n:>w$}");
425 + }
426 + let none = mine.iter().filter(|p| p.top1.is_none()).count();
427 + let recall = if mine.is_empty() {
428 + None
429 + } else {
430 + Some(correct as f64 / mine.len() as f64)
431 + };
432 + println!(" {:>6} {:>8}", none, pct(recall));
433 +
434 + // Precision needs the whole column, so it is counted here rather than
435 + // inside the row loop.
436 + let predicted_as = predictions
437 + .iter()
438 + .filter(|p| p.top1.as_deref() == Some(truth.as_str()))
439 + .count();
440 + if predicted_as == 0 {
441 + never_predicted.push(label_for_tag(truth));
442 + }
443 + counts.insert(
444 + truth.clone(),
445 + Counts {
446 + tp: correct,
447 + fp: predicted_as - correct,
448 + fn_: mine.len() - correct,
449 + },
450 + );
451 + }
452 + println!();
453 +
454 + let macro_recall = macro_average(classes, &counts, Counts::recall);
455 + let micro = counts.values().map(|c| c.tp).sum::<usize>() as f64 / predictions.len() as f64;
456 + println!(" macro-averaged recall {}", pct(macro_recall));
457 + println!(" overall top-1 accuracy {}", pct(Some(micro)));
458 + println!();
459 + println!(" The macro figure is the one that matters: it weights the 48-file");
460 + println!(" clap class the same as the 246-file tom class, so a layer that");
461 + println!(" gets the big classes right and ignores a small one cannot hide.");
462 + if never_predicted.is_empty() {
463 + println!(" Every class is predicted at least once.");
464 + } else {
465 + println!();
466 + println!(
467 + " NEVER PREDICTED: {}. This class is unreachable, not merely weak.",
468 + never_predicted.join(", ")
469 + );
470 + }
471 + println!();
472 +
473 + if let Some(m) = macro_recall {
474 + report.set("top1_macro_recall", (m * 10000.0).round() / 10000.0);
475 + }
476 + report.set("top1_accuracy", (micro * 10000.0).round() / 10000.0);
477 + report.set("never_predicted", never_predicted.len());
478 + for (tag, c) in &counts {
479 + let label = label_for_tag(tag);
480 + if let Some(r) = c.recall() {
481 + report.set(
482 + &format!("top1_{label}_recall"),
483 + (r * 10000.0).round() / 10000.0,
484 + );
485 + }
486 + if let Some(p) = c.precision() {
487 + report.set(
488 + &format!("top1_{label}_precision"),
489 + (p * 10000.0).round() / 10000.0,
490 + );
491 + }
492 + }
493 + counts
494 + }
495 +
496 + /// Per-class precision/recall/F1 at one score threshold, scored multi-label:
497 + /// every tag at or above the threshold is a prediction, so a sample can be right
498 + /// about one class and wrong about another in the same breath. That is how the
499 + /// layer behaves at runtime, and a top-1 view would not show a layer that applies
500 + /// three tags to everything.
Lines truncated