Skip to main content

max / audiofiles

Select k under an outer fold, so the sweep stops grading its own choice The k sweep calibrated thresholds out of fold and then picked k by reading every row of the result, which is choosing a hyperparameter on the test set. The caveat has been printed under the table since run 2 and flagged unfixed since; this fixes it the same way the thresholds were fixed. Each outer fold selects k by inner CV over its own training corpus, then scores the fold it never saw. Indexes are rebuilt rather than recycled from the sweep: a prediction there came from an index containing the outer test fold, so reusing them would leak exactly what this exists to stop. Result at family resolution: all five outer folds select k = 5, and pooled outer macro top-1 recall is 98.1% against the sweep's own 98.1% at k = 5. The two agreeing is the finding -- there was no selection bias to remove, and k = 5 beats the shipped k = 15 by 2.6 points on a sweep that is monotone in k. Changing DEFAULT_K is not done here; it has a second consumer.
Author: Max Johnson <me@maxj.phd> · 2026-08-07 17:46 UTC
Signed with PGP, not checked
Commit: 1114ce7afa2d6f8298a309bd383f75ff9153e0fe
Parent: 163504f
2 files changed, +276 insertions, -10 deletions
M Cargo.lock +4 -4
@@ -7297,10 +7297,6 @@
7297 7297 "winnow 1.0.4",
7298 7298 ]
7299 7299
7300 - [[patch.unused]]
7301 - name = "docengine"
7302 - version = "0.4.0"
7303 -
7304 7300 [[patch.unused]]
7305 7301 name = "kberg"
7306 7302 version = "0.1.0"
@@ -7308,3 +7304,7 @@
7308 7304 [[patch.unused]]
7309 7305 name = "painhours"
7310 7306 version = "0.1.0"
7307 +
7308 + [[patch.unused]]
7309 + name = "docengine"
7310 + version = "0.4.0"
M crates/audiofiles-bench/src/layer_eval.rs +272 -6
@@ -334,6 +334,7 @@
334 334 let calibrated = print_calibration(default_k, &classes, folds, space, &mut report);
335 335 if k_sweep.len() > 1 {
336 336 print_k_sweep(&by_k, &classes, folds, &mut report);
337 + print_nested_k(&rows, &fold_of, &classes, k_sweep, folds, &mut report);
337 338 }
338 339 print_verdict(&classes, &top1, &calibrated, space, &mut report);
339 340 report.write();
@@ -824,15 +825,280 @@
824 825 println!(" the precision bar. Those are the classes a shipped layer cannot apply");
825 826 println!(" at all, whatever the global default is set to.");
826 827 println!();
827 - println!(" CAVEAT, and it is the same one the thresholds were fixed for: this");
828 - println!(" table selects k on the data it reports. Thresholds are calibrated out");
829 - println!(" of fold, k is not, so reading the best row here and shipping that k");
830 - println!(" would be choosing a hyperparameter on the test set. Treat it as");
831 - println!(" evidence about the mechanism, and confirm a chosen k on a corpus this");
832 - println!(" has not seen, or under an outer fold that holds k out too.");
828 + println!(" CAVEAT: this table selects k on the data it reports. Thresholds are");
829 + println!(" calibrated out of fold, k is not, so reading the best row here and");
830 + println!(" shipping that k would be choosing a hyperparameter on the test set.");
831 + println!(" It is evidence about the mechanism. The outer fold below is the");
832 + println!(" number to quote instead.");
833 833 println!();
834 834 }
835 835
836 + /// Select `k` under an outer fold, so the selection is never scored on the data
837 + /// it was made from.
838 + ///
839 + /// The sweep above is the standard trap: it fits thresholds out of fold and then
840 + /// picks `k` by reading every row of the result. That is choosing a
841 + /// hyperparameter on the test set, and the honest correction is the same shape as
842 + /// the one that fixed the thresholds — hold the selection out too.
843 + ///
844 + /// Procedure. For each outer fold: take the other folds as a training corpus,
845 + /// split THOSE by an inner fold, select the `k` with the best inner macro
846 + /// calibrated recall, then build one index over the whole training corpus and
847 + /// score the outer fold at the selected `k`. Nothing about the outer fold is
848 + /// visible to the selection, so the pooled result is what "select k this way"
849 + /// generalises to.
850 + ///
851 + /// Indexes are rebuilt rather than reused from `by_k`: a prediction there was
852 + /// made by an index containing the outer test fold, so recycling them would leak
853 + /// exactly what this exists to stop. That costs an extra `folds * folds` index
854 + /// builds and no extra analysis, which is the cheap half.
855 + ///
856 + /// What to read: if the outer number matches the best row of the sweep, the
857 + /// sweep was not being flattered by its own selection and the mechanism finding
858 + /// stands. If it comes in below, the difference is the selection bias, and the
859 + /// outer number is the one that would survive contact with a user's library.
860 + fn print_nested_k(
861 + rows: &[Row],
862 + outer_of: &[Option<usize>],
863 + classes: &[String],
864 + k_sweep: &[usize],
865 + folds: usize,
866 + report: &mut Report,
867 + ) {
868 + println!("━━━ k UNDER AN OUTER FOLD ━━━");
869 + println!();
870 + println!(" k selected inside each outer fold's training corpus, then scored on the");
871 + println!(" outer fold it never saw. This is the k-sweep number with the selection");
872 + println!(" bias removed.");
873 + println!();
874 + println!(
875 + " {:>7} {:>10} {:>10} {:>16} {:>14}",
876 + "outer", "train", "k chosen", "inner recall", "outer recall"
877 + );
878 + println!(" {}", "─".repeat(64));
879 +
880 + let mut pooled: Vec<Prediction> = Vec::new();
881 + let mut chosen: Vec<usize> = Vec::new();
882 +
883 + for outer in 0..folds {
884 + let train: Vec<&Row> = rows
885 + .iter()
886 + .zip(outer_of)
887 + .filter(|(_, f)| **f != Some(outer))
888 + .map(|(r, _)| r)
889 + .collect();
890 + let test: Vec<&Row> = rows
891 + .iter()
892 + .zip(outer_of)
893 + .filter(|(_, f)| **f == Some(outer))
894 + .map(|(r, _)| r)
895 + .collect();
896 + if test.is_empty() {
897 + continue;
898 + }
899 +
900 + // Inner CV over the training corpus only.
901 + let train_rows: Vec<Row> = train.iter().map(|r| clone_row(r)).collect();
902 + let inner_of = rows::assign_folds(&train_rows, folds);
903 + let mut inner: BTreeMap<usize, Vec<Prediction>> =
904 + k_sweep.iter().map(|k| (*k, Vec::new())).collect();
905 + for i in 0..folds {
906 + let itrain: Vec<&Row> = train_rows
907 + .iter()
908 + .zip(&inner_of)
909 + .filter(|(_, f)| **f != Some(i))
910 + .map(|(r, _)| r)
911 + .collect();
912 + let itest: Vec<&Row> = train_rows
913 + .iter()
914 + .zip(&inner_of)
915 + .filter(|(_, f)| **f == Some(i))
916 + .map(|(r, _)| r)
917 + .collect();
918 + let Ok(db) = rows::local_db(&itrain) else {
919 + continue;
920 + };
921 + let Ok(index) = exemplar::build_index(&db) else {
922 + continue;
923 + };
924 + for row in itest {
925 + for &k in k_sweep {
926 + push_prediction(inner.entry(k).or_default(), &index, row, k, i);
927 + }
928 + }
929 + }
930 +
931 + // Selection criterion: macro calibrated recall, the same quantity the
932 + // sweep table ranks on, so the two are comparable.
933 + let score_of = |preds: &Vec<Prediction>| {
934 + let mut calibrated: BTreeMap<String, Counts> = BTreeMap::new();
935 + for class in classes {
936 + let points = class_points(preds, class);
937 + let (oof, _) = calibration::out_of_fold(
938 + &points,
939 + folds,
940 + GATE.target_precision,
941 + GATE.min_support,
942 + );
943 + calibrated.insert(class.clone(), oof);
944 + }
945 + macro_average(classes, &calibrated, Counts::recall).unwrap_or(0.0)
946 + };
947 + let Some((best_k, inner_recall)) = k_sweep
948 + .iter()
949 + .map(|k| (*k, score_of(&inner[k])))
950 + // total_cmp then the smaller k, so a tie ships the tighter
951 + // neighbourhood rather than whichever the map iterated first.
952 + .max_by(|a, b| a.1.total_cmp(&b.1).then(b.0.cmp(&a.0)))
953 + else {
954 + continue;
955 + };
956 + chosen.push(best_k);
957 +
958 + // Score the outer fold at the selected k, from an index over the whole
959 + // training corpus.
960 + let Ok(db) = rows::local_db(&train) else {
961 + continue;
962 + };
963 + let Ok(index) = exemplar::build_index(&db) else {
964 + continue;
965 + };
966 + let mut outer_preds: Vec<Prediction> = Vec::new();
967 + for row in &test {
968 + push_prediction(&mut outer_preds, &index, row, best_k, outer);
969 + }
970 + let outer_recall = {
971 + let mut top1: BTreeMap<String, Counts> = BTreeMap::new();
972 + for class in classes {
973 + let mine = outer_preds.iter().filter(|p| &p.truth == class).count();
974 + let correct = outer_preds
975 + .iter()
976 + .filter(|p| &p.truth == class && p.top1.as_deref() == Some(class.as_str()))
977 + .count();
978 + let predicted_as = outer_preds
979 + .iter()
980 + .filter(|p| p.top1.as_deref() == Some(class.as_str()))
981 + .count();
982 + top1.insert(
983 + class.clone(),
984 + Counts {
985 + tp: correct,
986 + fp: predicted_as - correct,
987 + fn_: mine - correct,
988 + },
989 + );
990 + }
991 + macro_average(classes, &top1, Counts::recall)
992 + };
993 +
994 + println!(
995 + " {:>7} {:>10} {:>10} {:>16} {:>14}",
996 + outer,
997 + train.len(),
998 + best_k,
999 + pct(Some(inner_recall)),
1000 + pct(outer_recall),
1001 + );
1002 + pooled.extend(outer_preds);
1003 + }
1004 + println!();
1005 +
1006 + if chosen.is_empty() {
1007 + println!(" no outer fold completed");
1008 + println!();
1009 + return;
1010 + }
1011 +
1012 + let mut top1: BTreeMap<String, Counts> = BTreeMap::new();
1013 + for class in classes {
1014 + let mine = pooled.iter().filter(|p| &p.truth == class).count();
1015 + let correct = pooled
1016 + .iter()
1017 + .filter(|p| &p.truth == class && p.top1.as_deref() == Some(class.as_str()))
1018 + .count();
1019 + let predicted_as = pooled
1020 + .iter()
1021 + .filter(|p| p.top1.as_deref() == Some(class.as_str()))
1022 + .count();
1023 + top1.insert(
1024 + class.clone(),
1025 + Counts {
1026 + tp: correct,
1027 + fp: predicted_as - correct,
1028 + fn_: mine - correct,
1029 + },
1030 + );
1031 + }
1032 + let pooled_recall = macro_average(classes, &top1, Counts::recall);
1033 + let agreed: BTreeSet<usize> = chosen.iter().copied().collect();
1034 +
1035 + println!(
1036 + " Pooled outer macro top-1 recall {} over {} predictions.",
1037 + pct(pooled_recall),
1038 + pooled.len()
1039 + );
1040 + if agreed.len() == 1 {
1041 + let k = *agreed.iter().next().unwrap_or(&DEFAULT_K);
1042 + println!(" Every outer fold selected k = {k}. A selection that does not move with");
1043 + println!(" the training data is one the corpus supports, not one it happened onto.");
1044 + if k != DEFAULT_K {
1045 + println!(
1046 + " It is NOT the shipped k ({DEFAULT_K}). That is a real finding, not a rounding:"
1047 + );
1048 + println!(" the runtime constant predates every measurement of this layer.");
1049 + }
1050 + report.set("nested_k_selected", k);
1051 + } else {
1052 + let spread: Vec<String> = agreed.iter().map(usize::to_string).collect();
1053 + println!(" Outer folds disagreed on k: {}.", spread.join(", "));
1054 + println!(" A selection that moves with the training data is not a property of the");
1055 + println!(" corpus, and shipping any single one of these is a coin toss dressed as a");
1056 + println!(" measurement. Read the sweep as a mechanism finding and leave k alone.");
1057 + report.set("nested_k_disagreed", agreed.len());
1058 + }
1059 + if let Some(r) = pooled_recall {
1060 + report.set("nested_k_outer_macro_recall", round4(r));
1061 + }
1062 + println!();
1063 + }
1064 +
1065 + /// Score one row against one index and push the prediction.
1066 + fn push_prediction(
1067 + into: &mut Vec<Prediction>,
1068 + index: &exemplar::ExemplarIndex,
1069 + row: &Row,
1070 + k: usize,
1071 + fold: usize,
1072 + ) {
1073 + // No `exclude_hash`: the row is not in this index at all, which is the
1074 + // property the fold split exists to give.
1075 + let scored = index.score(&row.vector, k, None);
1076 + into.push(Prediction {
1077 + truth: row.truth.clone().unwrap_or_default(),
1078 + top1: scored.first().map(|s| s.tag.clone()),
1079 + scores: scored.into_iter().map(|s| (s.tag, s.score)).collect(),
1080 + fold,
1081 + origin: row.origin.clone(),
1082 + });
1083 + }
1084 +
1085 + /// A `Row` copy, so the inner split can own its training corpus.
1086 + ///
1087 + /// `assign_folds` takes `&[Row]` rather than `&[&Row]`, and the inner fold is
1088 + /// assigned over a subset that only exists as borrows. Copying ~800 short vectors
1089 + /// once per outer fold is cheaper than threading a second lifetime through the
1090 + /// split.
1091 + fn clone_row(r: &Row) -> Row {
1092 + Row {
1093 + hash: r.hash.clone(),
1094 + vector: r.vector.clone(),
1095 + tags: r.tags.clone(),
1096 + truth: r.truth.clone(),
1097 + origin: r.origin.clone(),
1098 + name: r.name.clone(),
1099 + }
1100 + }
1101 +
836 1102 fn macro_average(
837 1103 classes: &[String],
838 1104 counts: &BTreeMap<String, Counts>,