max / audiofiles

Say plainly that the bundled layer is a cold-start feature layer-stability measures what the layer adds on top of a user's own labels, and the answer decays fast: on the samples filenames do not answer it is worth +7.2 points at 18 user labels, +2.9 at 88, and +1.4 at 352, where it changes one answer in 69. It answers a library it has never seen at 98.6%, which is the thing worth shipping. Nothing here changes the ship decision or the weight. Numbers are re-measured at k=5, not the k=15 the finding was filed against. At 15 the layer was inert on that population from 88 labels on; at 5 it is small but not zero, so the copy says fading rather than finished. Also drops two claims that stopped being true: official.rs said no harness measured family separation, and ml_classifier.md said no metric existed for whether the layer answers the same way twice. layer-eval and layer-stability are both built and both have run.

Author: Max Johnson <me@maxj.phd> · 2026-08-07 19:34 UTC

Signed with PGP, not checked

Commit: 4cb2262d5acca57380b534ebd042836f21cbea0a

Parent: c71c124

5 files changed, +76 insertions, -11 deletions

M docs/ml_classifier.md +34 -1

			@@ -186,13 +186,46 @@
186	186		was written before the first run. At family resolution both measured families clear
187	187		it (below). Accuracy is not the property that decides this, though: the layer's
188	188		model is the user's own library, so the question is whether it gives the same
189		-	answer as that library fills, and no metric for that exists yet.
	189	+	answer as that library fills. `layer-stability` measures that half, and at the
	190	+	weight the app actually builds the answer holds still (1.1% worst consecutive flip
	191	+	rate). It is measured over two families, so read it as a floor.
190	192		3. The layer would inherit the global 0.85 regardless. `include_policy` is `false`
191	193		in `afcl_gen`, so the calibrated per-class thresholds are not exported and no
192	194		measured number survives the trip into a shipped artifact.
193	195
194	196		All three are tracked in the audiofiles GoingsOn project.
195	197
	198	+	### What the layer is worth, and when
	199	+
	200	+	It is a cold-start feature. That is a real thing to be, and it is narrower than
	201	+	"a classifier ships with the app" sounds.
	202	+
	203	+	`layer-stability` compares the index the app builds (the user's own labels at 1.0
	204	+	beside the bundled layer at 0.5) against the same user labels with no layer at all,
	205	+	on a held-out probe. Measured 2026-08-07 at `DEFAULT_K` = 5, 1049 drum one-shots,
	206	+	178 probe samples of which 69 carry a filename no starter rule fires on. That 69 is
	207	+	the population the layer exists to serve, since the rest are already answered for
	208	+	free:
	209	+
	210	+	\| user labels \| accuracy with the layer \| without it \| delta \|
	211	+	\|---\|---\|---\|---\|
	212	+	\| 0 \| 98.6% \| nothing to answer with \| cold start \|
	213	+	\| 18 \| 98.6% \| 91.3% \| +7.2 \|
	214	+	\| 88 \| 100% \| 97.1% \| +2.9 \|
	215	+	\| 352 \| 100% \| 98.6% \| +1.4 \|
	216	+
	217	+	The layer answers a library it has never seen at 98.6%, and a user who has tagged a
	218	+	few dozen samples is getting most of what it has to give. By a few hundred labels
	219	+	their own tags carry the library and the layer is worth about a point: at 352 labels
	220	+	it changes one answer in 69.
	221	+
	222	+	Two things not to oversell from this. The cold-start window overlaps something the
	223	+	app already does without any classifier: browse axes give a new library structure
	224	+	from zero labels and zero exemplars (`af-browse-axes`). And the measurement is two
	225	+	families on a drums-only corpus. More classes plausibly widen the window, since the
	226	+	layer's advantage over a thin user library should grow with the number of things to
	227	+	be wrong about, so this table is a result at a resolution, not a permanent property.
	228	+
196	229		### Retracted: runs 1 and 2 answer the wrong question
197	230
198	231		The two tables below grade seven specific drum instruments. Both halves of that

M crates/audiofiles-browser/src/ui/classifier.rs +18

			@@ -1087,6 +1087,24 @@
1087	1087		.color(theme::content_muted()),
1088	1088		);
1089	1089		});
	1090	+	// The bundled layer is a head start, not an ongoing second
	1091	+	// opinion, and the measurement says so plainly: on the samples
	1092	+	// filenames do not answer it is worth +7.2 points of accuracy at
	1093	+	// 18 of the user's own labels and +1.4 at 352. Saying that here
	1094	+	// is cheaper than letting someone conclude it later and wonder
	1095	+	// what broke. Only the bundled layer gets the line; a layer
	1096	+	// someone imported is theirs to judge.
	1097	+	if layer.source.as_deref() == Some("bundled") {
	1098	+	ui.label(
	1099	+	egui::RichText::new(
	1100	+	"A starting point for a library with no tags yet. It counts for \
	1101	+	less as you tag your own, and by a few hundred tags yours carry \
	1102	+	the library.",
	1103	+	)
	1104	+	.small()
	1105	+	.color(theme::content_muted()),
	1106	+	);
	1107	+	}
1090	1108		// Row B: weight + remove (with inline two-step confirm, destructive, no undo).
1091	1109		ui.horizontal(\|ui\| {
1092	1110		ui.label(

M crates/audiofiles-browser/src/ui/review_library.rs +7

			@@ -5,6 +5,13 @@
5	5		//! answers are seen and decided on; nothing it proposes reaches a library until
6	6		//! someone here says so.
7	7		//!
	8	+	//! This screen matters most on a library with few tags, because that is when the
	9	+	//! bundled layer has something to add: measured at +7.2 points of accuracy at 18
	10	+	//! of the user's own labels and +1.4 at 352, on the samples filenames do not
	11	+	//! answer (`docs/ml_classifier.md`, "What the layer is worth, and when"). It does
	12	+	//! not stop being useful, since the user's own labels feed the same k-NN and the
	13	+	//! queue keeps working off their tagging, but the bundled layer's share of it fades.
	14	+	//!
8	15		//! # Grouped by tag, and that is the whole design
9	16		//!
10	17		//! "Are these 340 all kicks?" is a question one scan of a list answers. The same

M crates/audiofiles-core/src/analysis/exemplar.rs +3 -3

			@@ -20,9 +20,9 @@
20	20
21	21		/// Default neighbor count.
22	22		///
23		-	/// 5, not the 15 shipped until 2026-08-07. Selected under an outer fold — each fold
24		-	/// picks k by inner CV over its own training corpus, then scores the fold it never
25		-	/// saw — and all five folds chose 5. Worth 2.6 points of macro top-1 at family
	23	+	/// 5, not the 15 shipped until 2026-08-07. Selected under an outer fold, where each
	24	+	/// fold picks k by inner CV over its own training corpus and then scores the fold it
	25	+	/// never saw; all five folds chose 5. Worth 2.6 points of macro top-1 at family
26	26		/// resolution, and the queue-stability measurement that decides the ship question
27	27		/// (`layer-stability`, deployment weight) is unchanged at 1.1%. The measurements it
28	28		/// does worsen (index composition, index growth, accepted-tag feedback) were over

M crates/audiofiles-core/src/analysis/official.rs +14 -7

			@@ -3,13 +3,13 @@
3	3		//! Behind the `bundled-layer` feature, which is off. Nothing here is compiled
4	4		//! into a default build. The layer as generated covers drums and nothing else
5	5		//! (clap, cymbal, hihat, kick, snare, tom, percussion), so it would answer for a
6		-	//! bass or a vocal with the nearest drum it knows, and there is as yet no harness
7		-	//! measuring whether the 35-d feature vector separates instrument families at
8		-	//! all. A default classifier that is confidently wrong on a user's library is
9		-	//! worse than no default: it puts wrong tags on their samples with the app's
10		-	//! authority behind it. The machinery below is complete and tested; what it
11		-	//! waits on is a broader corpus and a measurement that justifies shipping.
12		-	//! Tracked in the audiofiles GoingsOn project.
	6	+	//! bass or a vocal with the nearest drum it knows. A default classifier that is
	7	+	//! confidently wrong on a user's library is worse than no default: it puts wrong
	8	+	//! tags on their samples with the app's authority behind it. The machinery below
	9	+	//! is complete and tested, and both halves of the measurement now exist
	10	+	//! (`layer-eval` for whether the answer is right, `layer-stability` for whether
	11	+	//! it is the same answer next month); what it waits on is a corpus that reaches
	12	+	//! past drums. Tracked in the audiofiles GoingsOn project.
13	13		//!
14	14		//! A fresh vault has no labelled exemplars, so k-NN has nothing to answer with
15	15		//! until the user has tagged enough of their own library to seed it. The bundled
			@@ -17,6 +17,13 @@
17	17		//! labelled corpus, carried as feature vectors and tags with no audio and no
18	18		//! hashes, which is the whole reason a `.afcl` can ship at all.
19	19		//!
	20	+	//! Day one is close to the whole of it, and the copy elsewhere should say so.
	21	+	//! Measured 2026-08-07: on the samples no filename rule answers, the layer is
	22	+	//! worth +7.2 points of accuracy at 18 user labels, +2.9 at 88, and +1.4 at 352,
	23	+	//! where it changes one answer in 69. It answers a library it has never seen at
	24	+	//! 98.6%. Full table in `docs/ml_classifier.md`, "What the layer is worth, and
	25	+	//! when"; two families only, so more classes could widen the window.
	26	+	//!
20	27		//! Generated by `cargo run --release -p audiofiles-bench -- afcl` and checked in
21	28		//! at `assets/official.afcl` beside this crate. It is a build artifact, not a
22	29		//! fixture: regenerate it deliberately.