max / audiofiles

Retire the ingest baseline: the code under it moved 1a0edd0 changed the import write path, so the two 2026-07-29 files measure a per-file directory fsync that no longer exists and an 8 KiB streaming buffer that is now 256 KiB. Both changes make import faster, which is the dangerous direction: a later run beats 168.4 files/s and reads as a win when it has not been compared against anything. Same trap as the corpus one the README already carries, from the other side. There the file set moved under a fixed measurement; here the measured code moved under a fixed file set. Says so, and says there is no ingest baseline until someone takes one.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: Max Johnson <me@maxj.phd> · 2026-08-16 17:24 UTC

Signed with PGP, not checked

Commit: 55bc1abbf57f63da50081e3df37b5b2380affa6c

Parent: 1a0edd0

1 file changed, +12 insertions, -1 deletion

M benchmarks/README.md +12 -1

			@@ -71,7 +71,7 @@
71	71		\| file \| what \|
72	72		\|---\|---\|
73	73		\| `ingest-2026-07-29-785fbeb.json` \| Pre-fanout control: flat blob store root. \|
74		-	\| `ingest-2026-07-29-1258f9e.json` \| The baseline. Hash-prefix sharded store. \|
	74	+	\| `ingest-2026-07-29-1258f9e.json` \| Was the baseline. Now a pre-`1a0edd0` control. \|
75	75
76	76		Taken back to back, machine and drive idle, identical corpus and batch size, a
77	77		fresh vault each. They come out 168.3 against 168.4 files/s and 106.5 MB/s each,
			@@ -79,3 +79,14 @@
79	79		expected result rather than a disappointing one: at 2,241 blobs a flat directory
80	80		was never the problem. The layout change is for the 289k-file case, where the flat
81	81		root lost about 90% of its throughput.
	82	+
	83	+	**Neither file is a comparison target any more, and there is no ingest baseline
	84	+	until someone takes one.** `1a0edd0` changed the import write path underneath
	85	+	them: the per-file directory fsync became one per shard directory at the end of
	86	+	the run, and the streaming hash/copy buffers went from 8 KiB to 256 KiB. Both
	87	+	make import faster, so a run that beats 168.4 files/s says nothing about the code
	88	+	it is testing until it has a post-`1a0edd0` baseline to sit against. Same trap as
	89	+	the corpus one above, from the other direction: there, the file set moved under a
	90	+	fixed measurement; here, the measured code moved under a fixed file set. Take the
	91	+	new baseline the way the rules above say, machine and drive idle, and this section
	92	+	can be rewritten around it.