# Benchmark baselines

Saved `AF_BENCH_JSON` output from `audiofiles-bench`, kept so a later run has
something to be compared against. Small, self-describing files: each records the
metrics plus the drive, filesystem, USB link speed and RAM it was taken on.

## Naming

```
<mode>-<YYYY-MM-DD>-<short-sha>.json
```

`mode` is the bench mode (`ingest`, `analysis`, `accuracy`, `layout`). The sha is
the commit the binary was built from. Both matter: a filename alone should say what
was measured and which code produced it.

`layout` is the odd one out: it is a checker that happens to time a sweep, not a
measurement with a baseline. It fabricates its own flat vaults, needs no corpus,
and exits non-zero when a scenario fails, so it is worth running before touching
anything under `store/`. Its throughput number moves with the drive rather than
with the code, so it is not a figure to defend a change with.

## Producing one

Set `AF_BENCH_JSON` and copy the result in. The corpus lives outside the repo
(gitignored, per-machine), so paths differ per box.

```
AF_BENCH_CORPUS=/media/max/T9/af-corpus/samples \
AF_BENCH_VAULT=/media/max/T9/af-bench-fresh \
AF_BENCH_BATCH=400 AF_BENCH_ANALYZE=600 \
AF_BENCH_JSON=/tmp/run.json \
  cargo run --release -p audiofiles-bench -- ingest
```

Diffing two runs:

```
diff <(jq -S . benchmarks/ingest-2026-07-29-1258f9e.json) <(jq -S . /tmp/run.json)
```

## What makes a run comparable

Only against another run on the same machine, the same drive, and the **same
corpus contents**. All three are part of the measurement, and only the first two
are recorded in the file.

The corpus one has already bitten. The 2026-07-29 baselines here read 2,241 files
where the earlier figure in wiki `af-benchmarks` read 1,761, because the four
120-file format arms added for the per-format decode section landed inside
`samples/` and the ingest walk picks them up. Aggregate throughput came out 168
files/s against the older 204, which looks like a 17% regression and is not one:
it is a different set of files, weighted differently. An A/B of the same corpus
across the two commits showed the change cost nothing (see below).

Rules that follow from that:

- **Take a baseline with the machine and drive idle.** Import is I/O bound. A run
  taken while a dataset was downloading to the same drive read 40.7 files/s
  against 204 for the same corpus, with no code change.
- **Use a fresh vault path.** Pointing at an existing scratch vault measures
  dedup, not import, because the store finds the blobs already there.
- **Do not compare across corpora.** If the corpus changed, re-measure both arms
  rather than reasoning about the delta.
- **A/B across commits by rebuilding, not by trusting an old file.** Checking out
  the older commit and running it back to back on the same corpus is a few
  minutes and removes every confound at once.

## Current files

| file | what |
|---|---|
| `ingest-2026-07-29-785fbeb.json` | Pre-fanout control: flat blob store root. |
| `ingest-2026-07-29-1258f9e.json` | Was the baseline. Now a pre-`1a0edd0` control. |

Taken back to back, machine and drive idle, identical corpus and batch size, a
fresh vault each. They come out 168.3 against 168.4 files/s and 106.5 MB/s each,
so sharding the blob directory costs nothing measurable at this size. That is the
expected result rather than a disappointing one: at 2,241 blobs a flat directory
was never the problem. The layout change is for the 289k-file case, where the flat
root lost about 90% of its throughput.

**Neither file is a comparison target any more, and there is no ingest baseline
until someone takes one.** `1a0edd0` changed the import write path underneath
them: the per-file directory fsync became one per shard directory at the end of
the run, and the streaming hash/copy buffers went from 8 KiB to 256 KiB. Both
make import faster, so a run that beats 168.4 files/s says nothing about the code
it is testing until it has a post-`1a0edd0` baseline to sit against. Same trap as
the corpus one above, from the other direction: there, the file set moved under a
fixed measurement; here, the measured code moved under a fixed file set. Take the
new baseline the way the rules above say, machine and drive idle, and this section
can be rewritten around it.