|
1 |
+ |
# Benchmark baselines
|
|
2 |
+ |
|
|
3 |
+ |
Saved `AF_BENCH_JSON` output from `audiofiles-bench`, kept so a later run has
|
|
4 |
+ |
something to be compared against. Small, self-describing files: each records the
|
|
5 |
+ |
metrics plus the drive, filesystem, USB link speed and RAM it was taken on.
|
|
6 |
+ |
|
|
7 |
+ |
## Naming
|
|
8 |
+ |
|
|
9 |
+ |
```
|
|
10 |
+ |
<mode>-<YYYY-MM-DD>-<short-sha>.json
|
|
11 |
+ |
```
|
|
12 |
+ |
|
|
13 |
+ |
`mode` is the bench mode (`ingest`, `analysis`, `accuracy`). The sha is the commit
|
|
14 |
+ |
the binary was built from. Both matter: a filename alone should say what was
|
|
15 |
+ |
measured and which code produced it.
|
|
16 |
+ |
|
|
17 |
+ |
## Producing one
|
|
18 |
+ |
|
|
19 |
+ |
Set `AF_BENCH_JSON` and copy the result in. The corpus lives outside the repo
|
|
20 |
+ |
(gitignored, per-machine), so paths differ per box.
|
|
21 |
+ |
|
|
22 |
+ |
```
|
|
23 |
+ |
AF_BENCH_CORPUS=/media/max/T9/af-corpus/samples \
|
|
24 |
+ |
AF_BENCH_VAULT=/media/max/T9/af-bench-fresh \
|
|
25 |
+ |
AF_BENCH_BATCH=400 AF_BENCH_ANALYZE=600 \
|
|
26 |
+ |
AF_BENCH_JSON=/tmp/run.json \
|
|
27 |
+ |
cargo run --release -p audiofiles-bench -- ingest
|
|
28 |
+ |
```
|
|
29 |
+ |
|
|
30 |
+ |
Diffing two runs:
|
|
31 |
+ |
|
|
32 |
+ |
```
|
|
33 |
+ |
diff <(jq -S . benchmarks/ingest-2026-07-29-1258f9e.json) <(jq -S . /tmp/run.json)
|
|
34 |
+ |
```
|
|
35 |
+ |
|
|
36 |
+ |
## What makes a run comparable
|
|
37 |
+ |
|
|
38 |
+ |
Only against another run on the same machine, the same drive, and the **same
|
|
39 |
+ |
corpus contents**. All three are part of the measurement, and only the first two
|
|
40 |
+ |
are recorded in the file.
|
|
41 |
+ |
|
|
42 |
+ |
The corpus one has already bitten. The 2026-07-29 baselines here read 2,241 files
|
|
43 |
+ |
where the earlier figure in wiki `af-benchmarks` read 1,761, because the four
|
|
44 |
+ |
120-file format arms added for the per-format decode section landed inside
|
|
45 |
+ |
`samples/` and the ingest walk picks them up. Aggregate throughput came out 168
|
|
46 |
+ |
files/s against the older 204, which looks like a 17% regression and is not one:
|
|
47 |
+ |
it is a different set of files, weighted differently. An A/B of the same corpus
|
|
48 |
+ |
across the two commits showed the change cost nothing (see below).
|
|
49 |
+ |
|
|
50 |
+ |
Rules that follow from that:
|
|
51 |
+ |
|
|
52 |
+ |
- **Take a baseline with the machine and drive idle.** Import is I/O bound. A run
|
|
53 |
+ |
taken while a dataset was downloading to the same drive read 40.7 files/s
|
|
54 |
+ |
against 204 for the same corpus, with no code change.
|
|
55 |
+ |
- **Use a fresh vault path.** Pointing at an existing scratch vault measures
|
|
56 |
+ |
dedup, not import, because the store finds the blobs already there.
|
|
57 |
+ |
- **Do not compare across corpora.** If the corpus changed, re-measure both arms
|
|
58 |
+ |
rather than reasoning about the delta.
|
|
59 |
+ |
- **A/B across commits by rebuilding, not by trusting an old file.** Checking out
|
|
60 |
+ |
the older commit and running it back to back on the same corpus is a few
|
|
61 |
+ |
minutes and removes every confound at once.
|
|
62 |
+ |
|
|
63 |
+ |
## Current files
|
|
64 |
+ |
|
|
65 |
+ |
| file | what |
|
|
66 |
+ |
|---|---|
|
|
67 |
+ |
| `ingest-2026-07-29-785fbeb.json` | Pre-fanout control: flat blob store root. |
|
|
68 |
+ |
| `ingest-2026-07-29-1258f9e.json` | The baseline. Hash-prefix sharded store. |
|
|
69 |
+ |
|
|
70 |
+ |
Taken back to back, machine and drive idle, identical corpus and batch size, a
|
|
71 |
+ |
fresh vault each. They come out 168.3 against 168.4 files/s and 106.5 MB/s each,
|
|
72 |
+ |
so sharding the blob directory costs nothing measurable at this size. That is the
|
|
73 |
+ |
expected result rather than a disappointing one: at 2,241 blobs a flat directory
|
|
74 |
+ |
was never the problem. The layout change is for the 289k-file case, where the flat
|
|
75 |
+ |
root lost about 90% of its throughput.
|