Skip to main content

max / audiofiles

Take the internal-deps gate with its crates.io view The gate graded three views: the working copy, the pushed manifests, and each provider at its last-fetched remote ref. None of them saw a crates.io sibling, so a consumer written against an unpublished version of a published crate passed here and failed anywhere the registry is the source. The fourth view reads the sparse index and blames consumer-side only, since blaming the provider would refuse the push that puts a crate on the tag Bento publishes from. Master copy: _private/infra/bootstrap/githooks/internal-deps.py.
Author: Max Johnson <me@maxj.phd> · 2026-08-28 21:54 UTC
Signed with PGP, not checked
Commit: 7fc85cb2025b163b04f26a1b55f86e6600185193
Parent: 613dd2e
1 file changed, +283 insertions, -13 deletions
M scripts/githooks/internal-deps.py +283 -13
@@ -12,7 +12,7 @@
12 12
13 13 python3 internal-deps.py ~/Code
14 14
15 - The run grades THREE views and fails on any of them:
15 + The run grades FOUR views and fails on any of them:
16 16
17 17 working copy what this machine builds today. The historical check.
18 18 as pushed the same question asked of the pushing repo's manifests AS THEY
@@ -22,6 +22,11 @@
22 22 PUSHED, read from its last-fetched remote-tracking ref. This is
23 23 the only view that predicts a build on a machine that is not
24 24 this one.
25 + as published to crates.io
26 + the same question for the siblings that are consumed from the
27 + REGISTRY rather than by git URL: the makeover suite, alloy_tui,
28 + pter. A requirement only the local checkout satisfies is a
29 + finding, and the remedy is to publish rather than to push.
25 30
26 31 WHY THE SECOND VIEW EXISTS. The first one reads manifests off the filesystem, so
27 32 an uncommitted forward-fix makes it grade text that git is not publishing. That
@@ -49,28 +54,56 @@
49 54 views were clean, and Sando could not resolve. The three views answer three
50 55 different questions and all of them matter.
51 56
57 + WHY THE FOURTH VIEW EXISTS. The three above all pair a requirement against the
58 + sibling repo its git URL names, and a makeover crate has no git URL in a consumer
59 + manifest: it is consumed from crates.io. So the failure class this gate exists to
60 + stop -- bump a library, forget its consumers -- had a second half that was not
61 + merely ungraded but confidently reported coherent. On 2026-08-28 quasi-basics
62 + required makeover-layout "^0.38.0" while the index's newest was 0.37.0 and 0.38.0
63 + existed in one working directory; `cargo check` could not select a version and
64 + this gate printed `internal deps coherent [as published] (42 requirements)`. The
65 + same afternoon, stripping the [patch.crates-io] blocks mid-publish put MNW/server
66 + on makeover "^3.1" against an index whose newest was 3.0.0, with the same verdict.
67 + That window -- manifest bumped, crate not yet on the index -- is a normal stage of
68 + any correct publish order, short when the publish succeeds and unbounded when it
69 + fails partway across a seven-crate suite. makeover-layout carries `links`, so one
70 + unpublished crate in the suite takes down every consumer in the graph at once.
71 +
72 + Blame for this view is CONSUMER-side only. Pushing the provider is how a crate
73 + reaches the index in the first place (Bento publishes from a pushed tag), so
74 + refusing that push because its own new version is unpublished would deadlock the
75 + release. What is worth refusing is a push that carries a requirement nothing on
76 + the index satisfies.
77 +
52 78 NETWORK. None on the happy path. The published view reads the last-fetched
53 79 remote-tracking ref, and only when a requirement FAILS against it does it fetch
54 80 that one repo's one branch and re-check, so a ref nobody has fetched since the
55 81 sibling was pushed cannot refuse a good push. A repo with no fetched remote at
56 - all is reported as ungraded, never as passing.
82 + all is reported as ungraded, never as passing. The crates.io view works the same
83 + way one layer over: it reads cargo's own sparse-index cache under
84 + ~/.cargo/registry/index, and only a requirement that fails against the cache
85 + costs one HTTPS fetch of that crate's index file, so a cache older than the
86 + sibling's release cannot refuse a good push either. A crate whose index file can
87 + be read from neither place is reported as ungraded.
57 88
58 89 WHAT IT DOES NOT GRADE, on purpose:
59 90
60 - crates.io deps the makeover suite and friends resolve from the registry,
61 - where working ahead of a release is normal and a tree
62 - version above the published one is not a finding. The
63 - sweep's `coherence` check grades those against the index.
91 + third-party deps a version requirement on somebody else's crate is not
92 + something this tree can forward-fix.
64 93 path deps no version requirement to be wrong about.
65 94 ranges and wildcards `>=`, `<`, `*` and comma lists are deliberate statements
66 95 about a span, not a pin that drifts. Counted as unchecked.
67 96 """
68 97
98 + import glob
99 + import json
69 100 import os
70 101 import re
71 102 import subprocess
72 103 import sys
73 104 import tomllib
105 + import urllib.error
106 + import urllib.request
74 107
75 108 # The forges that make a git URL ours. A dependency on somebody else's git repo
76 109 # is not something this tree can forward-fix.
@@ -86,6 +119,10 @@
86 119
87 120 DEP_SECTIONS = ("dependencies", "dev-dependencies", "build-dependencies")
88 121
122 + # The crates.io view's label. Long on purpose: "as published" already means the
123 + # git-URL siblings, and the remedy for the two is different.
124 + REGISTRY_LABEL = "as published to crates.io"
125 +
89 126
90 127 def manifests(root):
91 128 """Every Cargo.toml in the tree, shallow-walked."""
@@ -465,6 +502,205 @@
465 502 yield p, name, req
466 503
467 504
505 + # --- the crates.io view -------------------------------------------------------
506 +
507 + # Cargo's own sparse-index cache. One file per crate, holding the same
508 + # newline-delimited JSON the registry serves, prefixed by a format byte and an
509 + # etag and separated by NULs. Reading it is why the happy path costs no network.
510 + INDEX_CACHE_GLOB = os.path.expanduser(
511 + "~/.cargo/registry/index/index.crates.io-*/.cache"
512 + )
513 + INDEX_URL = "https://index.crates.io"
514 + INDEX_TIMEOUT = 5
515 +
516 +
517 + def index_prefix(name):
518 + """The registry's directory prefix for a crate name (cargo's own scheme)."""
519 + n = name.lower()
520 + if len(n) == 1:
521 + return "1"
522 + if len(n) == 2:
523 + return "2"
524 + if len(n) == 3:
525 + return os.path.join("3", n[0])
526 + return os.path.join(n[:2], n[2:4])
527 +
528 +
529 + def parse_index_blob(blob):
530 + """Every non-yanked version in an index file, newest first.
531 +
532 + Takes the cache format and the wire format both: the cache is the wire
533 + format with a header and NUL separators, so splitting on NUL and newline
534 + and keeping whatever parses as a version record covers each of them.
535 + """
536 + if isinstance(blob, bytes):
537 + blob = blob.decode("utf-8", "replace")
538 + out = []
539 + for chunk in blob.replace("\x00", "\n").splitlines():
540 + chunk = chunk.strip()
541 + if not chunk.startswith("{"):
542 + continue
543 + try:
544 + rec = json.loads(chunk)
545 + except ValueError:
546 + continue
547 + vers = rec.get("vers")
548 + if isinstance(vers, str) and not rec.get("yanked"):
549 + out.append(vers)
550 + out.sort(key=parse_version, reverse=True)
551 + return out
552 +
553 +
554 + def index_from_cache(name):
555 + """Published versions from cargo's sparse-index cache, or None if absent."""
556 + for cache in glob.glob(INDEX_CACHE_GLOB):
557 + path = os.path.join(cache, index_prefix(name), name.lower())
558 + try:
559 + with open(path, "rb") as fh:
560 + return parse_index_blob(fh.read())
561 + except OSError:
562 + continue
563 + return None
564 +
565 +
566 + def index_from_network(name):
567 + """Published versions from the sparse index itself, or None if unreachable."""
568 + url = f"{INDEX_URL}/{index_prefix(name)}/{name.lower()}"
569 + try:
570 + with urllib.request.urlopen(url, timeout=INDEX_TIMEOUT) as resp:
571 + return parse_index_blob(resp.read())
572 + except (urllib.error.URLError, OSError, ValueError):
573 + return None
574 +
575 +
576 + def published_versions(name, cache, refresh=False):
577 + """Every non-yanked published version of `name`, newest first.
578 +
579 + Reads cargo's cache first and goes to the network only when the caller says
580 + the cached answer was not good enough, which is the same escalation the
581 + remote-ref view does: a cache older than the sibling's release must not be
582 + able to refuse a good push.
583 + """
584 + if not refresh and name in cache:
585 + return cache[name]
586 + versions = None if refresh else index_from_cache(name)
587 + if versions is None:
588 + versions = index_from_network(name)
589 + cache[name] = versions
590 + return versions
591 +
592 +
593 + def publishable_index(docs, tree):
594 + """In-house crates that go to crates.io: name -> (version, manifest path).
595 +
596 + `publish = false` is the marker for everything that does not (quasi, shop,
597 + everycycle, wam), and it inherits from the workspace root the same way
598 + `version` does, so resolve it the same way rather than reading the member
599 + alone.
600 + """
601 + ws_publish = {}
602 + for p, doc in docs.items():
603 + if not doc:
604 + continue
605 + v = ((doc.get("workspace") or {}).get("package") or {}).get("publish")
606 + if isinstance(v, bool):
607 + ws_publish[os.path.dirname(p)] = v
608 +
609 + def publishes(manifest_path, pkg):
610 + v = pkg.get("publish")
611 + if isinstance(v, bool):
612 + return v
613 + if isinstance(v, list):
614 + return bool(v)
615 + if isinstance(v, dict) and v.get("workspace") is True:
616 + d = os.path.dirname(manifest_path)
617 + while d.startswith(tree):
618 + if d in ws_publish:
619 + return ws_publish[d]
620 + parent = os.path.dirname(d)
621 + if parent == d:
622 + break
623 + d = parent
624 + return True
625 +
626 + out = {}
627 + for name, (version, manifest) in crate_index(docs, tree).items():
628 + doc = docs.get(manifest) or {}
629 + pkg = doc.get("package")
630 + if isinstance(pkg, dict) and publishes(manifest, pkg):
631 + out[name] = (version, manifest)
632 + return out
633 +
634 +
635 + def registry_requirements(docs, publishable):
636 + """Every requirement on an in-house crate taken from the registry.
637 +
638 + The pairing is the mirror image of `requirements()`: a `version` and NO git
639 + URL, naming a crate this tree both holds and publishes. A path dependency is
640 + excluded for the same reason it is everywhere else here -- what it resolves
641 + against is the file next to it, not a release.
642 + """
643 + for p, doc in docs.items():
644 + if not doc:
645 + continue
646 + for table in dep_tables(doc):
647 + for key, spec in table.items():
648 + req = None
649 + if isinstance(spec, str):
650 + req = spec
651 + elif isinstance(spec, dict):
652 + if spec.get("git") or spec.get("path"):
653 + continue
654 + if isinstance(spec.get("version"), str):
655 + req = spec["version"]
656 + if req is None:
657 + continue
658 + name = key
659 + if isinstance(spec, dict) and isinstance(spec.get("package"), str):
660 + name = spec["package"]
661 + if name in publishable:
662 + yield p, name, req
663 +
664 +
665 + def analyze_registry(docs, tree):
666 + """Grade every registry requirement on an in-house crate against the index.
667 +
668 + The escalation is deliberate and is what keeps the happy path free: the
669 + local version is tried first, and the index is consulted only for a
670 + requirement the checkout is the one thing satisfying (or that nothing
671 + satisfies). Working ahead of a release is fine; requiring a version that
672 + exists in no published release is not, because no other machine can resolve
673 + it and `links` on makeover-layout makes that the whole graph at once.
674 +
675 + Returns (broken, graded, unchecked, ungraded).
676 + """
677 + publishable = publishable_index(docs, tree)
678 + index_cache = {}
679 + broken, graded, unchecked, ungraded = [], 0, 0, {}
680 +
681 + for consumer_manifest, name, req in registry_requirements(docs, publishable):
682 + local_version, provider_manifest = publishable[name]
683 + if satisfies(req, local_version) is None:
684 + unchecked += 1
685 + continue
686 + versions = published_versions(name, index_cache)
687 + if versions is None:
688 + ungraded.setdefault(name, "no index entry, cached or fetched")
689 + continue
690 + if not any(satisfies(req, v) for v in versions):
691 + # A cache older than the sibling's release would otherwise refuse a
692 + # good push, so pay for one fetch before calling it broken.
693 + versions = published_versions(name, index_cache, refresh=True) or versions
694 + graded += 1
695 + if not any(satisfies(req, v) for v in versions):
696 + newest = versions[0] if versions else "nothing published"
697 + broken.append(
698 + (consumer_manifest, name, req, newest, provider_manifest,
699 + "the index")
700 + )
701 + return broken, graded, unchecked, ungraded
702 +
703 +
468 704 def analyze(docs, tree):
469 705 """Grade every in-house git+version pair in `docs` against the tree on disk.
470 706
@@ -492,22 +728,28 @@
492 728 return broken, unchecked, absent, graded
493 729
494 730
495 - def split_blame(broken, repo):
496 - """Breaks this push owns, and breaks that were already there."""
731 + def split_blame(broken, repo, blame_provider=True):
732 + """Breaks this push owns, and breaks that were already there.
733 +
734 + `blame_provider` is off for the crates.io view: there, the provider side of
735 + a break is a crate that has been bumped and not released yet, and pushing it
736 + is how it reaches the index at all (Bento publishes from a pushed tag). Only
737 + the consumer side of that view is a push worth refusing.
738 + """
497 739 ours, theirs = [], []
498 740 for item in broken:
499 741 consumer_manifest, _name, _req, _have, provider_manifest, _src = item
500 742 mine = repo is not None and (
501 743 consumer_manifest.startswith(repo + os.sep)
502 - or provider_manifest.startswith(repo + os.sep)
744 + or (blame_provider and provider_manifest.startswith(repo + os.sep))
503 745 )
504 746 (ours if mine else theirs).append(item)
505 747 return ours, theirs
506 748
507 749
508 - def report(broken, repo, tree, label):
750 + def report(broken, repo, tree, label, blame_provider=True):
509 751 """Print one view's breaks. Returns True if this push has to be refused."""
510 - ours, theirs = split_blame(broken, repo)
752 + ours, theirs = split_blame(broken, repo, blame_provider)
511 753
512 754 def rel(path):
513 755 return os.path.relpath(path, tree)
@@ -575,6 +817,20 @@
575 817 if pub_broken and report(pub_broken, repo, tree, "as published"):
576 818 refuse = True
577 819
820 + # The fourth view: the siblings that are consumed from crates.io rather than
821 + # by git URL. Nothing above can see them, because the pairing every other
822 + # view makes is against the repo a git URL names.
823 + reg_broken, reg_graded, reg_unchecked, reg_ungraded = analyze_registry(
824 + consumer_view, tree
825 + )
826 + summaries.append(
827 + (REGISTRY_LABEL, reg_graded, reg_unchecked, set(), bool(reg_broken))
828 + )
829 + if reg_broken and report(
830 + reg_broken, repo, tree, REGISTRY_LABEL, blame_provider=False
831 + ):
832 + refuse = True
833 +
578 834 if refuse:
579 835 bad = {lbl for lbl, _g, _u, _a, broke in summaries if broke}
580 836 clean = [lbl for lbl, _g, _u, _a, broke in summaries if not broke]
@@ -590,6 +846,14 @@
590 846 )
591 847 # Which view broke is the diagnosis, so say what the views disagree
592 848 # about rather than only that they disagree.
849 + if REGISTRY_LABEL in bad:
850 + print(
851 + " A requirement above resolves against no version on crates.io, so the\n"
852 + " fix is to PUBLISH the sibling, not to push it. Until it is on the\n"
853 + " index no machine can resolve the graph, this one included once the\n"
854 + " [patch.crates-io] block is out of the way.",
855 + file=sys.stderr,
856 + )
593 857 if "as published" in bad and "working copy" not in bad:
594 858 print(
595 859 " The working copy is fine and the published tree is not, so the\n"
@@ -612,9 +876,13 @@
612 876 )
613 877 return 1
614 878
615 - for label, graded, unchecked, absent, _bad in summaries:
879 + for label, graded, unchecked, absent, bad in summaries:
880 + # A view that found a break somewhere else in the tree is not a view
881 + # that passed. Reaching here means the break is not this push's to fix,
882 + # which is a reason not to refuse and never a reason to print coherent.
883 + verdict = "internal deps coherent" if not bad else "internal deps BROKEN elsewhere"
616 884 print(
617 - f"pre-push: internal deps coherent [{label}] ({graded} requirements"
885 + f"pre-push: {verdict} [{label}] ({graded} requirements"
618 886 + (f", {unchecked} unchecked" if unchecked else "")
619 887 + (f", {len(absent)} crates not in this tree" if absent else "")
620 888 + ")."
@@ -624,6 +892,8 @@
624 892 print(
625 893 f"pre-push: [as published] {os.path.relpath(where, tree)} not graded ({why})."
626 894 )
895 + for name, why in sorted(reg_ungraded.items()):
896 + print(f"pre-push: [{REGISTRY_LABEL}] {name} not graded ({why}).")
627 897 if skipped_push_view:
628 898 # Never silently: a view that did not run must not read as one that passed.
629 899 print("pre-push: could not read the pushed commit; graded the working copy only.")