Skip to main content

max / makeover-touch

24.6 KB · 635 lines History Blame Raw
1 #!/usr/bin/env python3
2 """Do the tree's in-house `version` requirements still resolve?
3
4 DO NOT EDIT IN PLACE. The master is _private/infra/bootstrap/githooks/internal-deps.py.
5
6 Usage:
7 internal-deps.py <tree-root> [repo-root] [pushed-sha]
8
9 With a repo root, only pairs that repo is on either side of can fail the run;
10 everything else is reported as a note. Without one, every pair is graded, which
11 is the whole-tree report:
12
13 python3 internal-deps.py ~/Code
14
15 The run grades THREE views and fails on any of them:
16
17 working copy what this machine builds today. The historical check.
18 as pushed the same question asked of the pushing repo's manifests AS THEY
19 EXIST AT THAT COMMIT, against the rest of the tree on disk.
20 Needs a pushed sha, so it is skipped in the by-hand report.
21 as published every requirement against the version the provider has actually
22 PUSHED, read from its last-fetched remote-tracking ref. This is
23 the only view that predicts a build on a machine that is not
24 this one.
25
26 WHY THE SECOND VIEW EXISTS. The first one reads manifests off the filesystem, so
27 an uncommitted forward-fix makes it grade text that git is not publishing. That
28 is not hypothetical: on 2026-08-24 mnw-cli's `synckit-client` requirement had
29 been advanced to "0.9" in the working copy and never committed, this gate printed
30 `internal deps coherent (42 requirements)`, the push went out, and Sando failed
31 to resolve `^0.8` against 0.9.0 minutes later. The gate was checking a tree that
32 was not the tree being published, and nothing distinguished that from real
33 coherence.
34
35 WHAT IT GRADES. Every dependency in the tree that carries both a `git` URL on one
36 of our forges and a `version` requirement, against the version in the working
37 copy of the crate that URL names. That is the pairing cargo enforces and the one
38 that broke: a requirement of "0.11" against a sibling that has moved to 0.14 is
39 not a warning, it is a graph that will not resolve on any machine.
40
41 WHY DISK AND REMOTES BOTH. `~/Code/.cargo/config.toml` patches every one of these
42 dependencies to the working copy in the tree, so what is on disk here is what
43 every local build reads: a bump that has not been pushed yet breaks its consumers
44 on this machine just as thoroughly, and finding that out at push time is the
45 point. But cargo resolves a git dependency against the branch head at the URL, so
46 the `[patch]` block also HIDES an unpushed sibling from every local check. That
47 gap cost a production build on 2026-08-26 (infra `5c4928c1`): MNW required quasi
48 "^0.63", quasi's working copy was 0.63.0 and its `mnw/main` was 0.56.0, both disk
49 views were clean, and Sando could not resolve. The three views answer three
50 different questions and all of them matter.
51
52 NETWORK. None on the happy path. The published view reads the last-fetched
53 remote-tracking ref, and only when a requirement FAILS against it does it fetch
54 that one repo's one branch and re-check, so a ref nobody has fetched since the
55 sibling was pushed cannot refuse a good push. A repo with no fetched remote at
56 all is reported as ungraded, never as passing.
57
58 WHAT IT DOES NOT GRADE, on purpose:
59
60 crates.io deps the makeover suite and friends resolve from the registry,
61 where working ahead of a release is normal and a tree
62 version above the published one is not a finding. The
63 sweep's `coherence` check grades those against the index.
64 path deps no version requirement to be wrong about.
65 ranges and wildcards `>=`, `<`, `*` and comma lists are deliberate statements
66 about a span, not a pin that drifts. Counted as unchecked.
67 """
68
69 import os
70 import re
71 import subprocess
72 import sys
73 import tomllib
74
75 # The forges that make a git URL ours. A dependency on somebody else's git repo
76 # is not something this tree can forward-fix.
77 OURS = re.compile(r"(makenot\.work|git\.sr\.ht/~maxmj)", re.I)
78
79 # Directories that hold code we do not grade: retired, staged for deletion, or
80 # not ours. Mirrors the sweep's exclusions rather than inventing a second list.
81 SKIP_DIRS = {
82 "target", ".git", "node_modules", "dist", "vendor",
83 "_archive", "_scratch", "trash", "_meta", "vtebench",
84 }
85 MAX_DEPTH = 4
86
87 DEP_SECTIONS = ("dependencies", "dev-dependencies", "build-dependencies")
88
89
90 def manifests(root):
91 """Every Cargo.toml in the tree, shallow-walked."""
92 out = []
93 stack = [(root, 0)]
94 while stack:
95 d, depth = stack.pop()
96 try:
97 entries = list(os.scandir(d))
98 except OSError:
99 continue
100 for e in entries:
101 if e.is_file() and e.name == "Cargo.toml":
102 out.append(e.path)
103 elif e.is_dir() and e.name not in SKIP_DIRS and depth < MAX_DEPTH:
104 stack.append((e.path, depth + 1))
105 return out
106
107
108 def load(path):
109 try:
110 with open(path, "rb") as fh:
111 return tomllib.load(fh)
112 except (OSError, tomllib.TOMLDecodeError):
113 return None
114
115
116 def dep_tables(doc):
117 """Every dependency table in a manifest, including per-target and workspace."""
118 for section in DEP_SECTIONS:
119 table = doc.get(section)
120 if isinstance(table, dict):
121 yield table
122 for cfg in (doc.get("target") or {}).values():
123 if not isinstance(cfg, dict):
124 continue
125 for section in DEP_SECTIONS:
126 table = cfg.get(section)
127 if isinstance(table, dict):
128 yield table
129 ws = doc.get("workspace") or {}
130 table = ws.get("dependencies")
131 if isinstance(table, dict):
132 yield table
133
134
135 def parse_version(v):
136 """A version as a 3-tuple, prerelease dropped. Junk sorts as (0, 0, 0)."""
137 core = str(v).split("+")[0].split("-")[0]
138 parts = []
139 for piece in core.split(".")[:3]:
140 try:
141 parts.append(int(piece))
142 except ValueError:
143 parts.append(0)
144 while len(parts) < 3:
145 parts.append(0)
146 return tuple(parts)
147
148
149 def satisfies(req, version):
150 """Cargo's default (caret) requirement semantics. None means 'not graded'.
151
152 The rule that matters here is the 0.x one: under 0.1.0 and above, the MINOR
153 is the compatibility boundary, which is why a 0.11 requirement rejects 0.14
154 outright rather than treating it as a newer patch.
155 """
156 req = req.strip()
157 if not req or any(c in req for c in "<>*,~"):
158 return None
159 # A prerelease satisfies nothing that does not ask for a prerelease of the
160 # same version, so a plain requirement rejects it. This is the shape the
161 # maturity ladder produces at beta entry: a sibling at 1.0.0-beta.1 does not
162 # resolve for a consumer requiring "1.0", and cargo says so.
163 if "-" in str(version).split("+")[0] and "-" not in req:
164 return False
165 exact = req.startswith("=")
166 req = req.lstrip("^=").strip()
167 if not req:
168 return None
169 given = req.split(".")
170 try:
171 r = [int(p) for p in given[:3]]
172 except ValueError:
173 return None
174 v = parse_version(version)
175 if exact:
176 return tuple(v[: len(r)]) == tuple(r)
177 if r[0] > 0:
178 return v[0] == r[0] and v[1:] >= tuple(r[1:] + [0] * (2 - len(r[1:])))
179 if len(r) == 1:
180 return v[0] == 0
181 if r[1] > 0:
182 return v[0] == 0 and v[1] == r[1] and v[2] >= (r[2] if len(r) > 2 else 0)
183 # 0.0.x: every patch is its own compatibility island.
184 if len(r) > 2:
185 return v[:3] == (0, 0, r[2])
186 return v[0] == 0 and v[1] == 0
187
188
189 def git_lines(repo, *args):
190 """Run git in `repo` and return stdout lines, or None if it failed."""
191 try:
192 out = subprocess.run(
193 ["git", "-C", repo, *args],
194 capture_output=True, text=True, check=True,
195 )
196 except (OSError, subprocess.CalledProcessError):
197 return None
198 return out.stdout.splitlines()
199
200
201 def git_manifests(repo, sha):
202 """Repo-relative paths of every Cargo.toml at `sha`, or None if unreadable."""
203 lines = git_lines(repo, "ls-tree", "-r", "--name-only", sha)
204 if lines is None:
205 return None
206 out = []
207 for rel in lines:
208 if os.path.basename(rel) != "Cargo.toml":
209 continue
210 if any(part in SKIP_DIRS for part in rel.split("/")):
211 continue
212 out.append(rel)
213 return out
214
215
216 def load_at(repo, sha, rel):
217 """One manifest as it exists at `sha`. None if missing or unparseable."""
218 lines = git_lines(repo, "show", f"{sha}:{rel}")
219 if lines is None:
220 return None
221 try:
222 return tomllib.loads("\n".join(lines))
223 except tomllib.TOMLDecodeError:
224 return None
225
226
227 def pushed_view(docs, repo, sha):
228 """`docs` with everything under `repo` replaced by its content at `sha`.
229
230 The rest of the tree stays as it is on disk, which is what a local build
231 resolves against either way. Returns None if the commit cannot be read, so
232 the caller can skip the view rather than invent a verdict about it.
233 """
234 rels = git_manifests(repo, sha)
235 if rels is None:
236 return None
237 out = {k: v for k, v in docs.items() if not k.startswith(repo + os.sep)}
238 for rel in rels:
239 out[os.path.join(repo, rel)] = load_at(repo, sha, rel)
240 return out
241
242
243 def repo_of(path, tree):
244 """The git repo `path` belongs to, or None if it is not in one under `tree`."""
245 d = os.path.dirname(path)
246 while d.startswith(tree):
247 if os.path.exists(os.path.join(d, ".git")):
248 return d
249 if d == tree:
250 break
251 parent = os.path.dirname(d)
252 if parent == d:
253 break
254 d = parent
255 return None
256
257
258 def publishing_ref(repo, cache):
259 """The remote-tracking ref a git dependency on `repo` would resolve against.
260
261 Cargo reads the branch head at the URL, and every in-house dependency URL is
262 on one of our forges (CLAUDE.md, "what each remote is for": `mnw` is the
263 public face, `srht` a backup, `astra` the private mirror). So prefer `mnw`,
264 then any other remote whose URL is ours, and fall back to `origin`.
265
266 Returns `<remote>/<branch>` or None when the repo has no such remote or the
267 ref has never been fetched. None is not a verdict: the caller reports the
268 repo as ungraded rather than inventing one.
269 """
270 if repo in cache:
271 return cache[repo]
272 ref = None
273 lines = git_lines(repo, "remote", "-v") or []
274 urls = {}
275 for line in lines:
276 parts = line.split()
277 if len(parts) >= 2:
278 urls.setdefault(parts[0], parts[1])
279 order = [r for r in ("mnw",) if r in urls]
280 order += [r for r, u in urls.items() if r not in order and OURS.search(u)]
281 order += [r for r in ("origin",) if r in urls and r not in order]
282 for remote in order:
283 head = git_lines(repo, "symbolic-ref", "--quiet", f"refs/remotes/{remote}/HEAD")
284 candidates = []
285 if head:
286 candidates.append(head[0].rsplit("/", 1)[-1])
287 candidates += ["main", "master"]
288 for branch in candidates:
289 if git_lines(repo, "rev-parse", "--verify", "--quiet",
290 f"refs/remotes/{remote}/{branch}"):
291 ref = f"{remote}/{branch}"
292 break
293 if ref:
294 break
295 cache[repo] = ref
296 return ref
297
298
299 def version_at(repo, ref, rel, cache):
300 """A crate's version in `repo` at `ref`, following a workspace inheritance.
301
302 `rel` is the manifest's path relative to the repo. Returns None when the
303 manifest is not at that ref at all, which is what a crate added since the
304 last push looks like.
305 """
306 key = (repo, ref, rel)
307 if key in cache:
308 return cache[key]
309 version = None
310 doc = load_at(repo, ref, rel)
311 if doc:
312 pkg = doc.get("package")
313 if isinstance(pkg, dict):
314 v = pkg.get("version")
315 if isinstance(v, str):
316 version = v
317 elif isinstance(v, dict) and v.get("workspace") is True:
318 # Walk up to the workspace root as it exists at the same ref.
319 d = os.path.dirname(rel)
320 while True:
321 root_rel = os.path.join(d, "Cargo.toml") if d else "Cargo.toml"
322 root = load_at(repo, ref, root_rel) if root_rel != rel else None
323 inherited = (
324 ((root or {}).get("workspace") or {}).get("package") or {}
325 ).get("version")
326 if isinstance(inherited, str):
327 version = inherited
328 break
329 if not d:
330 break
331 d = os.path.dirname(d)
332 cache[key] = version
333 return version
334
335
336 def analyze_published(docs, disk_docs, tree, repo, sha):
337 """Grade every requirement against what its provider has actually PUSHED.
338
339 This is the view that predicts a build somewhere other than this machine.
340 The other two read the provider's version off the filesystem, and the
341 `[patch]` block in ~/Code/.cargo/config.toml means that is what a local
342 build resolves -- but a git dependency resolves against the branch head at
343 the URL, so an unpushed sibling passes both of them and fails everywhere
344 else. That is exactly what happened on 2026-08-26: MNW required quasi
345 "^0.63", quasi's working copy was 0.63.0 and `mnw/main` was 0.56.0, both
346 existing views were clean, and Sando build 72 could not resolve.
347
348 The repo being pushed is read at `sha` rather than at its remote, since what
349 it is about to publish is the thing to grade. Every other repo is read at
350 its last-fetched remote ref: no network on the happy path. A break is
351 re-checked after fetching that one repo, so a stale ref cannot refuse a push
352 on its own.
353
354 Returns (broken, graded, ungraded), where ungraded maps a repo to why.
355 """
356 ref_cache, version_cache, fetched = {}, {}, set()
357 versions_on_disk = crate_index(disk_docs, tree)
358 broken, graded, ungraded = [], 0, {}
359
360 for consumer_manifest, name, req in requirements(docs):
361 known = versions_on_disk.get(name)
362 if known is None:
363 continue # Not in this tree; the disk views already say so.
364 provider_manifest = known[1]
365 provider_repo = repo_of(provider_manifest, tree)
366 if provider_repo is None:
367 ungraded.setdefault(os.path.dirname(provider_manifest), "not a git repo")
368 continue
369 if repo is not None and provider_repo == repo and sha:
370 # The repo under the hook: what it is about to publish is `sha`,
371 # which the "as pushed" view already read off disk into `docs`.
372 continue
373 ref = publishing_ref(provider_repo, ref_cache)
374 if ref is None:
375 ungraded.setdefault(provider_repo, "no fetched remote to read")
376 continue
377 rel = os.path.relpath(provider_manifest, provider_repo)
378 have = version_at(provider_repo, ref, rel, version_cache)
379 if have is None:
380 ungraded.setdefault(provider_repo, f"{name} is not at {ref} yet")
381 continue
382 verdict = satisfies(req, have)
383 if verdict is None:
384 continue
385 if not verdict and provider_repo not in fetched:
386 # Only now, and only for this one repo: a ref nobody has fetched
387 # since the sibling was pushed would otherwise refuse a good push.
388 fetched.add(provider_repo)
389 remote, branch = ref.split("/", 1)
390 git_lines(provider_repo, "fetch", "--quiet", remote, branch)
391 version_cache.pop((provider_repo, ref, rel), None)
392 have = version_at(provider_repo, ref, rel, version_cache) or have
393 verdict = satisfies(req, have)
394 graded += 1
395 if not verdict:
396 broken.append(
397 (consumer_manifest, name, req, have, provider_manifest, ref)
398 )
399 return broken, graded, ungraded
400
401
402 def crate_index(docs, tree):
403 """Crate name -> (version, manifest path), workspace inheritance resolved.
404
405 A member saying `version.workspace = true` gets its number from the root,
406 and reporting it as 0.0.0 would be a false break.
407 """
408 ws_version = {}
409 for p, doc in docs.items():
410 if not doc:
411 continue
412 v = ((doc.get("workspace") or {}).get("package") or {}).get("version")
413 if isinstance(v, str):
414 ws_version[os.path.dirname(p)] = v
415
416 def resolve_version(manifest_path, pkg):
417 v = pkg.get("version")
418 if isinstance(v, str):
419 return v
420 d = os.path.dirname(manifest_path)
421 while d.startswith(tree):
422 if d in ws_version:
423 return ws_version[d]
424 parent = os.path.dirname(d)
425 if parent == d:
426 break
427 d = parent
428 return None
429
430 versions = {}
431 for p, doc in docs.items():
432 if not doc:
433 continue
434 pkg = doc.get("package")
435 if not isinstance(pkg, dict) or not isinstance(pkg.get("name"), str):
436 continue
437 v = resolve_version(p, pkg)
438 if v:
439 versions[pkg["name"]] = (v, p)
440 return versions
441
442
443 def requirements(docs):
444 """Every in-house git+version pair: (consumer manifest, crate, requirement).
445
446 A dependency qualifies when it carries both a `git` URL on one of our forges
447 and a `version`. That is the pairing cargo enforces and the one that breaks:
448 a requirement of "0.11" against a sibling that has moved to 0.14 is not a
449 warning, it is a graph that will not resolve on any machine.
450 """
451 for p, doc in docs.items():
452 if not doc:
453 continue
454 for table in dep_tables(doc):
455 for key, spec in table.items():
456 if not isinstance(spec, dict):
457 continue
458 git = spec.get("git")
459 req = spec.get("version")
460 if not isinstance(git, str) or not isinstance(req, str):
461 continue
462 if not OURS.search(git):
463 continue
464 name = spec.get("package") if isinstance(spec.get("package"), str) else key
465 yield p, name, req
466
467
468 def analyze(docs, tree):
469 """Grade every in-house git+version pair in `docs` against the tree on disk.
470
471 Returns (broken, unchecked, absent, graded), where a broken entry is
472 (consumer manifest, crate, requirement, version found, provider manifest,
473 source label). The source label is None here: this view reads the version
474 off a manifest, and naming the manifest already says where it came from.
475 """
476 versions = crate_index(docs, tree)
477 broken, unchecked, absent, graded = [], 0, set(), 0
478 for p, name, req in requirements(docs):
479 known = versions.get(name)
480 if known is None:
481 # A repo that is not on this machine (ripgrow lives on mbp only).
482 # Not a finding: nothing here can be wrong about it.
483 absent.add(name)
484 continue
485 verdict = satisfies(req, known[0])
486 if verdict is None:
487 unchecked += 1
488 continue
489 graded += 1
490 if not verdict:
491 broken.append((p, name, req, known[0], known[1], None))
492 return broken, unchecked, absent, graded
493
494
495 def split_blame(broken, repo):
496 """Breaks this push owns, and breaks that were already there."""
497 ours, theirs = [], []
498 for item in broken:
499 consumer_manifest, _name, _req, _have, provider_manifest, _src = item
500 mine = repo is not None and (
501 consumer_manifest.startswith(repo + os.sep)
502 or provider_manifest.startswith(repo + os.sep)
503 )
504 (ours if mine else theirs).append(item)
505 return ours, theirs
506
507
508 def report(broken, repo, tree, label):
509 """Print one view's breaks. Returns True if this push has to be refused."""
510 ours, theirs = split_blame(broken, repo)
511
512 def rel(path):
513 return os.path.relpath(path, tree)
514
515 for consumer_manifest, name, req, have, provider_manifest, src in ours + theirs:
516 where = (
517 f"{src} has {have} ({rel(provider_manifest)})"
518 if src
519 else f"the tree has {have} ({rel(provider_manifest)})"
520 )
521 print(
522 f" [{label}] {rel(consumer_manifest)}: requires {name} \"{req}\", {where}",
523 file=sys.stderr,
524 )
525 if repo is None:
526 return bool(broken)
527 if not ours:
528 # Somebody else's skew. Worth seeing, never worth blocking this push on:
529 # a gate that fails for a reason the pusher cannot fix is a gate that
530 # gets bypassed by reflex, and then it is not a gate.
531 if theirs:
532 print(
533 f"pre-push: [{label}] {len(theirs)} unresolvable requirements "
534 "elsewhere in the tree (listed above, not this push's).",
535 )
536 return False
537 return True
538
539
540 def main():
541 if len(sys.argv) < 2:
542 print(__doc__.strip(), file=sys.stderr)
543 return 2
544 tree = os.path.realpath(sys.argv[1])
545 repo = os.path.realpath(sys.argv[2]) if len(sys.argv) > 2 else None
546 sha = sys.argv[3] if len(sys.argv) > 3 else None
547
548 docs = {p: load(p) for p in manifests(tree)}
549
550 views = [("working copy", docs)]
551 skipped_push_view = False
552 if repo and sha:
553 pushed = pushed_view(docs, repo, sha)
554 if pushed is None:
555 skipped_push_view = True
556 else:
557 views.append(("as pushed", pushed))
558
559 refuse = False
560 summaries = []
561 for label, view in views:
562 broken, unchecked, absent, graded = analyze(view, tree)
563 summaries.append((label, graded, unchecked, absent, bool(broken)))
564 if broken and report(broken, repo, tree, label):
565 refuse = True
566
567 # The third view: what the rest of the tree has actually PUSHED. Graded from
568 # the most authoritative consumer view available, so the requirements read
569 # are the ones about to be published.
570 consumer_view = views[-1][1]
571 pub_broken, pub_graded, ungraded = analyze_published(
572 consumer_view, docs, tree, repo, sha
573 )
574 summaries.append(("as published", pub_graded, 0, set(), bool(pub_broken)))
575 if pub_broken and report(pub_broken, repo, tree, "as published"):
576 refuse = True
577
578 if refuse:
579 bad = {lbl for lbl, _g, _u, _a, broke in summaries if broke}
580 clean = [lbl for lbl, _g, _u, _a, broke in summaries if not broke]
581 print("", file=sys.stderr)
582 print(
583 "pre-push: this push leaves a dependency that cannot resolve.\n"
584 " A version requirement states which major a consumer was written against,\n"
585 " so bumping a library and fixing its consumers is one pass (CLAUDE.md,\n"
586 " \"a breaking bump of an in-house crate is forward-fixed, in the same pass\").\n"
587 " Fix: bump the requirement in the manifests above, make the consumers\n"
588 " compile, and push them with this one.",
589 file=sys.stderr,
590 )
591 # Which view broke is the diagnosis, so say what the views disagree
592 # about rather than only that they disagree.
593 if "as published" in bad and "working copy" not in bad:
594 print(
595 " The working copy is fine and the published tree is not, so the\n"
596 " difference is what has been PUSHED: a git dependency resolves against\n"
597 " the branch head at the URL, and ~/Code/.cargo/config.toml's [patch]\n"
598 " block hides that locally. Push the sibling named above first.",
599 file=sys.stderr,
600 )
601 elif "as pushed" in bad and "working copy" not in bad:
602 print(
603 " The working copy is fine and the commit is not, so the difference is\n"
604 " what is COMMITTED. An uncommitted manifest edit is the usual cause.",
605 file=sys.stderr,
606 )
607 elif clean:
608 print(
609 f" Note: the {clean[0]} view is clean, so the views disagree; the one\n"
610 " that failed is named on each line above.",
611 file=sys.stderr,
612 )
613 return 1
614
615 for label, graded, unchecked, absent, _bad in summaries:
616 print(
617 f"pre-push: internal deps coherent [{label}] ({graded} requirements"
618 + (f", {unchecked} unchecked" if unchecked else "")
619 + (f", {len(absent)} crates not in this tree" if absent else "")
620 + ")."
621 )
622 for where, why in sorted(ungraded.items()):
623 # Never silently: a repo nobody could read is not a repo that passed.
624 print(
625 f"pre-push: [as published] {os.path.relpath(where, tree)} not graded ({why})."
626 )
627 if skipped_push_view:
628 # Never silently: a view that did not run must not read as one that passed.
629 print("pre-push: could not read the pushed commit; graded the working copy only.")
630 return 0
631
632
633 if __name__ == "__main__":
634 sys.exit(main())
635