Skip to main content

max / alloy

install: create the account the install was for Two failures stood between a deployed disk and a usable machine, and the second was only reachable once the first was fixed. bootc install to-disk prints "Unmounting filesystems" and leaves a read-only bind of the target's /ostree/bootc/storage at /run/bootc/storage. A block device has one superblock, so mounting the same filesystem again returns the existing read-only one rather than a fresh mount, and mount -o rw reports "source write-protected" and exits 0. The device was never write-protected: blockdev --getro answered 0 throughout, which is why clearing a read-only flag on it, the obvious reading of that message, changed nothing. Release what bootc left behind before mounting, deepest first. Then useradd and chpasswd both failed against the mounted deployment with --root, useradd saying "failure while writing changes to /etc/passwd" over a filesystem that takes a touch in the same directory. The same commands with the same arguments under chroot write the same files and exit 0. Run the target's own shadow-utils inside it, which is also what should happen: the account is made by the tools the installed machine will run. systemd-firstboot --root is unaffected and keeps it. Verified end to end in a VM: the installer completes, and the disk boots on its own to a login as the account it was given, SELinux enforcing, hostname applied, the account in wheel. The medium's two verbose entries become one quiet debug entry. rd.debug and systemd.log_level=debug printed over the installer and over the shell meant for reading it; dmesg in that shell held the same answers.
Author: Max Johnson <me@maxj.phd> · 2026-07-21 21:48 UTC
Signed with PGP, not checked
Commit: 30c15ab2d898b920a851c2d886cb38ad62b8fc6d
Parent: cb8cf57
2 files changed, +141 insertions, -45 deletions
M build/make-iso.sh +12 -11
@@ -193,17 +193,18 @@
193 193 initrd /boot/initramfs.img
194 194 }
195 195
196 - # The verbose entries also arm alloy-debug-shell@.service, which puts a root
197 - # shell on tty2 and on the serial console. The live medium has no account and
198 - # a locked root, so without this a failure inside the live system can only be
199 - # read off whatever the installer printed before it stopped.
200 - menuentry "Install Alloy (verbose, with a root shell on tty9)" {
201 - linux /boot/vmlinuz $LIVE $SELINUX_PERMISSIVE alloy.debug rd.debug rd.shell systemd.log_level=debug
202 - initrd /boot/initramfs.img
203 - }
204 -
205 - menuentry "Install Alloy (SELinux off, verbose, with a root shell on tty9)" {
206 - linux /boot/vmlinuz $LIVE $SELINUX_OFF alloy.debug rd.debug rd.shell systemd.log_level=debug
196 + # alloy.debug arms alloy-debug-shell@.service, which puts a root shell on tty9
197 + # and on the serial console. The live medium has no account and a locked root,
198 + # so without this a failure inside the live system can only be read off
199 + # whatever the installer printed before it stopped.
200 + #
201 + # Quiet, like the default. This entry used to add rd.debug and
202 + # systemd.log_level=debug, which with console=tty0 print over the installer TUI
203 + # and over the shell itself. Every question asked of the live system so far has
204 + # been answered from the shell, with dmesg and findmnt, which hold the same
205 + # information and hold still while being read.
206 + menuentry "Install Alloy (root shell on tty9)" {
207 + linux /boot/vmlinuz $CMDLINE alloy.debug
207 208 initrd /boot/initramfs.img
208 209 }
209 210
@@ -289,24 +289,68 @@
289 289 .arg(disk)
290 290 }
291 291
292 - /// Clear the read-only flag on a block device.
292 + /// Ask lsblk which mountpoints are backed by `device`, one per line.
293 293 ///
294 - /// Immediately after `bootc install to-disk` completes, the target's root
295 - /// partition reads as write-protected, and mounting it gives
296 - /// "WARNING: source write-protected, mounted read-only" followed by every
297 - /// configure command failing with `Read-only file system`. The install itself
298 - /// is fine: the same disk, examined from another machine afterwards, reports
299 - /// `blockdev --getro` of 0 and mounts read-write.
294 + /// lsblk rather than `findmnt --source`, which answers the same question and
295 + /// reads better, because findmnt exits 1 when nothing matches and a non-zero
296 + /// exit fails the whole sequence. "Nothing is mounted" is the ordinary case
297 + /// here, not an error. lsblk prints an empty list and exits 0, and it does
298 + /// report bind mounts of subdirectories, which is the case that matters.
299 + fn mounts_of(device: &str) -> Invocation {
300 + Invocation::new("lsblk").args(["-n", "-o", "MOUNTPOINTS", device])
301 + }
302 +
303 + /// Mountpoints from [`mounts_of`] output, deepest first.
300 304 ///
301 - /// So this clears a flag that is set in the live session and does not outlive
302 - /// it. What sets it has not been identified; the candidates are bootc's own
303 - /// finalize and a stale flag left by the partition table re-read. That is why
304 - /// the mount is verified afterwards rather than trusted: if the cause turns
305 - /// out to be something this does not address, the install should say so
306 - /// rather than proceed to write an account into a read-only filesystem and
307 - /// report success.
308 - fn clear_read_only(device: &str) -> Invocation {
309 - Invocation::new("blockdev").args(["--setrw", device])
305 + /// `bootc install to-disk` prints "Unmounting filesystems" and then leaves one
306 + /// of its own behind: a bind of the target's `/ostree/bootc/storage` at
307 + /// `/run/bootc/storage`, mounted read-only. That single leftover is why the
308 + /// installer could not write the account. A block device has one superblock,
309 + /// so a second mount of an already-mounted filesystem does not get its own:
310 + /// the kernel hands back the existing one, read-only flag included, and
311 + /// `mount -o rw` reports "source write-protected, mounted read-only" and
312 + /// exits 0. The device itself is not write-protected. `blockdev --getro`
313 + /// answers 0 throughout, which is why clearing a read-only flag on it, the
314 + /// obvious reading of that message, changed nothing.
315 + ///
316 + /// Deepest first so that a leftover nested inside another comes off before the
317 + /// mount it sits under.
318 + fn leftover_mounts(listing: &str) -> Vec<String> {
319 + let mut mounts: Vec<String> = listing
320 + .lines()
321 + .map(str::trim)
322 + .filter(|line| !line.is_empty())
323 + .map(str::to_string)
324 + .collect();
325 + mounts.sort_by_key(|mount| std::cmp::Reverse(mount.len()));
326 + mounts
327 + }
328 +
329 + /// Unmount one leftover so the target's superblock is released.
330 + fn unmount(target: &str) -> Invocation {
331 + Invocation::new("umount").arg(target)
332 + }
333 +
334 + /// Run one of the target's own programs inside it, rather than pointing the
335 + /// live system's copy at it with `--root`.
336 + ///
337 + /// shadow-utils' `--root` does not work against a mounted ostree deployment
338 + /// here. `useradd --root` and `chpasswd --root` both exit 1, the first saying
339 + /// "failure while writing changes to /etc/passwd", while the same commands
340 + /// with the same arguments under `chroot` exit 0 and write the files. Checked
341 + /// directly against a deployed target: the mount is `rw` and `touch` inside
342 + /// the deployment's `/etc` succeeds either way, so the tree is writable and
343 + /// the difference is in how shadow-utils reaches it.
344 + ///
345 + /// Only shadow-utils is affected. `systemd-firstboot --root` writes the
346 + /// hostname into the same deployment without complaint, so it keeps its own
347 + /// flag rather than being wrapped for symmetry.
348 + ///
349 + /// Running the target's binaries also describes what should happen: the
350 + /// account is created by the shadow-utils the installed machine will run,
351 + /// reading that machine's `/etc/login.defs`.
352 + fn in_target(root: &str, program: &str) -> Invocation {
353 + Invocation::new("chroot").arg(root).arg(program)
310 354 }
311 355
312 356 /// Ask findmnt for the mount options in effect at `mount`.
@@ -570,20 +614,20 @@
570 614 // administer itself.
571 615 //
572 616 // --no-create-home despite the account needing one: --create-home
573 - // resolves the path against --root, which puts the directory in the
617 + // resolves the path inside the target, which puts the directory in the
574 618 // deployment's own var, where nothing will ever look for it. useradd
575 - // says as much — "chown on '/var/home/<user>' failed" — because the
619 + // says as much, "chown on '/var/home/<user>' failed", because the
576 620 // stateroot var ships with cache, lib, log, run and tmp and no home at
577 621 // all. The directory is made below, in the var that is real.
578 622 // Prints "Creating mailbox file: No such file or directory" and exits
579 623 // 0 anyway: the stateroot var has no mail spool for it to write into.
580 624 // Harmless, but the run screen streams this, so the user watching
581 625 // their install reads it as a failure. Left alone rather than
582 - // papered over -- a spool nothing on the image delivers to would be
626 + // papered over: a spool nothing on the image delivers to would be
583 627 // invented purely to quiet a warning.
584 628 Stage::Run(
585 - Invocation::new("useradd")
586 - .args(["--root", root, "--no-create-home"])
629 + in_target(root, "useradd")
630 + .arg("--no-create-home")
587 631 .args(["--home-dir", &installed_home])
588 632 .args(["--shell", LOGIN_SHELL])
589 633 .args(["--groups", "wheel"])
@@ -616,8 +660,8 @@
616 660 // the wire format, and with a trailing newline because chpasswd parses
617 661 // lines and a final one without it is silently ignored by some builds.
618 662 Stage::Run(
619 - Invocation::new("chpasswd")
620 - .args(["--encrypted", "--root", root])
663 + in_target(root, "chpasswd")
664 + .arg("--encrypted")
621 665 .stdin(Secret::new(format!("{username}:{hash}\n"))),
622 666 ),
623 667 // Upstream: "optional, but recommended to run as the penultimate step
@@ -695,7 +739,6 @@
695 739 let hostname = hostname.to_string();
696 740 let username = username.to_string();
697 741 let password = password.to_string();
698 - let target_disk = disk.to_string();
699 742
700 743 vec![
701 744 // --wipe is explicit rather than implied by the confirm the user just
@@ -737,10 +780,20 @@
737 780 then: Box::new(move |listing| {
738 781 let partition = root_partition(listing)?.to_string();
739 782 Ok(vec![
740 - // Both the disk and the partition: the flag can sit on
741 - // either, and a partition inherits its disk's.
742 - Stage::Run(clear_read_only(&target_disk)),
743 - Stage::Run(clear_read_only(&partition)),
783 + // Release whatever bootc left mounted on the partition
784 + // before mounting it. Its own leftover is read-only, and a
785 + // second mount of a filesystem that is already mounted
786 + // shares the first one's superblock rather than getting a
787 + // fresh one, so the read-only travels to this mount too.
788 + Stage::Resolve {
789 + invocation: mounts_of(&partition),
790 + then: Box::new(move |listing| {
791 + Ok(leftover_mounts(listing)
792 + .iter()
793 + .map(|target| Stage::Run(unmount(target)))
794 + .collect())
795 + }),
796 + },
744 797 Stage::Run(
745 798 Invocation::new("mount")
746 799 .args(["-o", "rw"])
@@ -2285,6 +2338,19 @@
2285 2338 .collect()
2286 2339 }
2287 2340
2341 + /// Whether `line` runs `program`, wrapped in a chroot or not.
2342 + ///
2343 + /// The target's own binaries are run inside it, so the command line for
2344 + /// useradd reads `chroot <root> useradd ...`. Matching only the first word
2345 + /// would make every test about those commands silently stop finding them.
2346 + fn runs(line: &str, program: &str) -> bool {
2347 + line.starts_with(program)
2348 + || line
2349 + .strip_prefix("chroot ")
2350 + .and_then(|rest| rest.split_once(' '))
2351 + .is_some_and(|(_root, command)| command.starts_with(program))
2352 + }
2353 +
2288 2354 /// The one configure command running `program`.
2289 2355 ///
2290 2356 /// By program rather than by index: the plan gains and loses steps, and a
@@ -2293,7 +2359,7 @@
2293 2359 fn command_starting(program: &str) -> String {
2294 2360 let mut found: Vec<String> = configured()
2295 2361 .into_iter()
2296 - .filter(|line| line.starts_with(program))
2362 + .filter(|line| runs(line, program))
2297 2363 .collect();
2298 2364
2299 2365 assert_eq!(found.len(), 1, "expected exactly one {program}: {found:#?}");
@@ -2346,7 +2412,7 @@
2346 2412 fn the_login_shell_is_checked_before_the_account_is_made() {
2347 2413 let lines = configured();
2348 2414 let check = &lines[0];
2349 - let useradd = lines.iter().position(|l| l.starts_with("useradd"));
2415 + let useradd = lines.iter().position(|l| runs(l, "useradd"));
2350 2416
2351 2417 assert!(check.contains("/etc/shells"), "{check}");
2352 2418 assert!(useradd.expect("the plan creates an account") > 0);
@@ -2394,14 +2460,15 @@
2394 2460 );
2395 2461 }
2396 2462
2397 - // The password reaches chpasswd already hashed: `chpasswd --root` hashes
2398 - // through the host's PAM, and the host's PAM cannot service the target
2399 - // ("Module is unknown"). --encrypted skips PAM entirely.
2463 + // The password reaches chpasswd already hashed. Without --encrypted
2464 + // chpasswd hashes through PAM, and a PAM stack reached across an install
2465 + // boundary is not something to depend on ("Module is unknown" is how the
2466 + // host's answered for the target). --encrypted skips PAM entirely.
2400 2467 #[test]
2401 2468 fn the_password_is_handed_over_already_hashed() {
2402 2469 let chpasswd = configured()
2403 2470 .into_iter()
2404 - .find(|line| line.starts_with("chpasswd"))
2471 + .find(|line| runs(line, "chpasswd"))
2405 2472 .expect("the plan sets a password");
2406 2473
2407 2474 assert!(chpasswd.contains("--encrypted"), "{chpasswd}");
@@ -2607,6 +2674,34 @@
2607 2674 assert!(writable_mount("ro,relatime,rwtest").is_err());
2608 2675 }
2609 2676
2677 + // What bootc actually leaves behind, as lsblk prints it: the target root
2678 + // mounted nowhere yet, and bootc's own read-only bind still up.
2679 + #[test]
2680 + fn a_leftover_bootc_mount_is_found() {
2681 + assert_eq!(leftover_mounts("/run/bootc/storage\n"), [
2682 + "/run/bootc/storage"
2683 + ]);
2684 + }
2685 +
2686 + // The ordinary case on a disk bootc has released properly. lsblk prints a
2687 + // blank line for a partition with no mountpoint, which is not a mountpoint
2688 + // called "".
2689 + #[test]
2690 + fn nothing_mounted_is_nothing_to_unmount() {
2691 + assert!(leftover_mounts("\n").is_empty());
2692 + assert!(leftover_mounts("").is_empty());
2693 + }
2694 +
2695 + // Deepest first, so a leftover inside another comes off before the mount
2696 + // it sits under: unmounting the outer one first would fail.
2697 + #[test]
2698 + fn nested_leftovers_unmount_from_the_inside_out() {
2699 + assert_eq!(leftover_mounts("/run/bootc\n/run/bootc/storage\n"), [
2700 + "/run/bootc/storage",
2701 + "/run/bootc"
2702 + ]);
2703 + }
2704 +
2610 2705 // Absent means running inside a container during development, where
2611 2706 // bootc installs the image it is running and needs no help.
2612 2707 #[test]