xfs mount failure on rawhide

[jeckersb]

Seeing this in a few places, notably over in https://src.fedoraproject.org/rpms/bootc/pull-request/93 but also in our CI here like https://github.com/bootc-dev/bootc/pull/2145/checks?check_run_id=71548302273

Creating root filesystem (xfs) on device /dev/loop0p3 (size=10.2 GB)
> mkfs.xfs -m uuid=f36f7d0d-28b6-4271-8674-d1d76e54229f -L root /dev/loop0p3
mount: /run/bootc/mounts/rootfs: wrong fs type, bad option, bad superblock on /dev/loop0p3, missing codepage or helper program, or other error.
       dmesg(1) may have more information after failed mount system call.
error: Installing to disk: Creating rootfs: Failed to run command: Command {
    program: "mount",
    args: [
        "mount",
        "/dev/loop0p3",
        "/run/bootc/mounts/rootfs",
    ],
    create_pidfd: false,
}

[jeckersb]

LLM debugging results:

Root Cause

The failure is caused by kernel module auto-loading not working from inside the nested podman container in bcvk's to_disk flow. Specifically:

1. The bcvk to_disk flow boots a QEMU VM with kernel 7.0 from the rawhide image
2. Inside the VM, it runs podman run --privileged --pid=host ... bootc install to-disk
3. bootc creates an xfs filesystem with mkfs.xfs (succeeds)
4. bootc calls mount /dev/vdb3 /run/bootc/mounts/rootfs (fails)
5. The kernel tries to auto-detect the filesystem type. Since xfs is a module (CONFIG_XFS_FS=m) and not pre-loaded, the kernel tries request_module("fs-xfs") via call_usermodehelper
6. call_usermodehelper fails silently — it never successfully runs modprobe. The dmesg shows the kernel tried ext4, isofs, FAT, hfs, hfsplus (all built-in or already loaded) but never xfs
7. Without the xfs module loaded, mount fails with "wrong fs type"
   Pre-loading xfs before the nested podman run (modprobe xfs) completely fixes the mount. The subsequent failure is just disk space (4GB too small for the image).

Why auto-loading fails

The kernel's call_usermodehelper runs /usr/sbin/modprobe in the init task's namespaces (PID 1 of the QEMU VM). The bcvk ephemeral VM has a virtiofs root. Something about this environment — possibly related to how the virtiofs root interacts with call_usermodehelper in kernel 7.0, or how the usermodehelper subprocess inherits the environment — prevents modprobe from executing successfully. This could be a kernel 7.0 regression or a long-standing issue that's only exposed now because xfs is a module while ext4 is built-in.

Fix Options

There are two levels of fix possible:

1. In bcvk (to_disk.rs): Add modprobe xfs ext4 btrfs 2>/dev/null || true to the install script before the nested podman run, to pre-load common filesystem modules. This is a robust workaround.
2. In bootc (configure-rootfs): The default of xfs for ostree variant is what triggers this, since ext4 is likely built into the kernel. Changing the default or aligning the filesystem Justfile variable with the rootfs build arg would also avoid the issue. But this only masks the problem for ext4; btrfs would have the same issue if it's a module.
3. In bootc (install/baseline.rs): Add an explicit modprobe <fstype> before calling mount in the install code. This would make bootc more resilient regardless of the runtime environment.

[jeckersb]

And after pressing it to iterate a bit more:

Definitive Root Cause

The mount utility inside the bootc install container fails blkid detection and falls back to iterating /etc/filesystems + /proc/filesystems. The ftrace proves this:

• PID 1390 (mount inside bootc container) tries: fs-iso9660, fs-vfat, fs-hfs, fs-hfsplus — these are exactly the non-builtin types from /etc/filesystems, in order
• It never requests fs-xfs because xfs isn't in /etc/filesystems and isn't in /proc/filesystems (not loaded yet)
• blkid detection would have identified the filesystem as xfs and called mount(2) with -t xfs directly, but for some reason blkid fails inside the bootc process context
  The reason blkid fails inside the bootc container but works in standalone tests is still unclear, but the fix is clear and should be applied in bootc: since bootc already knows the filesystem type (it just ran mkfs.<type>), it should pass -t <fstype> to the mount command.
  This is not a kernel regression — it's a userspace issue where:

1. The mount utility's blkid probe fails in the bootc install context
2. The fallback to /etc/filesystems doesn't include xfs
3. xfs is a kernel module (=m) not yet loaded, so it's absent from /proc/filesystems
   The proper fix is in bootc's crates/lib/src/install/baseline.rs — the mount call at line 452 should pass the filesystem type that was just used for mkfs, avoiding any dependency on blkid auto-detection or module auto-loading entirely.

[cgwalters]

since bootc already knows the filesystem type (it just ran mkfs.), it should pass -t to the mount command.

Yes...sure.

blkid detection would have identified the filesystem as xfs and called mount(2) with -t xfs directly, but for some reason blkid fails inside the bootc process context

This feels like the real bug though, I'd gather an strace from the mount or equivalent

[jeckersb]

FWIW explicitly passing the fstype to mount gets the install to work, but then when it tries to boot the vm with libvirt it looks like it hangs after grub hands off to the kernel, will need some more investigation to see if it's related or if this is an entirely new problem...

[jeckersb]

More assisted research

Complete root cause: util-linux 2.42 udev cache regression

Commit 8bdc2546d in util-linux (libmount: read from udev, add --disable-libmount-udev-support, merged between v2.40 and v2.42) introduced a new optimization: libmount reads device tags from the udev database instead of probing the device with libblkid. The udev path is tried first, and if it returns any tags, the blkid fallback is skipped.
The problem: After mkfs.xfs is run inside a container, the udev database in the QEMU VM is stale — it has partition tags (PARTUUID, PARTLABEL) from when sfdisk created the partitions, but it does NOT have ID_FS_TYPE because udev wasn't triggered to re-probe after mkfs. read_from_udev() returns 0 ("found 2 tags") and the blkid direct probe is never called. The mount utility then has no filesystem type, falls back to iterating /etc/filesystems + /proc/filesystems, and xfs (a kernel module) is never tried.
The bug is in libmount/src/cache.c in mnt_cache_read_tags():

#ifdef USE_LIBMOUNT_UDEV_SUPPORT
	if (read_from_udev(cache, devname) == 0)
		return 0;    // <-- returns "success" even if TYPE tag is missing
#endif
	return read_from_blkid(cache, devname);

read_from_udev() returns 0 when it finds any tags (PARTUUID, PARTLABEL), even if the critical TYPE tag is absent. This prevents the fallback to read_from_blkid() which would probe the device superblock directly and correctly detect xfs.

• Fedora 42: util-linux 2.40 — no udev cache, uses blkid direct probe → works
• Fedora 45 (rawhide): util-linux 2.42 — uses udev cache, misses TYPE → fails

Our fix

The fix in bootc (passing -t to mount since we already know the type from mkfs) is correct regardless, but this documents the upstream issue that should also be fixed in util-linux.

[cgwalters]

This also relates to https://systemd.io/BLOCK_DEVICE_LOCKING/

But yeah I think we shouldn't be relying on udev

[jeckersb]

We noticed on a live call that the system didn't boot because the xfs module was not included in the initramfs, so it could not mount the filesystem. It's worth pointing out that I just did a manual install of rawhide with anaconda. I had to go thorugh the advanced partitioning since xfs is not normally an option, but I was able to install onto an xfs rootfs and reboot into it. The system came up ok and did include the module in the initramfs in that case:

[root@fedora ~]# lsinitrd /boot/initramfs-7.0.0-62.fc45.x86_64.img | grep xfs.ko
-rw-r--r--   1 root     root      1152704 Jan 21 19:00 usr/lib/modules/7.0.0-62.fc45.x86_64/kernel/fs/xfs/xfs.ko.xz
[root@fedora ~]# findmnt /
TARGET SOURCE    FSTYPE OPTIONS
/      /dev/vda4 xfs    rw,relatime,seclabel,inode64,logbufs=8,logbsize=32k,noquota

[cgwalters]

For a package-based install Anaconda always regenerates the initramfs locally. For image based installs we use a "nohostonly" initramfs build and that's a totally different process.