Comment # 3 on bug 1231836 from Fabian Vogt

(In reply to Wenruo Qu from comment #2)
> I believe this bug is more about the init process, other than the btrfs
> itself.
> 
> The dmesg shows nothing wrong from btrfs, and I'm very confident that btrfs
> will be super noisy for anything unexpected (even for repairable corruption,
> it will output something about it).

FWICT it's the opposite, it shows that it's likely a kernel issue.

mount: /.snapshots: /dev/nvme0n1p2 already mounted on /.

That is printed by mount on -EBUSY, but btrfs explicitly handles mounting
subvolumes of the same filesystem separately, so unless /.snapshots is already
mounted this can't happen.

I had a quick look at kernel code (git grep EBUSY fs/btrfs) and found this
comment:

        /*
         * We got an EBUSY because our SB_RDONLY flag didn't match the existing
         * super block, so invert our setting here and retry the mount so we
         * can get our vfsmount.
         */

That triggered some unpleasant memories about the superblock ro flag flipping
during boot (boo#1156421). In this case a stress test would confirm that and
indeed it did!

Here's a reproducer:

#!/bin/sh
set -eu
# Create a btrfs image
dd if=/dev/zero of=btrfs bs=1M count=128
mkfs.btrfs btrfs
loop=$(losetup --show -f btrfs)

mkdir mnt
# With /.snapshot and /.snapshots/snapshot/1 subvolumes
mount "$loop" mnt
btrfs subvol create mnt/.snapshots
btrfs subvol set-default mnt/.snapshots
mkdir -p mnt/.snapshots/snapshot/1
btrfs subvol create mnt/.snapshots/snapshot/1/snapshot
umount mnt

mount "$loop" mnt
mkdir mnt/.snapshots

# First loop: make mnt ro/rw/ro/rw/ro/...
while mount -o remount,ro mnt; do mount -o remount,rw mnt; done &
bg=$!

# Second loop: mount and umount mnt/.snapshots
while mount "$loop" mnt/.snapshots -o subvol=.snapshots; do umount
mnt/.snapshots; done

# Cleanup
kill $bg
wait
umount -R mnt
losetup -d "$loop"
rmdir mnt

Here it fails within seconds:

...
Create subvolume 'mnt/.snapshots'
Create subvolume 'mnt/.snapshots/snapshot/1/snapshot'
mount: /tmp/mnt/.snapshots: /dev/loop0 already mounted on /tmp/mnt.
       dmesg(1) may have more information after failed mount system call.

Maybe SB_RDONLY flips while /.snapshots is being mounted? Just a guess:

1. / gets mounted ro in the initrd (SB readonly)
2. /.snapshots mount starts (SB readonly)
3. /usr/local mount starts (SB readonly)
4. /usr/local mount finishes (SB readwrite)
5. /.snapshots mount fails because SB is readwrite now?