It turns out that you can unmount btrfs subvolumes just fine; but at least one of them, /mnt/var, tends to be busy because libzypp uses /mnt/var/cache/zypp/packages/... for its package cache. If unmounting that one fails with a "busy" error, unmounting its parent Btrfs main volume at /mnt also fails with the same error.