On Sun, 2021-05-23 at 17:51 +0200, Johannes Kastl wrote:

Hi all,

I had to replace one of my nodes, and the replacement does not properly start SSH. No problem, mounting the rootfs from a rescue disk and chrooting and... /etc is read-only? Somehow the overlay for /etc does not work from within a chroot.

And I seem not to find any documentation on the exact layout of the btrfs subvolumes and the overlay. Is that simply not in the wiki (except the short summary in https://en.opensuse.org/Portal:MicroOS/Design#Filesystem_Layout)?

(As the machine has a 32 character root password that ansible created for me, I thought using a rescue stick would be easier...

Sorry for the long reply, been a busy week. My first thought before trying to answer this question is - 'Why?' Typically speaking I would say that any MicroOS/Kubic node that has issues should just be replaced and with combustion/ignition it should be easy to reproducibly deploy fresh nodes for your clusters. I'm not sure what you mean by the layout of the btrfs subvolumes..it is in the wiki Just like https://en.opensuse.org/Portal:MicroOS/Design#Filesystem_Layout says, it is only documenting the deviations from our defaults which are on https://en.opensuse.org/SDB:BTRFS#Default_Subvolumes The overlay itself is more complex, as its meant to be solely managed by transactional-update. But put simply, /etc maps to /var/lib/overlay/$SNAPSHOTNUM/etc where $SNAPSNOTNUM is the snapshot number currently in use. This is also documented in the fstab Does this help? Regards, -- Richard Brown Linux Distribution Engineer - Future Technology Team    Phone +4991174053-361 SUSE Software Solutions Germany GmbH,  Maxfeldstr. 5,  D-90409 Nuernberg (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer