I am wondering how to proceed with this.

Sorry, no idea how to repair btrfs.

I would try to check and repair from a rescue system.

I suggest you download and raw copy this file to an usb stick:

<http://download.opensuse.org/distribution/leap/15.5/live/openSUSE-Leap-15.5-Rescue-CD-x86_64-Build13.15-Media.iso>

Then boot it.

I booted into a live session and opened the luks encrypted volume. then

btrfsck --check /dev/mapper/system-root <- Failed with core dump

btrfsck --check --repair /dev/mapper/system-root <- Failed with core dump

Do you have a recommended place to report bugs for btrfs?