I'm running k3d (4.4.6) on Tumbleweed's 5.13.8-1 kernel to install Kubernetes
in Docker 20.10.6_ce-1.2.
After installing Kubernetes with `k3d cluster create`, I try to install Epinio
for testing.
After some time, the kernel crashes and the machine reboots when the timeout in
`/proc/sys/kernel/panic` is reached.
Sometimes the crash occurs during the first installation of Epinio and its
components, sometimes the crash occurs later, during testing. Even an
incomplete installation will eventually crash the kernel.
Other team members report a similar problem with the latest kernel.
Going back to kernel 5.3.12-1 is the only workaround.
I tried to create a kdump for the crash. Not sure if I should attach that, as
it's 184mb?
Here is the backtrace from dmesg:
[ 1433.295401] general protection fault, probably for non-canonical address
0xb00fcd7a229657cd: 0000 [#1] SMP NOPTI
[ 1433.295414] CPU: 0 PID: 26175 Comm: runc Kdump: loaded Tainted: G W
OE 5.13.8-1-debug #1 openSUSE Tumbleweed
[ 1433.295422] Hardware name: Dell Inc. Precision 5820 Tower X-Series/0X75JG,
BIOS 2.4.0 07/06/2020
[ 1433.295426] RIP: 0010:kmem_cache_alloc_node_trace+0x79/0x2d0
[ 1433.295438] Code: 89 c6 48 85 c0 0f 84 b4 00 00 00 0f 1f 44 00 00 48 c7 44
24 10 00 00 00 00 e9 2a 01 00 00 0f 1f 44 00 00 41 8b 56 28 48 01 c2 <4c> 8b 02
48 89 d1 4d 33 86 b8 00 00 00 48 0f c9 49 31 c8 48 8d 4b
[ 1433.295444] RSP: 0018:ffffab13052e3c88 EFLAGS: 00010282
[ 1433.295450] RAX: b00fcd7a229656cd RBX: 0000000000010a86 RCX:
0000000000000400
[ 1433.295454] RDX: b00fcd7a229657cd RSI: 0000000000000dc0 RDI:
ffff8a7f00042a00
[ 1433.295458] RBP: 0000000000000dc0 R08: ffff8a8e52e34140 R09:
ffff8a7f0809f800
[ 1433.295462] R10: 0000000000000011 R11: 0000000001320122 R12:
ffff8a7f00042a00
[ 1433.295466] R13: 0000000000000000 R14: ffff8a7f00042a00 R15:
ffffffffa34e6905
[ 1433.295470] FS: 00007f12e5e31f20(0000) GS:ffff8a8e52e00000(0000)
knlGS:0000000000000000
[ 1433.295475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1433.295479] CR2: 00007f4f471adf08 CR3: 000000030dcfe006 CR4:
00000000003706f0
[ 1433.295483] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[ 1433.295486] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7:
0000000000000400
[ 1433.295490] Call Trace:
[ 1433.295498] alloc_fair_sched_group+0xf5/0x1d0
[ 1433.295511] sched_create_group+0x2f/0x80
[ 1433.295522] cpu_cgroup_css_alloc+0xf/0x30
[ 1433.295529] cgroup_apply_control_enable+0x14e/0x330
[ 1433.295541] cgroup_mkdir+0x21f/0x470
[ 1433.295549] kernfs_iop_mkdir+0x54/0x80
[ 1433.295557] vfs_mkdir+0x12c/0x1e0
[ 1433.295565] do_mkdirat+0x127/0x150
[ 1433.295573] do_syscall_64+0x5e/0xb0
[ 1433.295585] ? syscall_exit_to_user_mode+0x18/0x40
[ 1433.295591] ? do_syscall_64+0x6e/0xb0
[ 1433.295599] ? syscall_exit_to_user_mode+0x18/0x40
[ 1433.295603] ? do_syscall_64+0x6e/0xb0
[ 1433.295610] ? syscall_exit_to_user_mode+0x18/0x40
[ 1433.295615] ? do_syscall_64+0x6e/0xb0
[ 1433.295622] ? do_syscall_64+0x6e/0xb0
[ 1433.295628] entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 1433.295637] RIP: 0033:0x4becdb
[ 1433.295642] Code: fa ff eb bd e8 26 be fa ff e9 61 ff ff ff cc e8 3b 8d fa
ff 48 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 48 8b 44 24 08 0f 05 <48> 3d 01
f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
[ 1433.295647] RSP: 002b:000000c0000e8670 EFLAGS: 00000202 ORIG_RAX:
0000000000000102
[ 1433.295652] RAX: ffffffffffffffda RBX: 000000c00002e000 RCX:
00000000004becdb
[ 1433.295656] RDX: 00000000000001ed RSI: 000000c0001401b0 RDI:
ffffffffffffff9c
[ 1433.295659] RBP: 000000c0000e86c8 R08: 0000000000000001 R09:
0000000000000001
[ 1433.295662] R10: 000000c0001401b0 R11: 0000000000000202 R12:
ffffffffffffffff
[ 1433.295665] R13: 0000000000000004 R14: 0000000000000003 R15:
0000000000000038
[ 1433.295674] Modules linked in: xt_owner xt_REDIRECT ipt_REJECT
vboxnetadp(OE) vboxnetflt(OE) vboxdrv(OE) xt_statistic xt_mark vxlan
ip6_udp_tunnel udp_tunnel xt_multiport xt_comment overlay xt_nat xt_tcpudp veth
af_packet xt_conntrack xt_MASQUERADE nf_conntrack_netlink xfrm_user xfrm_algo
xt_addrtype nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib br_netfilter bridge
stp llc nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
nft_chain_nat nf_tables ebtable_nat ebtable_broute ip6table_nat ip6table_mangle
ip6table_raw ip6table_security iptable_nat nf_nat dm_thin_pool nf_conntrack
dm_persistent_data dm_bio_prison dm_bufio nf_defrag_ipv6 nf_defrag_ipv4
iptable_mangle iptable_raw iptable_security ip_set nfnetlink ebtable_filter
ebtables ip6table_filter rfkill ip6_tables iptable_filter ip_tables x_tables
bpfilter intel_rapl_msr intel_rapl_common isst_if_common dmi_sysfs squashfs
pktcdvd nfit libnvdimm snd_hda_codec_realtek snd_hda_codec_generic iTCO_wdt
[ 1433.295771] intel_pmc_bxt x86_pkg_temp_thermal intel_powerclamp
ledtrig_audio snd_hda_codec_hdmi coretemp iTCO_vendor_support dell_smm_hwmon
kvm_intel snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi snd_hda_codec
dell_wmi snd_hda_core snd_hwdep dell_smbios kvm dcdbas snd_pcm mei_me snd_timer
sparse_keymap xfs irqbypass video e1000e snd dell_wmi_descriptor
intel_wmi_thunderbolt wmi_bmof efi_pstore mei i2c_i801 ioatdma soundcore
i2c_smbus dca nls_iso8859_1 nls_cp437 tiny_power_button vfat fat libcrc32c
acpi_tad button loop fuse configfs uas usb_storage amdgpu crct10dif_pclmul
crc32_pclmul crc32c_intel drm_ttm_helper ttm ghash_clmulni_intel iommu_v2
gpu_sched i2c_algo_bit drm_kms_helper syscopyarea sysfillrect sysimgblt
xhci_pci fb_sys_fops xhci_pci_renesas cec xhci_hcd rc_core drm nvme usbcore
nvme_core aesni_intel sr_mod crypto_simd cryptd cdrom serio_raw wmi vmd sg
dm_multipath dm_mod scsi_dh_rdac scsi_dh_emc scsi_dh_alua msr efivarfs