Hi,

-----Original Message----- From: Christian Gross Sent: 01 November 2021 01:35 To: arm@lists.opensuse.org Subject: [Fwd: Re: DE0-Nano-SoC boot fails on 15.3]

I think I found the problem. The freeze is caused by the CONFIG_FORTIFY_SOURCE kernel configuration item.

--- /boot/config-5.3.18-59.25-default 2021-09-23 14:34:53.000000000 +0000 +++ .config 2021-07-15 12:07:05.608153308 +0000 @@ -10294,7 +10294,7 @@ CONFIG_LSM_MMAP_MIN_ADDR=0 CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR=y # CONFIG_HARDENED_USERCOPY is not set -CONFIG_FORTIFY_SOURCE=y +# CONFIG_FORTIFY_SOURCE is not set # CONFIG_STATIC_USERMODEHELPER is not set # CONFIG_HIDDEN_AREA is not set CONFIG_SECURITY_SELINUX=y

At least the kernel is booting after changing it.

And it was only set to "y" for the 5.3.18-59.25-default kernel used in 15.3

config-5.14.11-2-default:# CONFIG_FORTIFY_SOURCE is not set config-5.3.18- 59.25-default:CONFIG_FORTIFY_SOURCE=y config-5.3.18-lp152.92-default:# CONFIG_FORTIFY_SOURCE is not set config- 5.3.18-lp152.95-default:# CONFIG_FORTIFY_SOURCE is not set config-5.3.18- lp152.95-mybuild:# CONFIG_FORTIFY_SOURCE is not set

Thanks for tracking this down! Could you report this kernel problem on https://bugzilla.opensuse.org please? Cheers, Guillaume

Regards,

Christian

---------------------------- Original Message ---------------------------- Subject: Re: DE0-Nano-SoC boot fails on 15.3 From: "Christian Gross" Date: Fri, October 29, 2021 4:27 pm To: arm@lists.opensuse.org --------------------------------------------------------------------------

The earlycon parameter helped

Loading Linux 5.3.18-59.25-default ... Loading initial ramdisk ... EFI stub: Entering in SVC mode with MMU enabled EFI stub: Booting Linux Kernel... EFI stub: Using DTB from configuration table EFI stub: Exiting boot services and installing virtual address map... [ 0.008983] detected buffer overflow in memcpy [ 0.013512] ------------[ cut here ]------------ [ 0.018109] kernel BUG at ../lib/string.c:1090! [ 0.022618] Internal error: Oops - BUG: 0 [#1] SMP ARM [ 0.027730] Modules linked in: [ 0.030771] Supported: No, Unreleased kernel [ 0.035025] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.3.18-59.25-default #1 openSUSE Leap 15.2 (unreleased) [ 0.044890] Hardware name: Altera SOCFPGA [ 0.048891] PC is at fortify_panic+0x1c/0x20 [ 0.053145] LR is at vprintk_emit+0x118/0x2a0 [ 0.057480] pc : [<c0da9168>] lr : [<c03e668c>] psr: 60000053 [ 0.063716] sp : ee949de0 ip : ee949d20 fp : ee949dec [ 0.068914] r10: c16321b4 r9 : c032ec34 r8 : c12b9118 [ 0.074113] r7 : 00000000 r6 : c1765658 r5 : 00000020 r4 : c176565c [ 0.080608] r3 : 48d9555b r2 : 48d9555b r1 : 00000000 r0 : 00000022 [ 0.087105] Flags: nZCv IRQs on FIQs off Mode SVC_32 ISA ARM Segment user [ 0.094291] Control: 10c5387d Table: 0020404a DAC: 00000055 [ 0.100010] Process swapper/0 (pid: 1, stack limit = 0x(ptrval)) [ 0.105987] Stack: (0xee949de0 to 0xee94a000) [ 0.110326] 9de0: ee949e0c ee949df0 c035fd20 c0da9158 c1763658 00000001 c1763668 ee945000 [ 0.118466] 9e00: ee949e34 ee949e10 c031e578 c035fc24 ee945000 00000001 2e14e000 c036e4cc [ 0.126605] 9e20: 00000001 ef6d42e8 ee949e54 ee949e38 c036e4fc c031e4ac c15862e8 00000056 [ 0.134745] 9e40: 2e14e000 c036e4cc ee949eb4 ee949e58 c036daa8 c036e4d8 c0dae498 c11b1208 [ 0.142884] 9e60: c11b1200 c1631f70 c1767140 c1743138 ef6d42e8 c1605fac ffffe000 00000003 [ 0.151024] 9e80: c047b824 00000001 2e14e000 ef6d42e8 c15862e8 00000001 00000056 c1605fb0 [ 0.159164] 9ea0: 2e14e000 00000000 ee949ef4 ee949eb8 c036f998 c036d9fc 00000000 c03e6820 [ 0.167303] 9ec0: 00000000 00000000 00000000 00000001 000000d4 c1640ed4 c16060e4 c1605fb0 [ 0.175442] 9ee0: 00000000 00000000 ee949f0c ee949ef8 c036fac8 c036f878 00000001 c1605fac [ 0.183582] 9f00: ee949f1c ee949f10 c036fb08 c036fa48 ee949f4c ee949f20 c142678c c036faf8 [ 0.191722] 9f20: 00000000 c1429b4c ee949f4c c15840e4 c15840e4 00000001 00000000 00000000 [ 0.199861] 9f40: ee949f94 ee949f50 c14013bc c14266ec c048c06c c031b8cc ee948000 00000000 [ 0.208000] 9f60: 00000000 00000000 00000000 00000000 c0db1ea8 00000000 00000000 00000000 [ 0.216139] 9f80: 00000000 00000000 ee949fac ee949f98 c0db1ec0 c14011bc 00000000 c0db1ea8 [ 0.224279] 9fa0: 00000000 ee949fb0 c03010b0 c0db1eb4 00000000 00000000 00000000 00000000 [ 0.232417] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 0.240556] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000 [ 0.248691] Backtrace: [ 0.251138] [<c0da914c>] (fortify_panic) from [<c035fd20>] (socfpga_a10_boot_secondary+0x0/0x10c) [ 0.259973] [<c035fc18>] (socfpga_boot_secondary) from [<c031e578>] (__cpu_up+0xd8/0x19c) [ 0.268113] r7:ee945000 r6:c1763668 r5:00000001 r4:c1763658 [ 0.273754] [<c031e4a0>] (__cpu_up) from [<c036e4fc>] (bringup_cpu+0x30/0x104) [ 0.280944] r9:ef6d42e8 r8:00000001 r7:c036e4cc r6:2e14e000 r5:00000001 r4:ee945000 [ 0.288655] [<c036e4cc>] (bringup_cpu) from [<c036daa8>] (cpuhp_invoke_callback+0xb8/0x7bc) [ 0.296967] r7:c036e4cc r6:2e14e000 r5:00000056 r4:c15862e8 [ 0.302603] [<c036d9f0>] (cpuhp_invoke_callback) from [<c036f998>] (_cpu_up+0x12c/0x1d0) [ 0.310657] r10:00000000 r9:2e14e000 r8:c1605fb0 r7:00000056 r6:00000001 r5:c15862e8 [ 0.318447] r4:ef6d42e8 [ 0.320971] [<c036f86c>] (_cpu_up) from [<c036fac8>] (do_cpu_up+0x8c/0xb0) [ 0.327815] r10:00000000 r9:00000000 r8:c1605fb0 r7:c16060e4 r6:c1640ed4 r5:000000d4 [ 0.335605] r4:00000001 [ 0.338128] [<c036fa3c>] (do_cpu_up) from [<c036fb08>] (cpu_up+0x1c/0x24) [ 0.344882] r5:c1605fac r4:00000001 [ 0.348446] [<c036faec>] (cpu_up) from [<c142678c>] (smp_init+0xac/0xf8) [ 0.355123] [<c14266e0>] (smp_init) from [<c14013bc>] (kernel_init_freeable+0x20c/0x40c) [ 0.363177] r8:00000000 r7:00000000 r6:00000001 r5:c15840e4 r4:c15840e4 [ 0.369853] [<c14011b0>] (kernel_init_freeable) from [<c0db1ec0>] (kernel_init+0x18/0x128) [ 0.378079] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0db1ea8 [ 0.385869] r4:00000000 [ 0.388394] [<c0db1ea8>] (kernel_init) from [<c03010b0>] (ret_from_fork+0x14/0x24) [ 0.395926] Exception stack(0xee949fb0 to 0xee949ff8) [ 0.400953] 9fa0: 00000000 00000000 00000000 00000000 [ 0.409093] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 0.417231] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 [ 0.423812] r5:c0db1ea8 r4:00000000 [ 0.427372] Code: e1a01000 e3070100 e34c012b ebd8f7cc (e7f001f2) [ 0.433447] ---[ end trace 4663108c1c9c0434 ]--- [ 0.438047] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b [ 0.445674] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

Regards,

Christian

On Fri, October 29, 2021 10:56 am, Matthias Brugger wrote:

...

...

On 28/10/2021 20:07, Christian Gross wrote:

...

Hi all,

I tested the U-Boot fix for several versions and it works perfect for all of them.

Version | Build | U.Boot | Kernel | Boot Leap 15.1 | 2019.05.17-Snapshot1.105 | 2019.01 | 4.12.14 | OK Leap 15.2 | 2020.07.08-Build1.276 | 2020.01 | 5.3.18 | OK Leap 15.3 | 2021.05.31-Build8.7 | 2021.01 | 5.3.18 | freezing Tumbleweed | 2021.09.21-Snapshot20211008 | 2021.10 | 5.14.9 | OK Tumbleweed | 2021.10.20-Snapshot20211023 | 2021.10 | 5.14.11-2 | OK

But it looks like there is an additional problem with Leap 15.3 which freezes at boot.

Booting `openSUSE Leap 15.3'

Loading Linux 5.3.18-59.25-default ... Loading initial ramdisk ... EFI stub: Entering in SVC mode with MMU enabled EFI stub: Booting Linux Kernel... EFI stub: Using DTB from configuration table EFI stub: Exiting boot services and installing virtual address map... Deasserting all peripheral resets [freeze]

I also upgraded a fixed Leap 15.2 to 15.3 and the same thing happens (Only without the "Deasserting all peripheral resets" message) for kernel 5.3.18-59.25-default. The older kernels 5.3.18-lp152.95 and 5.3.18-lp152.92 are booting without any problem.

Any ideas how to debug this further?

You can try to get more output by editing the kernel boot parameter in grub. Try 'earlycon'. If that does not help, try 'earlycon efi=debug'. Beware that you might need to pass memory region of your serial console to the earlycon parameter. Have a look at: https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.h tml

Regards, Matthias

IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

FYI it is now discussed in https://bugzilla.suse.com/show_bug.cgi?id=1192473 Regards, Matthias On 01/11/2021 01:35, Christian Gross wrote:

I think I found the problem. The freeze is caused by the CONFIG_FORTIFY_SOURCE kernel configuration item.

--- /boot/config-5.3.18-59.25-default 2021-09-23 14:34:53.000000000 +0000 +++ .config 2021-07-15 12:07:05.608153308 +0000 @@ -10294,7 +10294,7 @@ CONFIG_LSM_MMAP_MIN_ADDR=0 CONFIG_HAVE_HARDENED_USERCOPY_ALLOCATOR=y # CONFIG_HARDENED_USERCOPY is not set -CONFIG_FORTIFY_SOURCE=y +# CONFIG_FORTIFY_SOURCE is not set # CONFIG_STATIC_USERMODEHELPER is not set # CONFIG_HIDDEN_AREA is not set CONFIG_SECURITY_SELINUX=y

At least the kernel is booting after changing it.

And it was only set to "y" for the 5.3.18-59.25-default kernel used in 15.3

config-5.14.11-2-default:# CONFIG_FORTIFY_SOURCE is not set config-5.3.18-59.25-default:CONFIG_FORTIFY_SOURCE=y config-5.3.18-lp152.92-default:# CONFIG_FORTIFY_SOURCE is not set config-5.3.18-lp152.95-default:# CONFIG_FORTIFY_SOURCE is not set config-5.3.18-lp152.95-mybuild:# CONFIG_FORTIFY_SOURCE is not set

Regards,

Christian

---------------------------- Original Message ---------------------------- Subject: Re: DE0-Nano-SoC boot fails on 15.3 From: "Christian Gross" Date: Fri, October 29, 2021 4:27 pm To: arm@lists.opensuse.org --------------------------------------------------------------------------

The earlycon parameter helped

Loading Linux 5.3.18-59.25-default ... Loading initial ramdisk ... EFI stub: Entering in SVC mode with MMU enabled EFI stub: Booting Linux Kernel... EFI stub: Using DTB from configuration table EFI stub: Exiting boot services and installing virtual address map... [ 0.008983] detected buffer overflow in memcpy [ 0.013512] ------------[ cut here ]------------ [ 0.018109] kernel BUG at ../lib/string.c:1090! [ 0.022618] Internal error: Oops - BUG: 0 [#1] SMP ARM [ 0.027730] Modules linked in: [ 0.030771] Supported: No, Unreleased kernel [ 0.035025] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.3.18-59.25-default #1 openSUSE Leap 15.2 (unreleased) [ 0.044890] Hardware name: Altera SOCFPGA [ 0.048891] PC is at fortify_panic+0x1c/0x20 [ 0.053145] LR is at vprintk_emit+0x118/0x2a0 [ 0.057480] pc : [<c0da9168>] lr : [<c03e668c>] psr: 60000053 [ 0.063716] sp : ee949de0 ip : ee949d20 fp : ee949dec [ 0.068914] r10: c16321b4 r9 : c032ec34 r8 : c12b9118 [ 0.074113] r7 : 00000000 r6 : c1765658 r5 : 00000020 r4 : c176565c [ 0.080608] r3 : 48d9555b r2 : 48d9555b r1 : 00000000 r0 : 00000022 [ 0.087105] Flags: nZCv IRQs on FIQs off Mode SVC_32 ISA ARM Segment user [ 0.094291] Control: 10c5387d Table: 0020404a DAC: 00000055 [ 0.100010] Process swapper/0 (pid: 1, stack limit = 0x(ptrval)) [ 0.105987] Stack: (0xee949de0 to 0xee94a000) [ 0.110326] 9de0: ee949e0c ee949df0 c035fd20 c0da9158 c1763658 00000001 c1763668 ee945000 [ 0.118466] 9e00: ee949e34 ee949e10 c031e578 c035fc24 ee945000 00000001 2e14e000 c036e4cc [ 0.126605] 9e20: 00000001 ef6d42e8 ee949e54 ee949e38 c036e4fc c031e4ac c15862e8 00000056 [ 0.134745] 9e40: 2e14e000 c036e4cc ee949eb4 ee949e58 c036daa8 c036e4d8 c0dae498 c11b1208 [ 0.142884] 9e60: c11b1200 c1631f70 c1767140 c1743138 ef6d42e8 c1605fac ffffe000 00000003 [ 0.151024] 9e80: c047b824 00000001 2e14e000 ef6d42e8 c15862e8 00000001 00000056 c1605fb0 [ 0.159164] 9ea0: 2e14e000 00000000 ee949ef4 ee949eb8 c036f998 c036d9fc 00000000 c03e6820 [ 0.167303] 9ec0: 00000000 00000000 00000000 00000001 000000d4 c1640ed4 c16060e4 c1605fb0 [ 0.175442] 9ee0: 00000000 00000000 ee949f0c ee949ef8 c036fac8 c036f878 00000001 c1605fac [ 0.183582] 9f00: ee949f1c ee949f10 c036fb08 c036fa48 ee949f4c ee949f20 c142678c c036faf8 [ 0.191722] 9f20: 00000000 c1429b4c ee949f4c c15840e4 c15840e4 00000001 00000000 00000000 [ 0.199861] 9f40: ee949f94 ee949f50 c14013bc c14266ec c048c06c c031b8cc ee948000 00000000 [ 0.208000] 9f60: 00000000 00000000 00000000 00000000 c0db1ea8 00000000 00000000 00000000 [ 0.216139] 9f80: 00000000 00000000 ee949fac ee949f98 c0db1ec0 c14011bc 00000000 c0db1ea8 [ 0.224279] 9fa0: 00000000 ee949fb0 c03010b0 c0db1eb4 00000000 00000000 00000000 00000000 [ 0.232417] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 0.240556] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 00000000 00000000 [ 0.248691] Backtrace: [ 0.251138] [<c0da914c>] (fortify_panic) from [<c035fd20>] (socfpga_a10_boot_secondary+0x0/0x10c) [ 0.259973] [<c035fc18>] (socfpga_boot_secondary) from [<c031e578>] (__cpu_up+0xd8/0x19c) [ 0.268113] r7:ee945000 r6:c1763668 r5:00000001 r4:c1763658 [ 0.273754] [<c031e4a0>] (__cpu_up) from [<c036e4fc>] (bringup_cpu+0x30/0x104) [ 0.280944] r9:ef6d42e8 r8:00000001 r7:c036e4cc r6:2e14e000 r5:00000001 r4:ee945000 [ 0.288655] [<c036e4cc>] (bringup_cpu) from [<c036daa8>] (cpuhp_invoke_callback+0xb8/0x7bc) [ 0.296967] r7:c036e4cc r6:2e14e000 r5:00000056 r4:c15862e8 [ 0.302603] [<c036d9f0>] (cpuhp_invoke_callback) from [<c036f998>] (_cpu_up+0x12c/0x1d0) [ 0.310657] r10:00000000 r9:2e14e000 r8:c1605fb0 r7:00000056 r6:00000001 r5:c15862e8 [ 0.318447] r4:ef6d42e8 [ 0.320971] [<c036f86c>] (_cpu_up) from [<c036fac8>] (do_cpu_up+0x8c/0xb0) [ 0.327815] r10:00000000 r9:00000000 r8:c1605fb0 r7:c16060e4 r6:c1640ed4 r5:000000d4 [ 0.335605] r4:00000001 [ 0.338128] [<c036fa3c>] (do_cpu_up) from [<c036fb08>] (cpu_up+0x1c/0x24) [ 0.344882] r5:c1605fac r4:00000001 [ 0.348446] [<c036faec>] (cpu_up) from [<c142678c>] (smp_init+0xac/0xf8) [ 0.355123] [<c14266e0>] (smp_init) from [<c14013bc>] (kernel_init_freeable+0x20c/0x40c) [ 0.363177] r8:00000000 r7:00000000 r6:00000001 r5:c15840e4 r4:c15840e4 [ 0.369853] [<c14011b0>] (kernel_init_freeable) from [<c0db1ec0>] (kernel_init+0x18/0x128) [ 0.378079] r10:00000000 r9:00000000 r8:00000000 r7:00000000 r6:00000000 r5:c0db1ea8 [ 0.385869] r4:00000000 [ 0.388394] [<c0db1ea8>] (kernel_init) from [<c03010b0>] (ret_from_fork+0x14/0x24) [ 0.395926] Exception stack(0xee949fb0 to 0xee949ff8) [ 0.400953] 9fa0: 00000000 00000000 00000000 00000000 [ 0.409093] 9fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 [ 0.417231] 9fe0: 00000000 00000000 00000000 00000000 00000013 00000000 [ 0.423812] r5:c0db1ea8 r4:00000000 [ 0.427372] Code: e1a01000 e3070100 e34c012b ebd8f7cc (e7f001f2) [ 0.433447] ---[ end trace 4663108c1c9c0434 ]--- [ 0.438047] Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b [ 0.445674] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x0000000b ]---

Regards,

Christian

On Fri, October 29, 2021 10:56 am, Matthias Brugger wrote:

...

...

On 28/10/2021 20:07, Christian Gross wrote:

...

Hi all,

I tested the U-Boot fix for several versions and it works perfect for all of them.

Version | Build | U.Boot | Kernel | Boot Leap 15.1 | 2019.05.17-Snapshot1.105 | 2019.01 | 4.12.14 | OK Leap 15.2 | 2020.07.08-Build1.276 | 2020.01 | 5.3.18 | OK Leap 15.3 | 2021.05.31-Build8.7 | 2021.01 | 5.3.18 | freezing Tumbleweed | 2021.09.21-Snapshot20211008 | 2021.10 | 5.14.9 | OK Tumbleweed | 2021.10.20-Snapshot20211023 | 2021.10 | 5.14.11-2 | OK

But it looks like there is an additional problem with Leap 15.3 which freezes at boot.

Booting `openSUSE Leap 15.3'

Loading Linux 5.3.18-59.25-default ... Loading initial ramdisk ... EFI stub: Entering in SVC mode with MMU enabled EFI stub: Booting Linux Kernel... EFI stub: Using DTB from configuration table EFI stub: Exiting boot services and installing virtual address map... Deasserting all peripheral resets [freeze]

I also upgraded a fixed Leap 15.2 to 15.3 and the same thing happens (Only without the "Deasserting all peripheral resets" message) for kernel 5.3.18-59.25-default. The older kernels 5.3.18-lp152.95 and 5.3.18-lp152.92 are booting without any problem.

Any ideas how to debug this further?

You can try to get more output by editing the kernel boot parameter in grub. Try 'earlycon'. If that does not help, try 'earlycon efi=debug'. Beware that you might need to pass memory region of your serial console to the earlycon parameter. Have a look at: https://www.kernel.org/doc/html/latest/admin-guide/kernel-parameters.html

Regards, Matthias