http://bugzilla.opensuse.org/show_bug.cgi?id=1205777
Bug ID: 1205777 Summary: Pine64: Intermittent boot errors with "../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x3786d40 from `hd1'." Classification: openSUSE Product: openSUSE Tumbleweed Version: Current Hardware: aarch64 OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Bootloader Assignee: screening-team-bugs@suse.de Reporter: opensuse_buildservice@ojkastl.de QA Contact: qa-bugs@suse.de Found By: --- Blocker: ---
Pine64 (Hardware version is 2016) openSUSE MicroOS (the one based on Tumbleweed)
Every once in a while the boot fails after grub2 loaded the kernel and the initrd:
``` Booting `openSUSE MicroOS'
Loading Linux 6.0.8-1-default ... Loading initial ramdisk ... error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x3786d40 from `hd1'.
Press any key to continue...
```
Rebooting solves the issue most of the times.
A normal boot looks like this:
``` Booting `openSUSE MicroOS' Loading Linux 6.0.8-1-default ... Loading initial ramdisk ... EFI stub: Booting Linux Kernel... EFI stub: EFI_RNG_PROTOCOL unavailable EFI stub: Using DTB from configuration table EFI stub: Exiting boot services... [ 0.000000][ T0] Booting Linux on physical CPU 0x0000000000 [0x410fd034] [ 0.000000][ T0] Linux version 6.0.8-1-default (geeko@buildhost) (gcc (SUSE Linux) 12.2.1 20221020 [revision 0aaef83351473e8f4eb774f8f999bbe87a4866d7], GNU ld (GNU Binutils; openSUSE) [...] ```
Kind Regards, Johannes
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c1
--- Comment #1 from Johannes Kastl opensuse_buildservice@ojkastl.de --- New day, new errors. I am not sure if grub is somehow confused by the usb stick I have plugged as an external storage device. I'll remove it and see if it helps.
``` scanning bus usb@1c1a000 for devices... 1 USB Device(s) found scanning bus usb@1c1a400 for devices... 1 USB Device(s) found scanning bus usb@1c1b000 for devices... 1 USB Device(s) found scanning bus usb@1c1b400 for devices... 1 USB Device(s) found scanning usb for storage devices... 0 Storage Device(s) found Hit any key to stop autoboot: 0 switch to partitions #0, OK mmc0 is current device ** Invalid partition 3 ** Couldn't find partition mmc 0:3 ** Invalid partition 4 ** Couldn't find partition mmc 0:4 Scanning mmc 0:2... ** fs_devread read error - block ** fs_devread read error - block ** fs_devread read error - block bad tree block 2340093952, bytenr mismatch, want=2340093952, have=0 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 No EFI system partition BootOrder not defined EFI boot manager: Cannot load any image ** No partition table - mmc 0 ** Couldn't find partition mmc 0:2 ** No partition table - mmc 0 ** Couldn't find partition mmc 0:1 =>
```
``` error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0xee2aff0 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0xee2af80 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x1 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x87f0 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x8780 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x800 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0xee2aff0 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0xee2af80 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x8800 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0xee2aff0 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0xee2af80 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x1 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x87f0 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x8780 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x800 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0xee2aff0 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0xee2af80 from `hd1'. error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x8800 from `hd1'. error: ../../grub-core/commands/search.c:296:no such device: e590f1c9-3c90-4d9e-a0df-ff6e17559cf8. Loading Linux 6.0.8-1-default ... error: ../../grub-core/disk/efi/efidisk.c:602:failure reading sector 0x8880 from `hd1'. Loading initial ramdisk ... error: ../../grub-core/loader/arm64/efi/linux.c:300:you need to load the kernel first.
Press any key to continue... ```
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c2
--- Comment #2 from Johannes Kastl opensuse_buildservice@ojkastl.de --- Next error:
Sometimes to boot drops into the u-boot prompt, as no storage devices seem to be found.
U-Boot 2022.10 (Oct 04 2022 - 00:00:00 +0000) Allwinner Technology
CPU: Allwinner A64 (SUN50I) Model: Pine64+ DRAM: 2 GiB Core: 68 devices, 20 uclasses, devicetree: separate WDT: Not starting watchdog@1c20ca0 MMC: mmc@1c0f000: 0 Loading Environment from FAT... OK In: serial Out: serial Err: serial Net: eth0: ethernet@1c30000 starting USB... Bus usb@1c1a000: USB EHCI 1.00 Bus usb@1c1a400: USB OHCI 1.0 Bus usb@1c1b000: USB EHCI 1.00 Bus usb@1c1b400: USB OHCI 1.0 scanning bus usb@1c1a000 for devices... 1 USB Device(s) found scanning bus usb@1c1a400 for devices... 1 USB Device(s) found scanning bus usb@1c1b000 for devices... 1 USB Device(s) found scanning bus usb@1c1b400 for devices... 1 USB Device(s) found scanning usb for storage devices... 0 Storage Device(s) found Hit any key to stop autoboot: 0 =>
Typing "reset" reboots the device and suddenly the microSDXC card is found again and the device boots....
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c3
--- Comment #3 from Michal Suchanek msuchanek@suse.com --- Right, there is some bug in the rk3399 mmc driver in u-boot.
https://marc.info/?l=u-boot&m=165684945826950&w=2
No solution is known.
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c4
--- Comment #4 from Johannes Kastl opensuse_buildservice@ojkastl.de --- Hi Michal,
meh :-(
That renders the device pretty useless for use with openSUSE MicroOS, which has frequent reboots due to following Tumbleweed.
Did you try removing the sdcard and booting off of a USB device? USB-Stick, USB SSD something like this?
Kind Regards, Johannes
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c5
--- Comment #5 from Michal Suchanek msuchanek@suse.com --- No I got sidetracked because I found a problem with i2c bus which happens even earlier in the boot sequence.
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777
Stefan Seyfried seife@novell.slipkontur.de changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |seife@novell.slipkontur.de
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c6
--- Comment #6 from Stefan Seyfried seife@novell.slipkontur.de --- Johannes' machine is an allwinner sun50i, not a rk3399, but the problem might be similar.
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c7
--- Comment #7 from Michal Suchanek msuchanek@suse.com --- Pine64 is the vendor, and while they produce ROCK64 based on rk3328 and ROCKPro64 based on rk3399 they also produce PINE A64 based on A64 and PINE H64 based on H6.
This would be the Allwinner based PINE A64 which has nothing to do with rk3399 then.
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c8
--- Comment #8 from Johannes Kastl opensuse_buildservice@ojkastl.de --- Sorry Michal for being to vague.
Yes, this is a Pine A64, A64-DB-2G Rev B, date on the board says 2016-03-21.
Kind Regards, Johannes
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777
Matthias Brugger mbrugger@suse.com changed:
What |Removed |Added ---------------------------------------------------------------------------- CC| |guillaume.gardet@arm.com
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c9
--- Comment #9 from Johannes Kastl opensuse_buildservice@ojkastl.de --- What I fail to understand this far:
The board does not have a SPI flash (as far as I found out, so it needs to read the u-boot code from the sdcard (that for whatever reason is called mmc).
So the steps are: - board resets - board reads from sdcard from some offset - board boots into u-boot - u-boot fails to detect the sdcard OR u-boot is able to start grub, but grub fails to find the root device ("error: ../../grub-core/commands/search.c:296:no such device: e590f1c9-3c90-4d9e-a0df-ff6e17559cf8.") OR grub uses the wrong device ("failure reading sector 0x3786d40 from `hd1'.", which seems to have disappeared now that I removed the USB stick
If u-boot is booted, at least the board itself seems to find the sdcard. But u-boot does not? This would point to a bug in u-boot?
Kind Regards, Johannes
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c10
--- Comment #10 from Michal Suchanek msuchanek@suse.com --- Indeed, the situation is very much similar on rk3399 - the u-boot driver is likely missing a step in initializing the mmc bus or card.
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c11
--- Comment #11 from Johannes Kastl opensuse_buildservice@ojkastl.de --- Hi Michal,
Indeed, the situation is very much similar on rk3399 - the u-boot driver is likely missing a step in initializing the mmc bus or card.
This means there are two problems: - u-boot failing to detect the sdcard it was booted from - grub2 being loaded but failing to detect the root device or boot device or load the kernel or anything like that.
The latter one is the one, that I cannot solve even if the u-boot watchdog were working. A grub2 hanging with "Press any key to continue" makes the device unreachable and useless.
Any idea how to tackle that?
Kind Regards, Johannes
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c12
--- Comment #12 from Stefan Seyfried seife@novell.slipkontur.de --- in grub2, it might be enough to just add
boot reboot
to the end of each menuentry. Probably easiest done by changing the end of the linux_entry() function of /etc/grub.d/10_linux
This will cause the boot to fall through, if the kernel can't be loaded and then reboot
if the kenrel can be loaded but the initrd can't, this will cause a kernel boot with panic "root fs not found", but just add "panic=1" to the kernel cmdline to reboot immediately in this case.
Your menu entry needs to look like this:
menuentry 'kernel xyz' --... { ... some ... stuff ... put in by ... grub2-mkconfig echo "loading linux-xyz..." linux /path/to/kernel kernelopts=opts panic=1 echo "loading initrd..." initrd /path/to/initrd boot reboot }
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c13
--- Comment #13 from Johannes Kastl opensuse_buildservice@ojkastl.de --- Thanks Seife!
I gave this a try and now I am in a boot loop:
``` Booting /efi\boot\bootaa64.efi Please press 't' to show the boot menu on this console error: ../../grub-core/video/video.c:761:no suitable video mode found. error: ../../grub-core/commands/boot.c:196:you need to load the kernel first. resetting ... ```
Not sure why suddenly it does no longer show the grub2 console but rather asks for pressing 't'. I need to investigate what I did wrong :-)
But at least it seems like the automatic reboot is working.
Unfortunately it errors out after some loops:
``` U-Boot SPL 2022.10 (Oct 04 2022 - 00:00:00 +0000) DRAM: 2048 MiB Trying to boot from MMC1 spl: mmc init failed with error: -110 SPL: failed to boot from all boot devices ### ERROR ### Please RESET the board ### ```
But that might be the other error, the one where u-boot does not boot..
Kind Regards, Johannes
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c14
--- Comment #14 from Johannes Kastl opensuse_buildservice@ojkastl.de --- Bootloop solved. My bad, missed a bracket...
I also rebuilt u-boot-pine64plus with enabled wdt (at least I hope).
https://build.opensuse.org/project/show/home:ojkastl_buildservice:u-boot_wit...
I installed it and rebooted, but I think I need to somehow write out the image for u-boot.
I'll keep you posted.
Johannes
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c15
--- Comment #15 from Matthias Brugger mbrugger@suse.com --- (In reply to Johannes Kastl from comment #9)
What I fail to understand this far:
The board does not have a SPI flash (as far as I found out, so it needs to read the u-boot code from the sdcard (that for whatever reason is called mmc).
So the steps are:
- board resets
- board reads from sdcard from some offset
- board boots into u-boot
- u-boot fails to detect the sdcard OR u-boot is able to start grub, but grub fails to find the root device
("error: ../../grub-core/commands/search.c:296:no such device: e590f1c9-3c90-4d9e-a0df-ff6e17559cf8.") OR grub uses the wrong device ("failure reading sector 0x3786d40 from `hd1'.", which seems to have disappeared now that I removed the USB stick
If u-boot is booted, at least the board itself seems to find the sdcard. But u-boot does not? This would point to a bug in u-boot?
Yes, Grub usese the EFI implementation of U-Boot to access the HW. And if U-Boot access to the SD card is buggy, so will be the access from Grub.
http://bugzilla.opensuse.org/show_bug.cgi?id=1205777 http://bugzilla.opensuse.org/show_bug.cgi?id=1205777#c16
--- Comment #16 from Johannes Kastl opensuse_buildservice@ojkastl.de --- (In reply to Matthias Brugger from comment #15)
Yes, Grub usese the EFI implementation of U-Boot to access the HW. And if U-Boot access to the SD card is buggy, so will be the access from Grub.
Aaaah, that explains it.
In the meantime Seife's grub workaround allowed the board to build multiple times, but just half an hour ago it dropped into the grub shell.
``` grub > ```
And there it sat waiting for some command...
Johannes