-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hi,
My home server (Leap 42.3) crashed during a yast online update, and I
can't boot it up. I need help diagnosing what happened, but more
importantly, how to recover it.
The machine is UEFI and _had_ secure boot enabled.
I simply get black screen after the bios logo, no grub display. Black, no
response to the keyboard that I can see.
On my desktop machine I downloaded:
"openSUSE-Tumbleweed-Rescue-CD-x86_64-Snapshot20180318-Media.iso" and
burned to an usb stick. I managed to boot it up after disabling secure
boot. I can't convince YaST to enable the Spanish keyboard. Has no joe nor
mc, only "vi" (vi is a pain with US keyboard layout on a Spanish
keyboard). Network manager does not respond to clicks (responds once in a
time, then goes silent for a minute or two). I had to tell yast
to use wicked instead. So I got network, killed the firewall, started ssh,
created a root password (1234, if you ask me), and got in via ssh with
decent keyboard.
The last thing I see on the log (thankfully I had syslog on the machine, I
would not know how to read journal of dead system) is os-prober running:
/var/log/messages:
...
<1.5> 2018-03-20T22:27:57.938415+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/10freedos
<1.5> 2018-03-20T22:27:57.954019+01:00 Isengard 10freedos - - - debug: /dev/sdc1 is not a FAT partition: exiting
<1.5> 2018-03-20T22:27:57.960368+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/10qnx
<1.5> 2018-03-20T22:27:57.975764+01:00 Isengard 10qnx - - - debug: /dev/sdc1 is not a QNX4 partition: exiting
<1.5> 2018-03-20T22:27:57.981400+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/20macosx
<1.5> 2018-03-20T22:27:57.994089+01:00 Isengard macosx-prober - - - debug: /dev/sdc1 is not an HFS+ partition: exiting
<1.5> 2018-03-20T22:27:58.003571+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/20microsoft
<1.5> 2018-03-20T22:27:58.020164+01:00 Isengard 20microsoft - - - debug: Skipping legacy bootloaders on UEFI system
<1.5> 2018-03-20T22:27:58.025907+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/30utility
<1.5> 2018-03-20T22:27:58.040597+01:00 Isengard 30utility - - - debug: /dev/sdc1 is not a FAT partition: exiting
<1.5> 2018-03-20T22:27:58.046488+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/40lsb
<1.5> 2018-03-20T22:27:58.058352+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/70hurd
<1.5> 2018-03-20T22:27:58.069980+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/80minix
<1.5> 2018-03-20T22:27:58.084577+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/83haiku
<1.5> 2018-03-20T22:27:58.099056+01:00 Isengard 83haiku - - - debug: /dev/sdc1 is not a BeFS partition: exiting
<1.5> 2018-03-20T22:27:58.104596+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/90linux-distro
<1.5> 2018-03-20T22:27:58.123799+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/90solaris
<1.5> 2018-03-20T22:27:58.135526+01:00 Isengard 50mounted-tests - - - debug: running subtest /usr/lib/os-probes/mounted/efi
<1.5> 2018-03-20T22:27:58.177568+01:00 Isengard os-prober - - - debug: running /usr/lib/os-probes/50mounted-tests on /dev/sdc2
<1.5> 2018-03-20T22:27:58.200879+01:00 Isengard 50mounted-tests - - - debug: /dev/sdc2 is a LUKS partition; skipping
<1.5> 2018-03-20T22:27:58.205838+01:00 Isengard os-prober - - - debug: os detected by /usr/lib/os-probes/50mounted-tests
<3.5> 2018-03-20T22:28:19.866697+01:00 Isengard dbus 1148 - - [system] Reloaded configuration
<3.6> 2018-03-20T22:28:19.877465+01:00 Isengard dbus-daemon 1148 - - Unknown username "systemd-network" in message bus configuration file
<3.5> 2018-03-20T22:28:19.889628+01:00 Isengard dbus 1148 - - [system] Reloaded configuration
<3.6> 2018-03-20T22:28:20.385417+01:00 Isengard dbus-daemon 1148 - - Unknown username "systemd-network" in message bus configuration file
<3.5> 2018-03-20T22:28:20.395558+01:00 Isengard dbus 1148 - - [system] Reloaded configuration
<3.6> 2018-03-20T22:28:20.401848+01:00 Isengard dbus-daemon 1148 - - Unknown username "systemd-network" in message bus configuration file
<3.5> 2018-03-20T22:28:20.414031+01:00 Isengard dbus 1148 - - [system] Reloaded configuration
<3.6> 2018-03-20T22:28:20.579689+01:00 Isengard systemd-sysusers 26699 - - Creating group systemd-network with gid 470.
<3.6> 2018-03-20T22:28:20.580433+01:00 Isengard systemd-sysusers 26699 - - Creating user systemd-network (systemd Network Management) with uid 470 and gid 470.
<3.5> 2018-03-20T22:28:20.587314+01:00 Isengard nscd - - - 1129 monitored file `/etc/group` was moved into place, adding watch
<3.5> 2018-03-20T22:28:20.589487+01:00 Isengard nscd - - - 1129 monitored file `/etc/passwd` was moved into place, adding watch
<3.5> 2018-03-20T22:28:20.762900+01:00 Isengard systemd 1 - - Reexecuting.
<3.6> 2018-03-20T22:28:20.874619+01:00 Isengard systemd 1 - - systemd 228 running in system mode. (+PAM -AUDIT +SELINUX -IMA +APPARMOR -SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT -GNUTLS +ACL +XZ -LZ4 +SECCOMP +BLKID -ELFUTILS +KMOD -IDN)
<3.6> 2018-03-20T22:28:20.881127+01:00 Isengard systemd 1 - - Detected architecture x86-64.
<3.4> 2018-03-20T22:28:21.006496+01:00 Isengard systemd 1 - - nss-lookup.target: Dependency Before=nss-lookup.target dropped
<3.6> 2018-03-20T22:28:21.356896+01:00 Isengard systemd 1 - - Reloading.
<3.4> 2018-03-20T22:28:21.578113+01:00 Isengard systemd 1 - - nss-lookup.target: Dependency Before=nss-lookup.target dropped
<3.6> 2018-03-20T22:28:55.931363+01:00 Isengard systemd 1 - - Reloading.
<3.4> 2018-03-20T22:28:56.145780+01:00 Isengard systemd 1 - - nss-lookup.target: Dependency Before=nss-lookup.target dropped
<3.6> 2018-03-20T22:28:56.226067+01:00 Isengard systemd 1 - - Reloading.
<3.4> 2018-03-20T22:28:56.502914+01:00 Isengard systemd 1 - - nss-lookup.target: Dependency Before=nss-lookup.target dropped
<3.6> 2018-03-20T22:28:56.567912+01:00 Isengard systemd 1 - - Closed udev Control Socket.
<3.6> 2018-03-20T22:28:56.608803+01:00 Isengard systemd 1 - - Stopping udev Control Socket.
<3.3> 2018-03-20T22:28:56.615726+01:00 Isengard systemd 1 - - systemd-udevd-control.socket: Socket service systemd-udevd.service already active, refusing.
<3.3> 2018-03-20T22:28:56.616377+01:00 Isengard systemd 1 - - Failed to listen on udev Control Socket.
<3.6> 2018-03-20T22:28:56.626341+01:00 Isengard systemd 1 - - Closed udev Kernel Socket.
<3.6> 2018-03-20T22:28:56.630180+01:00 Isengard systemd 1 - - Stopping udev Kernel Socket.
<3.3> 2018-03-20T22:28:56.634330+01:00 Isengard systemd 1 - - systemd-udevd-kernel.socket: Socket service systemd-udevd.service already active, refusing.
<3.3> 2018-03-20T22:28:56.635065+01:00 Isengard systemd 1 - - Failed to listen on udev Kernel Socket.
<3.6> 2018-03-20T22:28:56.635569+01:00 Isengard systemd 1 - - Reloading.
<3.4> 2018-03-20T22:28:56.844171+01:00 Isengard systemd 1 - - nss-lookup.target: Dependency Before=nss-lookup.target dropped
<3.6> 2018-03-20T22:28:56.908439+01:00 Isengard systemd 1 - - Stopping udev Kernel Device Manager...
<3.6> 2018-03-20T22:28:56.917362+01:00 Isengard systemd 1 - - Stopped udev Kernel Device Manager.
<3.6> 2018-03-20T22:28:56.917988+01:00 Isengard systemd 1 - - Listening on udev Kernel Socket.
<3.6> 2018-03-20T22:28:56.921780+01:00 Isengard systemd 1 - - Listening on udev Control Socket.
<3.6> 2018-03-20T22:28:56.934881+01:00 Isengard systemd 1 - - Starting udev Kernel Device Manager...
<3.6> 2018-03-20T22:28:56.943390+01:00 Isengard systemd-udevd 27692 - - Network interface NamePolicy= disabled by default.
<3.6> 2018-03-20T22:28:57.006406+01:00 Isengard systemd 1 - - Started udev Kernel Device Manager.
And died, suddenly. Black screen when I looked. The system has a new UPS,
so power did not fail.
/var/log/zypp/history:
2018-03-20 22:28:52|install|yast2|3.2.43-2.3.1|x86_64||download.opensuse.org-oss_1|b5037cca5b07e60124caff0cd63e2742aba4d8219bef6da7b314b0bcc6638277|
2018-03-20 22:28:52|install|dhcp-client|4.3.3-11.3.1|x86_64||download.opensuse.org-oss_1|74bb46c281463504334f8533d15a128a015eae4d86b84aaeb6fb322975e6a6e8|
2018-03-20 22:28:52|install|typelib-1_0-WebKit2-4_0|2.18.5-8.1|x86_64||download.opensuse.org-oss_1|570b50e9a6dfbafa8accc2d89dd79e8271602039fef4e050da9b30dd499ad8fa|
2018-03-20 22:28:54|install|shotwell|0.22.0+git.20160103-16.1|x86_64||download.opensuse.org-oss_1|54bcf5cebd4ca40644925eb10402151dc6a228402499e0701f907563d4211616|
2018-03-20 22:28:57|install|udev|228-47.1|x86_64||download.opensuse.org-oss_1|e8d28bbc820e881c529f998f6289d1c953006182346bd97a5ec8a9251563dcf3|
# 2018-03-20 22:28:57 libedata-cal-1_2-28-3.22.7-7.3.1.x86_64.rpm installed ok
# Additional rpm output:
# /sbin/ldconfig: /usr/lib64/libscintilla.so.3 is not a symbolic link
#
# /sbin/ldconfig: /usr/lib64/libscintilla.so.3 is not a symbolic link
#
#
2018-03-20 22:28:57|install|libedata-cal-1_2-28|3.22.7-7.3.1|x86_64||download.opensuse.org-oss_1|f9ae2575118b0898cb2f799cc720a52fa044e4fcc6681dbadba8779476f1da17|
2018-03-20 22:28:57|install|java-1_8_0-openjdk|1.8.0.161-21.1|x86_64||download.opensuse.org-oss_1|6238c300ff999fe6e0389377a7ebb67771c9a3a845492014debf64fe3deabd20|
2018-03-20 22:28:57|install|java-1_7_0-openjdk|1.7.0.171-48.3|x86_64||download.opensuse.org-oss_1|faa57b26c632f86b266f2ada096b830bdc3047c496de212d50d886ca6bf09b1e|
2018-03-20 22:28:57|install|icewm-lite|1.3.12-6.3.1|x86_64||download.opensuse.org-oss_1|de773a7b2c1eb79814a8b2442265a2c3271d687b3ad1b3c01e23c453ff070b6d|
/var/log/YaST2/y2log:
2018-03-20 22:28:53 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):249 Executing 'rpm' '--root' '/' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--noglob' '--force' '--nodeps' '--' '/var/cache/zypp/packages/download.opensuse.org-oss_1/x86_64/shotwell-0.22.0+git.20160103-16.1.x86_64.rpm'
2018-03-20 22:28:53 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):412 pid 27599 launched
2018-03-20 22:28:54 <1> Isengard(32077) [zypp++] ExternalProgram.cc(checkStatus):513 Pid 27599 successfully completed
2018-03-20 22:28:54 <1> Isengard(32077) [Ruby] modules/PackageCallbacks.rb:520 Additional RPM otput:
2018-03-20 22:28:54 <1> Isengard(32077) [zypp::posttrans++] RpmPostTransCollector.cc(tmpDir):153 /var/adm/update-scripts/posttransnNI30P
2018-03-20 22:28:54 <1> Isengard(32077) [zypp::posttrans] RpmPostTransCollector.cc(collectScriptFromPackage):70 COLLECT posttrans: /var/adm/update-scripts/posttransnNI30P/udev-228-47.1.x86_64.rpmyxSVK9{- 0700 0/0 size 214}
2018-03-20 22:28:54 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):249 Executing 'rpm' '--root' '/' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--noglob' '--force' '--nodeps' '--noposttrans' '--' '/var/cache/zypp/packages/download.opensuse.org-oss_1/x86_64/udev-228-47.1.x86_64.rpm'
2018-03-20 22:28:54 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):412 pid 27622 launched
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(checkStatus):513 Pid 27622 successfully completed
2018-03-20 22:28:57 <1> Isengard(32077) [Ruby] modules/PackageCallbacks.rb:520 Additional RPM otput:
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):249 Executing 'rpm' '--root' '/' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--noglob' '--force' '--nodeps' '--' '/var/cache/zypp/packages/download.opensuse.org-oss_1/x86_64/libedata-cal-1_2-28-3.22.7-7.3.1.x86_64.rpm'
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):412 pid 27695 launched
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(checkStatus):513 Pid 27695 successfully completed
2018-03-20 22:28:57 <1> Isengard(32077) [Ruby] modules/PackageCallbacks.rb:520 Additional RPM otput:
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):249 Executing 'rpm' '--root' '/' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--noglob' '--force' '--nodeps' '--' '/var/cache/zypp/packages/download.opensuse.org-oss_1/x86_64/java-1_8_0-openjdk-1.8.0.161-21.1.x86_64.rpm'
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):412 pid 27698 launched
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(checkStatus):513 Pid 27698 successfully completed
2018-03-20 22:28:57 <1> Isengard(32077) [Ruby] modules/PackageCallbacks.rb:520 Additional RPM otput:
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):249 Executing 'rpm' '--root' '/' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--noglob' '--force' '--nodeps' '--' '/var/cache/zypp/packages/download.opensuse.org-oss_1/x86_64/java-1_7_0-openjdk-1.7.0.171-48.3.x86_64.rpm'
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):412 pid 27699 launched
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(checkStatus):513 Pid 27699 successfully completed
2018-03-20 22:28:57 <1> Isengard(32077) [Ruby] modules/PackageCallbacks.rb:520 Additional RPM otput:
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):249 Executing 'rpm' '--root' '/' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--noglob' '--force' '--nodeps' '--' '/var/cache/zypp/packages/download.opensuse.org-oss_1/x86_64/icewm-lite-1.3.12-6.3.1.x86_64.rpm'
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):412 pid 27700 launched
2018-03-20 22:28:57 <1> Isengard(32077) [zypp++] ExternalProgram.cc(checkStatus):513 Pid 27700 successfully completed
2018-03-20 22:28:57 <1> Isengard(32077) [Ruby] modules/PackageCallbacks.rb:520 Additional RPM otput:
2018-03-20 22:28:58 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):249 Executing 'rpm' '--root' '/' '--dbpath' '/var/lib/rpm' '-U' '--percent' '--noglob' '--force' '--nodeps' '--' '/var/cache/zypp/packages/download.opensuse.org-oss_1/x86_64/grub2-x86_64-efi-2.02-7.1.x86_64.rpm'
2018-03-20 22:28:58 <1> Isengard(32077) [zypp++] ExternalProgram.cc(start_program):412 pid 27704 launched
So the last thing that was being installed was
"grub2-x86_64-efi-2.02-7.1.x86_64.rpm" at 22:28:58, but systemd and udev
were doing something that I don't understand.
Now, how do I recover from this?
I downloaded "bootinfoscript" and run it. RESULTS.txt attached.
sda is the server disk. sda4 is the root, sda5 home.
sdb is the rescue system.
localhost:~ # lsblk --output NAME,KNAME,RA,RM,RO,SIZE,TYPE,FSTYPE,LABEL,PARTLABEL,MOUNTPOINT,UUID,PARTUUID,WWN,MODEL,ALIGNMENT
NAME KNAME RA RM RO SIZE TYPE FSTYPE LABEL PARTLABEL MOUNTPOINT UUID PARTUUID WWN MODEL ALIGNMENT
loop0 loop0 512 0 1 529.8M loop squash /run/initramf 0
loop1 loop1 512 0 1 3G loop ext4 /run/rootfsba a7f2c9f2-6b5d-4fe8-bb0a-9b8a9001a59e 0
sda sda 512 0 0 111.8G disk KINGSTON SMS 0
├─sda1 sda1 512 0 0 156M part vfat primary BD39-068A cada5ef3-03a1-4d0b-a984-49c1c16c75bb 0
├─sda2 sda2 512 0 0 9G part swap Swap primary dee28afc-9697-4f8c-9b42-da0cf6da0ff1 53321f59-61c7-48f0-a358-fedac3882160 0
├─sda3 sda3 512 0 0 9G part ext4 Other primary 2d5ceee1-bb49-4afd-8b36-0fb76937172c 375c1f95-f5f8-434d-8268-8a4a708b6a8a 0
├─sda4 sda4 512 0 0 20G part ext4 System primary /run/media/li 0d457df1-b43d-4587-aa5a-6c919bcbedb8 45be0ffb-ec39-4fb3-b44b-dc87d96c05ab 0
└─sda5 sda5 512 0 0 70G part crypto primary ff42b205-a2a7-4d9a-84e9-4729c6775108 78016944-086f-4603-9673-2485beb1388d 0
sdb sdb 512 1 0 7.2G disk iso966 openSUSE_Tumbleweed_Rescue_CD 2018-03-19-12-59-39-31 TransMemory 0
├─sdb1 sdb1 512 1 0 15M part vfat BOOT 3993-FBBB 407025f9-01 0
├─sdb2 sdb2 512 1 0 619.8M part iso966 openSUSE_Tumbleweed_Rescue_CD /run/initramf 2018-03-19-12-59-36-28 407025f9-02 0
└─sdb3 sdb3 512 1 0 6.6G part ext4 cow /run/overlayf 5466dd47-a5ba-40fe-8ee0-d730735fcab2 407025f9-03 0
localhost:~ #
localhost:~ # df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.8G 8.0K 3.8G 1% /dev
tmpfs 3.9G 0 3.9G 0% /dev/shm
tmpfs 3.9G 18M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sdb2 619M 619M 0 100% /run/initramfs/live
/dev/loop0 530M 530M 0 100% /run/initramfs/squashfs_container
/dev/loop1 2.9G 2.1G 623M 78% /run/rootfsbase
/dev/sdb3 6.5G 316M 5.8G 6% /run/overlayfs
LiveOS_rootfs 6.5G 316M 5.8G 6% /
tmpfs 784M 8.0K 784M 1% /run/user/1000
tmpfs 784M 0 784M 0% /run/user/0
/dev/sda4 20G 14G 4.7G 76% /run/media/linux/System
localhost:~ #
So, disk is not full.
The machine contains two openSUSE Linux systems; the other is labelled
"Other". I can boot none, as the display goes completely black when Grub
should display.
What do I do to recover this machine?
Try to install grub again? I don't know if it is installed at all at this
point.
I have a headache...
- --
Cheers
Carlos E. R.
(from 42.3 x86_64 "Malachite" at Telcontar)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iEYEARECAAYFAlqxl5MACgkQtTMYHG2NR9URgQCfYRD+2LHUXBAi+YkR5qV8sC6j
s1YAnRp1x0RNxOschbgISqAnba1jIw8T
=mqmY
-----END PGP SIGNATURE-----