[opensuse-virtual] Chainloaded Xen Dom0 on EFI hardware FAILs at grub config generation after upgrade from Opensuse 13.2 -> Leap 42.1. Has the Xen-on-EFI prep process changed?

PGNet Dev

17 Nov 2015 17 Nov '15

00:42

I've been running a Xen Dom0 server Opensuse 13.2 Xen from /Virtualization kernel from Kernel:Stable rpm -qa | egrep "kernel-xen-4|^xen-4|grub2-x" | sort grub2-x86_64-efi-2.02~beta2-68.2.x86_64 grub2-x86_64-xen-2.02~beta2-68.2.x86_64 kernel-xen-4.3.0-4.1.g734b32c.x86_64 xen-4.6.0_02-397.1.x86_64 on EFI hardware. I've been chainloading xen.efi. I'm attempting an upgrade on the server, from Opensuse 13.2 -> Leap. After the upgrade, but before a reboot, generation of the grub config now FAILs during the mkinitrd step(s) @ console /usr/sbin/grub2-mkconfig Generating grub configuration file ... # # DO NOT EDIT THIS FILE # # It is automatically generated by grub2-mkconfig using templates # from /etc/grub.d and settings from /etc/default/grub # ### BEGIN /etc/grub.d/00_header ### if [ -f ${config_directory}/grubenv ]; then load_env -f ${config_directory}/grubenv elif [ -s $prefix/grubenv ]; then load_env fi if [ "${env_block}" ] ; then load_env -f "${env_block}" fi if [ "${next_entry}" ] ; then set default="${next_entry}" set next_entry= save_env next_entry if [ "${env_block}" ] ; then save_env -f "${env_block}" next_entry fi set boot_once=true else set default="${saved_entry}" fi if [ x"${feature_menuentry_id}" = xy ]; then menuentry_id_option="--id" else menuentry_id_option="" fi export menuentry_id_option if [ "${prev_saved_entry}" ]; then set saved_entry="${prev_saved_entry}" save_env saved_entry set prev_saved_entry= save_env prev_saved_entry set boot_once=true fi function savedefault { if [ -z "${boot_once}" ]; then saved_entry="${chosen}" save_env saved_entry fi } function load_video { if [ x$feature_all_video_module = xy ]; then insmod all_video else insmod efi_gop insmod efi_uga insmod ieee1275_fb insmod vbe insmod vga insmod video_bochs insmod video_cirrus fi } serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1 terminal_input console serial terminal_output console serial if [ x${boot_once} = xtrue ]; then set timeout=0 elif [ x$feature_timeout_style = xy ] ; then set timeout_style=menu set timeout=8 # Fallback normal timeout code in case the timeout_style feature is # unavailable. else set timeout=8 fi ### END /etc/grub.d/00_header ### ... ### BEGIN /etc/grub.d/20_linux_xen ### Found hypervisor: /boot/xen-4.6.0_02-397.gz Found linux image: /boot/vmlinuz-4.3.0-4.g734b32c-xen Found initrd image: /boot/initrd-4.3.0-4.g734b32c-xen menuentry 'openSUSE Leap 42.1, with Xen hypervisor' --class opensuse --class gnu-linux --class gnu --class os --class xen $menuentry_id_option 'xen-gnulinux-simple-4716053b-39da-4cc1-a7ce-72171b862357' { echo 'Loading Xen 4.6.0_02-397 with Linux 4.3.0-4.g734b32c-xen ...' chainloader $cmdpath/xen-4.6.0_02-397.efi xen-4.6.0_02-397.efi config.1 } cp: cannot stat '/vmlinuz-4.3.0-4.g734b32c-xen': No such file or directory Where ls -al /boot/efi/EFI/opensuse total 2.3M drwxrwxr-x 2 root root 4.0K Nov 16 17:16 ./ drwxrwxr-x 4 root root 4.0K Mar 28 2015 ../ -rwxrwxr-x 1 root root 133K Nov 16 15:26 grubx64.efi* -rwxrwxr-x 1 root root 42 Nov 16 17:16 grub.xen-files* -rwxrwxr-x 1 root root 990 Nov 16 17:16 xen-4.6.0_02-397.cfg* -rwxrwxr-x 1 root root 2.1M Nov 11 13:18 xen-4.6.0_02-397.efi* Have the prep steps needed changed? -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

Show replies by date

PGNet Dev

17 Nov 17 Nov

00:52

New subject: [opensuse-virtual] Re: Chainloaded Xen Dom0 on EFI hardware FAILs at grub config generation after upgrade from Opensuse 13.2 -> Leap 42.1. Has the Xen-on-EFI prep process changed?

On 11/16/2015 04:42 PM, PGNet Dev wrote:

...

I've been running a Xen Dom0 server

Also should note, on my server, I'd had/used cat /boot/grub2/custom.cfg menuentry 'openSUSE kernel-default, symlink, DEBUG' { load_video set gfxpayload=1920x1080 insmod gzio insmod part_gpt gpt insmod diskfilter mdraid1x insmod ext2 linuxefi /vmlinuz root=/dev/VG0/ROOT ro dolvm lvmwait=/dev/mapper/VG0-ROOT rootfstype=ext4 rootflags=journal_checksum ... initrdefi /initrd } menuentry 'Xen EFI' { insmod part_gpt gpt insmod gzio insmod diskfilter mdraid1x insmod ext2 insmod search_fs_uuid insmod chain search --fs-uuid 986E-2DB9 --set chainloader /EFI/XEN/xen.efi } after, cp -f \ /usr/lib64/efi/xen.efi \ /boot/vmlinuz-xen \ /boot/initrd-xen \ /boot/efi/EFI/XEN/ -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

PGNet Dev

00:59

New subject: [opensuse-virtual] Re: Chainloaded Xen Dom0 on EFI hardware FAILs at grub config generation after upgrade from Opensuse 13.2 -> Leap 42.1. Has the Xen-on-EFI prep process changed?

Looking at the diff between /etc/grub.d/20_linux_xen before and after the upgrade, https://www.diffchecker.com/ysgnbi7z there's clearly a lot of code added for Xen-on-EFI processing/preparation -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

PGNet Dev

19:18

New subject: [opensuse-virtual] Re: Chainloaded Xen Dom0 on EFI hardware FAILs at grub config generation after upgrade from Opensuse 13.2 -> Leap 42.1. Has the Xen-on-EFI prep process changed?

Is this a Xen package issue, and appropriately discussed here on opensuse-virtual? Or, rather kernel or grub2 ? -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

Olaf Hering

18 Nov 18 Nov

09:48

New subject: [opensuse-virtual] Re: Chainloaded Xen Dom0 on EFI hardware FAILs at grub config generation after upgrade from Opensuse 13.2 -> Leap 42.1. Has the Xen-on-EFI prep process changed?

On Tue, Nov 17, PGNet Dev wrote:

...

Is this a Xen package issue, and appropriately discussed here on opensuse-virtual? Or, rather kernel or grub2 ?

Likely a bug in grub2. Bug #955493 will track this. I dont have an EFI capable system at hand, so cant help much with the issue. Olaf -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

PGNet Dev

20:09

New subject: [opensuse-virtual] Chainloaded Leap 42.1 Xen Dom0 on EFI hardware -- network offline, xen init failing: "Failed to start The Xen xenstore"

On 11/18/2015 01:48 AM, Olaf Hering wrote:

...

Likely a bug in grub2. Bug #955493 will track this. I dont have an EFI capable system at hand, so cant help much with the issue.

Grub-related, it seems. Missing/incorrectly populated `startup.nsh` is the problem. Details at bug, with manual workaround. But, AFTER the manual workaround, on system boot, network's down -- no xen bridge -- and there appears to be a launch problem with xen init: With the aformentioned edit/repair to `startup.nsh`, I can reliably boot the machine. But, network, as mentioned, is offline. After boot, the physical interfaces exist ip link show 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 68:1c:a2:XX:XX:XX brd ff:ff:ff:ff:ff:ff 3: eth1: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 68:1c:a2:XX:XX:XY brd ff:ff:ff:ff:ff:ff 5: eth2: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 0c:c4:7a:XX:XX:XZ brd ff:ff:ff:ff:ff:ff There's no longer any xen bridge, or properly-named eth interface, here eno1 cat /etc/sysconfig/network/ifcfg-br0 NAME='br0' BRIDGE='yes' BRIDGE_PORTS='eno1' STARTMODE='auto' BOOTPROTO='static' BRIDGE_STP='off' BRIDGE_FORWARDDELAY='0' BRIDGE_AGEINGTIME='0' IPADDR0='XX.XX.XX.2/22' ETHTOOL_OPTIONS='' MTU='1500' NETWORK='' BROADCAST='' REMOTE_IPADDR='' USERCONTROL='no' POST_UP_SCRIPT=/etc/sysconfig/network/scripts/br0-config.sh cat /etc/sysconfig/network/scripts/br0-config.sh #!/bin/sh ethtool -K br0 tso off gso off ethtool -K eno1 tso off gso off cat /etc/sysconfig/network/ifcfg-eno1 STARTMODE='auto' BOOTPROTO='none' only ifconfig lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:14 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:980 (980.0 b) TX bytes:980 (980.0 b) a try at manual ifup also fails ifup br0 wicked: ifcfg-br0: unable to qualify post-up script - unknown script type '(null):/etc/sysconfig/network/scripts/br0-config.sh' `dmesg` reports no obvious network errors, dmesg | egrep -i "eth|br0|eno" [ 6.789845] Initialising virtual ethernet driver. [ 6.954169] FUJITSU Extended Socket Network Device Driver - version 1.0 - Copyright (c) 2015 FUJITSU LIMITED [ 7.030614] e1000e: Intel(R) PRO/1000 Network Driver - 3.2.6-k [ 7.203574] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 7.204301] r8169 0000:0a:00.0 eth0: RTL8168evl/8111evl at 0xffffc9000006a000, 68:1c:a2:XX:XX:XX, XID 0c900800 IRQ 60 [ 7.204303] r8169 0000:0a:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko] [ 7.204339] r8169 Gigabit Ethernet driver 2.3LK-NAPI loaded [ 7.205000] r8169 0000:0b:00.0 eth1: RTL8168evl/8111evl at 0xffffc9000006c000, 68:1c:a2:XX:XX:XY, XID 0c900800 IRQ 61 [ 7.205002] r8169 0000:0b:00.0 eth1: jumbo features [frames: 9200 bytes, tx checksumming: ko] [ 7.214583] igb: Intel(R) Gigabit Ethernet Network Driver - version 5.3.0-k [ 7.324135] igb 0000:10:00.0: added PHC on eth2 [ 7.324136] igb 0000:10:00.0: Intel(R) Gigabit Ethernet Network Connection [ 7.324138] igb 0000:10:00.0: eth2: (PCIe:2.5Gb/s:Width x1) 0c:c4:7a:XX:XX:XZ [ 7.324204] igb 0000:10:00.0: eth2: PBA No: 011A00-000 [ 7.576496] e1000e 0000:00:19.0 eth3: registered PHC clock [ 7.576499] e1000e 0000:00:19.0 eth3: (PCI Express:2.5GT/s:Width x1) 0c:c4:7a:XX:XX:XW [ 7.576500] e1000e 0000:00:19.0 eth3: Intel(R) PRO/1000 Network Connection [ 7.576556] e1000e 0000:00:19.0 eth3: MAC: 11, PHY: 12, PBA No: 0110FF-0FF [ 17.948864] e1000e 0000:00:19.0 rename5: renamed from eth3 [ 17.999474] igb 0000:10:00.0 eth3: renamed from eth2 [ 18.049069] e1000e 0000:00:19.0 eth2: renamed from rename5 [ 169.255750] igb 0000:10:00.0: removed PHC on eth3 Taking a look at console output, there a problem with xen init ... [ 23.759860] xen:evtchn: Event-channel device installed [ 23.821839] No iBFT detected. [ 24.009822] usbcore: registered new interface driver usbback [ 24.015940] pciback: backend is vpci [ 24.024580] blktap_device_init: blktap device major 252 [ 24.024583] blktap_ring_init: blktap ring major: 243 [ 24.037250] systemd[1]: Received SIGRTMIN+21 from PID 447 (plymouthd). [ 77.512048] systemd[1]: systemd-journald.service watchdog timeout! [ 114.500062] systemd[1]: xenstored.service start operation timed out. Terminating. [ 114.500207] systemd[1]: Failed to start The Xen xenstore. [ 114.500217] systemd[1]: Dependency failed for xencommons. [ 114.500229] systemd[1]: Dependency failed for qemu for xen dom0 disk backend. [ 114.500235] systemd[1]: Dependency failed for xen-init-dom0, initialise Dom0 configuration (xenstore nodes, JSON configuration stub). [ 114.500242] systemd[1]: Dependency failed for Xenconsoled - handles logging from guest consoles and hypervisor. [ 114.500248] systemd[1]: Dependency failed for Xendomains - start and stop guests on boot and shutdown. [ 114.500265] systemd[1]: Unit xenstored.service entered failed state. [ 114.500616] systemd[1]: Cannot add dependency job for unit logrotate.timer, ignoring: Unit logrotate.timer failed to load: No such file or directory. [ 114.500716] systemd[1]: Mounted Mount /proc/xen files. [ 114.500727] systemd[1]: Starting The Xen xenstore... [ 137.500071] systemd[1]: serial-getty@xvc0.service holdoff time over, scheduling restart. [ 137.500350] systemd[1]: Cannot add dependency job for unit logrotate.timer, ignoring: Unit logrotate.timer failed to load: No such file or directory. [ 137.500443] systemd[1]: Stopping Serial Getty on xvc0... [ 137.500509] systemd[1]: Starting Serial Getty on xvc0... [ 137.501048] systemd[1]: Started Serial Getty on xvc0. [ 167.750050] systemd[1]: systemd-journald.service stop-sigterm timed out. Killing. [ 167.750567] systemd[1]: systemd-journald.service: main process exited, code=killed, status=9/KILL [ 167.750701] systemd[1]: Unit systemd-journald.service entered failed state. [ 167.750771] systemd[1]: systemd-journald.service has no holdoff time, scheduling restart. [ 167.750817] systemd[1]: Stopping Journal Service... [ 167.750843] systemd[1]: Starting Journal Service... [ 167.823905] systemd-journald[1493]: File /var/log/journal/ff2dfa896235475385422eea2cb7fe4b/system.journal corrupted or uncleanly shut down, renaming and replacing. [ 168.840182] systemd-journald[1493]: Failed to open runtime journal: Device or resource busy [ 169.104166] systemd[1]: Started Journal Service. [ 169.253804] Unable to read sysrq code in control/sysrq [ 169.255750] igb 0000:10:00.0: removed PHC on eth3 [ 169.264906] pciback 0000:10:00.0: seizing device [ 169.328047] systemd[1]: Cannot add dependency job for unit logrotate.timer, ignoring: Unit logrotate.timer failed to load: No such file or directory. [ 170.224153] systemd-journald[1493]: Received request to flush runtime journal from PID 1 [ 172.004051] systemd[1]: Cannot add dependency job for unit logrotate.timer, ignoring: Unit logrotate.timer failed to load: No such file or directory. xl list Name ID Mem VCPUs State Time(s) (null) 0 2988 1 r----- 35.0 ps ax | grep -i xen 14 ? S 0:00 [xenwatch] 15 ? S 0:00 [xenbus] 1451 ? S< 0:00 [xen_pciback_wor] 1483 ? Ss 0:00 /usr/sbin/xenstored --no-fork 3310 xvc0 R+ 0:00 grep --color=auto -i xen -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

Olaf Hering

19 Nov 19 Nov

09:35

New subject: [opensuse-virtual] Chainloaded Leap 42.1 Xen Dom0 on EFI hardware -- network offline, xen init failing: "Failed to start The Xen xenstore"

On Wed, Nov 18, PGNet Dev wrote:

...

After boot, the physical interfaces exist

With their kernel names (ethN).

...

There's no longer any xen bridge, or properly-named eth interface, here eno1

cat /etc/sysconfig/network/ifcfg-br0 NAME='br0' BRIDGE='yes' BRIDGE_PORTS='eno1'

eno1 does not exist as interface name.

...

wicked: ifcfg-br0: unable to qualify post-up script - unknown script type '(null):/etc/sysconfig/network/scripts/br0-config.sh'

Not sure if these hooks are supported that way. Perhaps wicked already has ethtool knobs.

...

Taking a look at console output, there a problem with xen init

At least xen works on my Leap installation. Not sure why EFI would make any difference. Are there stale xen related .service files or sysv scripts which prevent xenstored.service/xenstored.socked from starting? Our xencommons.service is supposed to start everything. Does "systemctl start xencommons.service" give any errors? Could be that it fails because its dependencies are in failed state. If in doubt open a bug and attach journalctl -b output. The posted output also shows journal errors, perhaps the issue is unrelated to xen. Olaf -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

PGNet Dev

14:43

New subject: [opensuse-virtual] Chainloaded Leap 42.1 Xen Dom0 on EFI hardware -- network offline, xen init failing: "Failed to start The Xen xenstore"

On 11/19/2015 01:35 AM, Olaf Hering wrote:

...

...
There's no longer any xen bridge, or properly-named eth interface, here eno1

cat /etc/sysconfig/network/ifcfg-br0 NAME='br0' BRIDGE='yes' BRIDGE_PORTS='eno1'

eno1 does not exist as interface name.

Hm. It certainly _used_ to prior to the upgrade ...

...

...
wicked: ifcfg-br0: unable to qualify post-up script - unknown script type '(null):/etc/sysconfig/network/scripts/br0-config.sh'

Not sure if these hooks are supported that way. Perhaps wicked already has ethtool knobs.

Again, it worked absolutely fine prior to the upgrade; the script executed, and the knobs' values were verified to be set. That preceding "(null):" is a problem, but iirc is due to the lack of properly init'd xenstored

...

At least xen works on my Leap installation. Not sure why EFI would make any difference. Are there stale xen related .service files or sysv scripts which prevent xenstored.service/xenstored.socked from starting? Our xencommons.service is supposed to start everything.

I made a little progress. I read this http://lists.xenproject.org/archives/html/xen-devel/2015-10/msg01262.html exec'd these systemctl enable xenstored.service systemctl enable xenconsoled.service systemctl enable xen-init-dom0.service systemctl enable xen-qemu-dom0-disk-backend.service systemctl enable xendomains.service So now, systemctl list-unit-files | grep -i xen proc-xen.mount static var-lib-xenstored.mount static xen-dom0-modules.service disabled xen-init-dom0.service enabled xen-qemu-dom0-disk-backend.service enabled xen-watchdog.service disabled xencommons.service enabled xenconsoled.service enabled xendomains.service enabled xenstored.service enabled xenstored.socket enabled xenstored_ro.socket enabled Also found & removed these from the /etc/default/grub - GRUB_CMDLINE_LINUX_XEN_REPLACE=" (.*) rdloaddriver=xennet rdloaddriver=xenblk" + GRUB_CMDLINE_LINUX_XEN_REPLACE=" (.*) " Since I'd been using my own, custom xen.cfg, I'd never noticed these before. Here's the current, post-upgrade systemd script status With BOTH of those changes, I no longer see Xen dependency problems reported on boot, dmesg | grep -i xen ... [ 0.000000] Linux version 4.3.0-4.g734b32c-xen (geeko@buildhost) (gcc version 5.2.1 20151008 [gcc-5-branch revision 228597] (SUSE Linux) ) #1 SMP Sat Nov 14 16:19:19 UTC 2015 (734b32c) [ 0.000000] Command line: root=UUID=471... rootfstype=ext4 rootflags=journal_checksum noresume showopts noquiet video=vesa:off video=efifb:1024x768 video=o [ 0.000000] Xen-provided machine memory map: [ 0.000000] e820: Xen-provided physical RAM map: [ 0.000000] Xen: [mem 0x0000000000000000-0x00000000c07fffff] usable [ 0.000000] Kernel command line: root=UUID=471... rootfstype=ext4 rootflags=journal_checksum noresume showopts noquiet video=vesa:off video=efifb:1024x768o [ 0.000000] Xen reported: 3092.928 MHz processor. [ 0.000000] clocksource: xen: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.377137] xen_mem: Initialising balloon driver. [ 0.388028] clocksource: Switched to clocksource xen [ 1.698180] Xen virtual console successfully installed as xvc0 [ 5.023083] usb usb1: Manufacturer: Linux 4.3.0-4.g734b32c-xen xhci-hcd [ 5.243846] usb usb2: Manufacturer: Linux 4.3.0-4.g734b32c-xen xhci-hcd [ 5.329938] usb usb3: Manufacturer: Linux 4.3.0-4.g734b32c-xen xhci-hcd [ 5.331195] usb usb4: Manufacturer: Linux 4.3.0-4.g734b32c-xen xhci-hcd [ 5.588164] usb usb5: Manufacturer: Linux 4.3.0-4.g734b32c-xen ehci_hcd [ 5.604160] usb usb6: Manufacturer: Linux 4.3.0-4.g734b32c-xen ehci_hcd

...

Does "systemctl start xencommons.service" give any errors? Could be that it fails because its dependencies are in failed state.

> > systemctl status xencommons.service xencommons.service - xencommons Loaded: loaded (/usr/lib/systemd/system/xencommons.service; enabled) Active: inactive (dead) > systemctl start xencommons.service Welcome to emergency mode! After logging in, type "journalctl -xb" to view system logs, "systemctl reboot" to reboot, "systemctl default" to try again to boot into default mode. Give root password for maintenance (or press Control-D to continue):

...

If in doubt open a bug and attach journalctl -b output. The posted output also shows journal errors, perhaps the issue is unrelated to xen.

First, at a glance, in the journalctl -- not dmesg -- output I _do_ see: Nov 18 19:15:56 xensvr systemd-udevd[812]: Network interface NamePolicy= disabled on kernel commandline, ignoring. I've no idea what it's referring to, as dmesg | grep -i "kernel command" [ 0.000000] Kernel command line: root=UUID=471... rootfstype=ext4 rootflags=journal_checksum noresume showopts noquiet video=vesa:off video=efifb:1024x768 video=o=HDMI-A-1:1920x1080@60 xencons=xvc console=tty0 console=xvc0 clocksource=xen apparmor=0 selinux=0 nomodeset noquiet showopts elevator=cfq mce=off plymouth.enable=0 log_buf_len=10M print_fatal_signals=1 loglevel=6 systemd.log_level=warning systemd.log_target=kmsg earlyprintk=xen which derives directly from cat /etc/default/grub GRUB_CMDLINE_LINUX_XEN_REPLACE=" rootfstype=ext4 rootflags=journal_checksum noresume showopts noquiet video=vesa:off video=efifb:1024x768 video=HDMI-A-1:1920x1080@60 xencons=xvc console=tty0 console=xvc0 clocksource=xen apparmor=0 selinux=0 nomodeset noquiet showopts elevator=cfq mce=off plymouth.enable=0 log_buf_len=10M print_fatal_signals=1 loglevel=6 systemd.log_level=warning systemd.log_target=kmsg earlyprintk=xen" but "NamePolicy= disabled" certainly looks suspicious. -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

Olaf Hering

15:10

New subject: [opensuse-virtual] Chainloaded Leap 42.1 Xen Dom0 on EFI hardware -- network offline, xen init failing: "Failed to start The Xen xenstore"

On Thu, Nov 19, PGNet Dev wrote:

...

On 11/19/2015 01:35 AM, Olaf Hering wrote:

...
eno1 does not exist as interface name. Hm. It certainly _used_ to prior to the upgrade ...

It might be caused by the fact that Leap uses SLE12 as a base for systemd, which forces net.ifnames=0. Try to boot with net.ifnames=1 on kernel cmdline.

...

Again, it worked absolutely fine prior to the upgrade; the script executed, and the knobs' values were verified to be set.

Likely by ifup scripts instead of wicked.

...

That preceding "(null):" is a problem, but iirc is due to the lack of properly init'd xenstored

wicked and xenstored are unrelated.

...

...
At least xen works on my Leap installation. Not sure why EFI would make any difference. Are there stale xen related .service files or sysv scripts which prevent xenstored.service/xenstored.socked from starting? Our xencommons.service is supposed to start everything.

I made a little progress.

I read this

http://lists.xenproject.org/archives/html/xen-devel/2015-10/msg01262.html

exec'd these

systemctl enable xenstored.service systemctl enable xenconsoled.service systemctl enable xen-init-dom0.service systemctl enable xen-qemu-dom0-disk-backend.service systemctl enable xendomains.service

So now,

systemctl list-unit-files | grep -i xen proc-xen.mount static var-lib-xenstored.mount static xen-dom0-modules.service disabled xen-init-dom0.service enabled xen-qemu-dom0-disk-backend.service enabled xen-watchdog.service disabled xencommons.service enabled xenconsoled.service enabled xendomains.service enabled xenstored.service enabled xenstored.socket enabled xenstored_ro.socket enabled

Which one was missing? xencommons.service is supposed to start them all.

...

Also found & removed these from the /etc/default/grub

- GRUB_CMDLINE_LINUX_XEN_REPLACE=" (.*) rdloaddriver=xennet rdloaddriver=xenblk" + GRUB_CMDLINE_LINUX_XEN_REPLACE=" (.*) "

This is unrelated I think.

...

Nov 18 19:15:56 xensvr systemd-udevd[812]: Network interface NamePolicy= disabled on kernel commandline, ignoring.

I've no idea what it's referring to, as

Likely because its forced to be off in the sources. Olaf -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

PGNet Dev

15:54

New subject: [opensuse-virtual] Chainloaded Leap 42.1 Xen Dom0 on EFI hardware -- network offline, xen init failing: "Failed to start The Xen xenstore"

On 11/19/2015 07:10 AM, Olaf Hering wrote:

...

It might be caused by the fact that Leap uses SLE12 as a base for systemd, which forces net.ifnames=0. Try to boot with net.ifnames=1 on kernel cmdline.

bingo! adding net.ifnames=1 now, after boot, net's up ip link show | grep DEFAULT 1: lo: mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default 2: enp10s0: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 3: enp11s0: mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000 5: eno1: mtu 1500 qdisc pfifo_fast master br0 state UP mode DEFAULT group default qlen 1000 6: br0: mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 Immediately after boot, I can ping outbound. I can't SSH *to* the machine -- sshd's not starting on boot. I *can* restart it after boot, and subsequently SSH in. WIll chase that down ... suspect some changed dependency .... So, appears there IS this default-value change from OS 13.2 -> Leap. If that's the case, I can't imagine that this won't crop up for others, and netnaming became standard with OS 13.2, and some netconfigs no doubt addressed those names. Is the need for the command-line parameter addition/change documented? Since Leap now tracks SLES, will the default stay @ netnames = 0, requiring a spec'n on command line? Should that setting be taken instead from, e.g., /etc/default/* or /etc/sysconfig/* ? Curious, what's TW's default value? Same as 13.2 or same as Leap? Did *it* change as well?

...

...
Again, it worked absolutely fine prior to the upgrade; the script executed, and the knobs' values were verified to be set.

Likely by ifup scripts instead of wicked.

What is the wicked-native alternative?

...

...
That preceding "(null):" is a problem, but iirc is due to the lack of properly init'd xenstored

wicked and xenstored are unrelated.

Sure. Still now, after boot xl list Name ID Mem VCPUs State Time(s) (null) 0 2988 1 r----- 21.7 which leaves me with "why?"

...

...
systemctl list-unit-files | grep -i xen proc-xen.mount static var-lib-xenstored.mount static xen-dom0-modules.service disabled xen-init-dom0.service enabled xen-qemu-dom0-disk-backend.service enabled xen-watchdog.service disabled xencommons.service enabled xenconsoled.service enabled xendomains.service enabled xenstored.service enabled xenstored.socket enabled xenstored_ro.socket enabled

Which one was missing? xencommons.service is supposed to start them all.

Prior to my 'enables', I'd only checked systemctl list-unit-files ... xen-dom0-modules.service disabled xen-init-dom0.service disabled xen-qemu-dom0-disk-backend.service disabled xen-watchdog.service disabled xencommons.service enabled xenconsoled.service disabled xendomains.service enabled xenstored.service disabled ...

...

...
Also found & removed these from the /etc/default/grub

- GRUB_CMDLINE_LINUX_XEN_REPLACE=" (.*) rdloaddriver=xennet rdloaddriver=xenblk" + GRUB_CMDLINE_LINUX_XEN_REPLACE=" (.*) "

This is unrelated I think.

I'm not finding recent docs on rdloaddriver= usage, particularly wrt =xennet, =xenblk. Something clearly set those ... Now I've a booted system, with networking, I note on reboot a Xen+EFI failure: shutdown -r now ... Rebooting. [ 919.933417] reboot: Restarting system (XEN) [2015-11-19 16:42:17] Hardware Dom0 shutdown: rebooting machine (XEN) [2015-11-19 16:42:17] APIC error on CPU0: 40(00) (XEN) [2015-11-19 16:42:17] ----[ Xen-4.6.0_02-397 x86_64 debug=n Tainted: C ]---- (XEN) [2015-11-19 16:42:17] CPU: 0 (XEN) [2015-11-19 16:42:17] RIP: e008:[<000000003e670340>] 000000003e670340 (XEN) [2015-11-19 16:42:17] RFLAGS: 0000000000010202 CONTEXT: hypervisor (d0v0) (XEN) [2015-11-19 16:42:17] rax: 000000003e670340 rbx: 0000000000000000 rcx: 0000000000000000 (XEN) [2015-11-19 16:42:17] rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) [2015-11-19 16:42:17] rbp: 0000000000000000 rsp: ffff83002c8dfdc0 r8: 0000000000000000 (XEN) [2015-11-19 16:42:18] r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) [2015-11-19 16:42:18] r12: 0000000000000000 r13: 0000000000000cf9 r14: 0000000000000065 (XEN) [2015-11-19 16:42:18] r15: ffff830000000000 cr0: 0000000080050033 cr4: 00000000001526e0 (XEN) [2015-11-19 16:42:18] cr3: 00000008abd60000 cr2: 000000003e670340 (XEN) [2015-11-19 16:42:18] ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) [2015-11-19 16:42:18] Xen stack trace from rsp=ffff83002c8dfdc0: (XEN) [2015-11-19 16:42:18] 000000003efe42f6 0000000000000065 ffff82d08022b8ca efff000000000000 (XEN) [2015-11-19 16:42:18] ffff82d080269000 00000008abd60000 ffff82d08022bada 0000000675611000 (XEN) [2015-11-19 16:42:18] 0000000000000000 0000000000152660 000000000000e008 0000000000000292 (XEN) [2015-11-19 16:42:18] 0000000000000000 00000000fffffffe ffff82d0801822c8 0000000000000000 (XEN) [2015-11-19 16:42:18] 0000000000000010 000083002c8dfe98 0000000000000000 000000e126a3d666 (XEN) [2015-11-19 16:42:18] 0000000000000001 0000000000000001 ffff8308abd1f000 ffff8308abd1f158 (XEN) [2015-11-19 16:42:18] 0000000000000002 00000000fee1dead ffff82d08012a04f ffff8800bb008528 (XEN) [2015-11-19 16:42:18] ffff82d0801058b1 0000000000000000 0000000000000000 ffff8800b1ee7dc0 (XEN) [2015-11-19 16:42:18] 0000000001234567 ffffffff80a52c60 ffff82d080129182 00000001bb008528 (XEN) [2015-11-19 16:42:18] ffff82d08012a0bb ffff83002d6e5000 ffff83002d6e5000 ffff8800bb003df0 (XEN) [2015-11-19 16:42:18] ffff83002d6e5000 ffff8800b1ee7dc0 ffff82d080228070 ffff8800bb008528 (XEN) [2015-11-19 16:42:18] ffff8800bb0084e8 0000000000000000 ffff8800bb00bc00 ffff8800b1ee7dc0 (XEN) [2015-11-19 16:42:18] 0000000000000000 0000000000000296 0000000000000008 000000000000059e (XEN) [2015-11-19 16:42:18] 0000000000000000 000000000000001d ffffffff800113aa 0000000000000001 (XEN) [2015-11-19 16:42:18] ffff8800b1ee7dbc 0000000000000002 0001010000000000 ffffffff800113aa (XEN) [2015-11-19 16:42:18] 000000000000e033 0000000000000296 ffff8800b1ee7da0 000000000000e02b (XEN) [2015-11-19 16:42:18] 0000000000000000 0000000000000000 0000000000000000 0000000000000000 (XEN) [2015-11-19 16:42:18] 0000000000000000 ffff83002d6e5000 0000000000000000 0000000000000000 (XEN) [2015-11-19 16:42:18] Xen call trace: (XEN) [2015-11-19 16:42:18] [<000000003e670340>] 000000003e670340 (XEN) [2015-11-19 16:42:18] [<ffff82d08022b8ca>] efi_rs_enter+0xfa/0x120 (XEN) [2015-11-19 16:42:18] [<ffff82d08022bada>] efi_reset_system+0x3a/0x60 (XEN) [2015-11-19 16:42:18] [<ffff82d0801822c8>] machine_restart+0x208/0x2d0 (XEN) [2015-11-19 16:42:18] [<ffff82d08012a04f>] hwdom_shutdown+0xbf/0xc0 (XEN) [2015-11-19 16:42:18] [<ffff82d0801058b1>] domain_shutdown+0xf1/0x100 (XEN) [2015-11-19 16:42:18] [<ffff82d080129182>] do_sched_op+0x1d2/0x410 (XEN) [2015-11-19 16:42:18] [<ffff82d08012a0bb>] __do_softirq+0x6b/0x90 (XEN) [2015-11-19 16:42:18] [<ffff82d080228070>] lstar_enter+0xa0/0xa5 (XEN) [2015-11-19 16:42:18] (XEN) [2015-11-19 16:42:18] Pagetable walk from 000000003e670340: (XEN) [2015-11-19 16:42:18] L4[0x000] = 00000008abd5f063 ffffffffffffffff (XEN) [2015-11-19 16:42:18] L3[0x000] = 000000002c87a063 ffffffffffffffff (XEN) [2015-11-19 16:42:18] L2[0x1f3] = 000000003e5ff063 ffffffffffffffff (XEN) [2015-11-19 16:42:18] L1[0x070] = 800000003e670163 ffffffffffffffff (XEN) [2015-11-19 16:42:18] (XEN) [2015-11-19 16:42:18] **************************************** (XEN) [2015-11-19 16:42:18] Panic on CPU 0: (XEN) [2015-11-19 16:42:18] FATAL PAGE FAULT (XEN) [2015-11-19 16:42:18] [error_code=0011] (XEN) [2015-11-19 16:42:18] Faulting linear address: 000000003e670340 (XEN) [2015-11-19 16:42:18] **************************************** (XEN) [2015-11-19 16:42:18] (XEN) [2015-11-19 16:42:18] Reboot in five seconds... (XEN) [2015-11-19 16:42:23] Resetting with ACPI MEMORY or I/O RESET_REG. The system _does_ continue the reboot, ... [ OK ] Reached target Shutdown. dracut Warning: Killing all remaining processes mdadm: stopped /dev/md2 mdadm: stopped /dev/md3 mdadm: stopped /dev/md4 mdadm: stopped /dev/md1 mdadm: stopped /dev/md0 Rebooting. [ 919.933417] reboot: Restarting system (XEN) [2015-11-19 16:42:17] Hardware Dom0 shutdown: rebooting machine (XEN) [2015-11-19 16:42:17] APIC error on CPU0: 40(00) (XEN) [2015-11-19 16:42:17] ----[ Xen-4.6.0_02-397 x86_64 debug=n Tainted: C ]---- (XEN) [2015-11-19 16:42:17] CPU: 0 (XEN) [2015-11-19 16:42:17] RIP: e008:[<000000003e670340>] 000000003e670340 (XEN) [2015-11-19 16:42:17] RFLAGS: 0000000000010202 CONTEXT: hypervisor (d0v0) (XEN) [2015-11-19 16:42:17] rax: 000000003e670340 rbx: 0000000000000000 rcx: 0000000000000000 (XEN) [2015-11-19 16:42:17] rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) [2015-11-19 16:42:17] rbp: 0000000000000000 rsp: ffff83002c8dfdc0 r8: 0000000000000000 (XEN) [2015-11-19 16:42:18] r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) [2015-11-19 16:42:18] r12: 0000000000000000 r13: 0000000000000cf9 r14: 0000000000000065 (XEN) [2015-11-19 16:42:18] r15: ffff830000000000 cr0: 0000000080050033 cr4: 00000000001526e0 (XEN) [2015-11-19 16:42:18] cr3: 00000008abd60000 cr2: 000000003e670340 (XEN) [2015-11-19 16:42:18] ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008 (XEN) [2015-11-19 16:42:18] Xen stack trace from rsp=ffff83002c8dfdc0: (XEN) [2015-11-19 16:42:18] 000000003efe42f6 0000000000000065 ffff82d08022b8ca efff000000000000 (XEN) [2015-11-19 16:42:18] ffff82d080269000 00000008abd60000 ffff82d08022bada 0000000675611000 (XEN) [2015-11-19 16:42:18] 0000000000000000 0000000000152660 000000000000e008 0000000000000292 (XEN) [2015-11-19 16:42:18] 0000000000000000 00000000fffffffe ffff82d0801822c8 0000000000000000 (XEN) [2015-11-19 16:42:18] 0000000000000010 000083002c8dfe98 0000000000000000 000000e126a3d666 (XEN) [2015-11-19 16:42:18] 0000000000000001 0000000000000001 ffff8308abd1f000 ffff8308abd1f158 Xen 4.6.0_02-397 (c/s ) EFI loader Using configuration file 'xen-4.6.0_02-397.cfg' vmlinuz-4.3.0-4.g734b32c-xen: 0x000000002bfdc000-0x000000002c4ddac0 initrd-4.3.0-4.g734b32c-xen: 0x000000002b081000-0x000000002bfdb194 0x0000:0x00:0x19.0x0: ROM: 0x10000 bytes at 0x329c6018 0x0000:0x04:0x00.0x0: ROM: 0x8000 bytes at 0x329b6018 0x0000:0x10:0x00.0x0: ROM: 0x10800 bytes at 0x3299e018 __ __ _ _ __ ___ ___ ____ _____ ___ _____ \ \/ /___ _ __ | || | / /_ / _ \ / _ \___ \ |___ // _ \___ | \ // _ \ '_ \ | || |_| '_ \| | | | | | | |__) |__ |_ \ (_) | / / / \ __/ | | | |__ _| (_) | |_| | | |_| / __/|__|__) \__, |/ / /_/\_\___|_| |_| |_|(_)___(_)___/___\___/_____| |____/ /_//_/ |_____| (XEN) Xen version 4.6.0_02-397 (abuild@suse.de) (gcc (SUSE Linux) 4.8.5) debug=n Tue Nov 10 22:16:24 UTC 2015 -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

PGNet Dev

22:03

New subject: [opensuse-virtual] Chainloaded Leap 42.1 Xen Dom0 on EFI hardware -- network offline, xen init failing: "Failed to start The Xen xenstore"

In addition to the earlier trace, I've also noticed wicked & xenstored "acting up", delaying boot / login sequence. Booting kernel-xen + Xen on OPensuse Leap, boot sequence stalls at login prompt ... [ OK ] Mounted mount xenstore file system. [ OK ] Listening on xenstore ro socket. [ OK ] Listening on xenstore socket. Starting The Xen xenstore... Welcome to openSUSE Leap 42.1 - Kernel 4.3.0-6.g6b3b033-xen (xvc0). xensvr login: root Password: for ~ 90+ seconds. non responsive. then proceeds login: timed out after 60 seconds Stopping Serial Getty on xvc0... [ OK ] Stopped Serial Getty on xvc0. Starting Serial Getty on xvc0... [ OK ] Started Serial Getty on xvc0. Welcome to openSUSE Leap 42.1 - Kernel 4.3.0-6.g6b3b033-xen (xvc0). and reoffers login xensvr login: root Password: which now completes immediately Last login: Thu Nov 19 14:25:04 on xvc0 Have a lot of fun... $ once logged in, systemd-analyze blame 2min 25.819s wicked.service 55.550s xenstored.service 1.784s lvm2-pvscan@9:1.service 1.367s dev-mapper-VG0\x2dROOT.device 975ms lvm2-pvscan@9:3.service ... So wicked and xenstore are 'culprits'. And, I also see xl list Name ID Mem VCPUs State Time(s) (null) 0 2988 1 r----- 20.9 Now ... the why? There's too many moving parts :-/ Is this a wicked, xen, or other problem? -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

PGNet Dev

20 Nov 20 Nov

00:49

New subject: [SOLVED ... mostly] Re: [opensuse-virtual] Chainloaded Leap 42.1 Xen Dom0 on EFI hardware -- network offline, xen init failing: "Failed to start The Xen xenstore"

Prior to upgrade, 'nfsserver' was up and running. After upgrade, I found that systemctl status nfsserver reports 'dead/inactive'. And systemctl enable nfsserver systemctl start nfsserver will not start under any circumstance. Exec'ing a forced reinstall zypper -n install --force \ rpcbind \ nfs-client \ nfs-kernel-server \ nfsidmap \ quota \ quota-nfs \ ypbind then shutdown -r now systemctl enable nfsserver systemctl start nfsserver now works. After another shutdown -r now Now, xen server boot completes, Dom0 & DomU are up xl list Name ID Mem VCPUs State Time(s) Domain-0 0 2987 1 r----- 16.5 template 1 1024 1 -b---- 3.6 & sshd is up & accessible (bug: https://bugzilla.opensuse.org/show_bug.cgi?id=955899) Although 'wicked' and 'xenstored' are both still slow starters: systemd-analyze blame 59.479s wicked.service 52.115s xenstored.service 13.928s xendomains.service 3.548s lvm2-pvscan@9:1.service 2.505s user@0.service 1.394s lvm2-pvscan@9:3.service ... -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

3086

Age (days ago)

3089

Last active (days ago)

List overview

Download

11 comments

2 participants

participants (2)

Olaf Hering
PGNet Dev

[opensuse-virtual] Chainloaded Xen Dom0 on EFI hardware FAILs at grub config generation after upgrade from Opensuse 13.2 -> Leap 42.1. Has the Xen-on-EFI prep process changed?

tags

participants (2)