http://bugzilla.suse.com/show_bug.cgi?id=919154 grant k changed: What |Removed |Added ---------------------------------------------------------------------------- URL| |http://lists.opensuse.org/o | |pensuse-kernel/2015-02/msg0 | |0017.html Found By|--- |Community User -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 Charles Arnold changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jbeulich@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 grant k changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(grantksupport@ope | |ramail.com) | --- Comment #2 from grant k --- Created attachment 624675 --> http://bugzilla.suse.com/attachment.cgi?id=624675&action=edit output of `journalctl -b-1` from failed guest boot

Please attach (don't inline) the full log such that we have an indication as to what is going wrong, or at the very least where guest booting actually stops.

as requested -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 --- Comment #4 from grant k --- (In reply to Jan Beulich from comment #3)

So the only failures I seem to find are NFS related. Nothing Xen specific. Did you check that the guest's networking in general is okay?

Booting to kernel 3.18, the guest boots without error. Networking is fine -- both inbound and outbound. NFS is fully function. This has been a working DomU for a long time. Booting to kernel 3.19, the guest is no longer accessible via network. I can't get to it even via the xen console; I can't test any function. My only option is to reboot to a different working kernel. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(jbeulich@suse.com | |) | --- Comment #6 from Jan Beulich --- Any diagnostic information on the state of networking in the guest would help, including frontend and backend state (in xenstore). Without that I'm afraid a general kernel/NFS person may need to look at this first, as so far there's no indication of any Xen specific issue here. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 --- Comment #8 from grant k --- I've reproduced the problem on a local Dom0/DomU. with Dom0 lsb_release -rd Description: openSUSE 13.2 (Harlequin) (x86_64) Release: 13.2 uname -rm 3.19.0-4.g7f0e735-xen x86_64 rpm -qa | grep -i ^xen xen-4.5.0_03-350.7.x86_64 xen-devel-4.5.0_03-350.7.x86_64 xen-libs-4.5.0_03-350.7.x86_64 xen-tools-4.5.0_03-350.7.x86_64 DomU lsb_release -rd Description: openSUSE 13.2 (Harlequin) (x86_64) Release: 13.2 uname -rm 3.19.0-4.g7f0e735-xen x86_64 grep -i ^xen | sort xen-libs-4.5.0_03-350.7.x86_64 xen-tools-domU-4.5.0_03-350.7.x86_64 @ Dom0 xenstore-ls -------------------------------- tool = "" xenstored = "" local = "" domain = "" 0 = "" domid = "0" name = "Domain-0" device-model = "" 0 = "" state = "running" 1 = "" state = "running" memory = "" target = "1218048" static-max = "1310720" freemem-slack = "1218832" libxl = "" disable_udev = "1" backend = "" vbd = "" 1 = "" 51712 = "" frontend = "/local/domain/1/device/vbd/51712" params = "/dev/VG0/test_boot" script = "/etc/xen/scripts/block" physical-device = "fe:e" frontend-id = "1" online = "1" removable = "0" bootable = "1" state = "4" dev = "xvda" type = "phy" mode = "w" device-type = "disk" discard-enable = "1" max-ring-page-order = "0" feature-max-indirect-segments = "64" feature-flush-cache = "1" feature-barrier = "1" feature-discard = "0" sectors = "524288" info = "0" sector-size = "512" physical-sector-size = "512" 51728 = "" frontend = "/local/domain/1/device/vbd/51728" params = "/dev/VG0/test_swap" script = "/etc/xen/scripts/block" physical-device = "fe:10" frontend-id = "1" online = "1" removable = "0" bootable = "1" state = "4" dev = "xvdb" type = "phy" mode = "w" device-type = "disk" discard-enable = "1" max-ring-page-order = "0" feature-max-indirect-segments = "64" feature-flush-cache = "1" feature-barrier = "1" feature-discard = "0" sectors = "1048576" info = "0" sector-size = "512" physical-sector-size = "512" 51744 = "" frontend = "/local/domain/1/device/vbd/51744" params = "/dev/VG0/test_root" script = "/etc/xen/scripts/block" physical-device = "fe:f" frontend-id = "1" online = "1" removable = "0" bootable = "1" state = "4" dev = "xvdc" type = "phy" mode = "w" device-type = "disk" discard-enable = "1" max-ring-page-order = "0" feature-max-indirect-segments = "64" feature-flush-cache = "1" feature-barrier = "1" feature-discard = "0" sectors = "10485760" info = "0" sector-size = "512" physical-sector-size = "512" vfb = "" 1 = "" 0 = "" frontend = "/local/domain/1/device/vfb/0" frontend-id = "1" online = "1" state = "4" vnc = "1" vnclisten = "127.0.0.1" vncdisplay = "99" vncunused = "1" sdl = "0" opengl = "0" feature-resize = "1" hotplug-status = "connected" request-update = "1" vkbd = "" 1 = "" 0 = "" frontend = "/local/domain/1/device/vkbd/0" frontend-id = "1" online = "1" state = "4" feature-abs-pointer = "1" hotplug-status = "connected" console = "" 1 = "" 0 = "" frontend = "/local/domain/1/console" frontend-id = "1" online = "1" state = "1" protocol = "vt100" vif = "" 1 = "" 0 = "" frontend = "/local/domain/1/device/vif/0" frontend-id = "1" online = "1" state = "4" script = "/etc/xen/scripts/vif-bridge" vifname = "vifT.int" mac = "00:16:3e:01:01:01" bridge = "brT" handle = "0" type = "vif" feature-sg = "1" feature-gso-tcpv4 = "1" feature-gso-tcpv6 = "1" feature-ipv6-csum-offload = "1" feature-rx-copy = "1" feature-rx-flip = "0" hotplug-status = "connected" 1 = "" vm = "/vm/dc30e3c1-fdfd-49d8-8a16-0629688d4472" name = "template" cpu = "" 0 = "" availability = "online" 1 = "" availability = "online" memory = "" static-max = "1048576" target = "1048576" videoram = "0" device = "" suspend = "" event-channel = "9" vbd = "" 51712 = "" backend = "/local/domain/0/backend/vbd/1/51712" backend-id = "0" state = "4" virtual-device = "51712" device-type = "disk" protocol = "x86_64-abi" ring-ref = "8" event-channel = "12" 51728 = "" backend = "/local/domain/0/backend/vbd/1/51728" backend-id = "0" state = "4" virtual-device = "51728" device-type = "disk" protocol = "x86_64-abi" ring-ref = "9" event-channel = "13" 51744 = "" backend = "/local/domain/0/backend/vbd/1/51744" backend-id = "0" state = "4" virtual-device = "51744" device-type = "disk" protocol = "x86_64-abi" ring-ref = "10" event-channel = "14" vfb = "" 0 = "" backend = "/local/domain/0/backend/vfb/1/0" backend-id = "0" state = "4" page-ref = "1669939" event-channel = "10" protocol = "x86_64-abi" feature-update = "1" vkbd = "" 0 = "" backend = "/local/domain/0/backend/vkbd/1/0" backend-id = "0" state = "4" request-abs-pointer = "1" page-ref = "1669930" event-channel = "11" vif = "" 0 = "" backend = "/local/domain/0/backend/vif/1/0" backend-id = "0" state = "4" handle = "0" mac = "00:16:3e:01:01:01" tx-ring-ref = "768" rx-ring-ref = "769" event-channel = "15" request-rx-copy = "1" feature-rx-notify = "1" feature-no-csum-offload = "0" feature-ipv6-csum-offload = "1" feature-sg = "1" feature-gso-tcpv4 = "1" feature-gso-tcpv6 = "1" control = "" shutdown = "" platform-feature-multiprocessor-suspend = "1" platform-feature-xs_reset_watches = "1" data = "" domid = "1" store = "" port = "1" ring-ref = "1417603" console = "" backend = "/local/domain/0/backend/console/1/0" backend-id = "0" limit = "1048576" type = "xenconsoled" output = "pty" tty = "/dev/pts/0" port = "2" ring-ref = "1417602" vnc-listen = "127.0.0.1" vnc-port = "5999" image = "" device-model-pid = "3004" vm = "" dc30e3c1-fdfd-49d8-8a16-0629688d4472 = "" name = "template" uuid = "dc30e3c1-fdfd-49d8-8a16-0629688d4472" image = "" ostype = "linux" kernel = "/var/run/xen/bootloader.1.d/boot_kernel.7YOrHZ" ramdisk = "/var/run/xen/bootloader.1.d/boot_ramdisk.88KedB" cmdline = "root=/dev/xvdc1 resume=/dev/xvdb1 kbdtype=us headless text quiet nofb selinux=0 apparmor-0 edd=off splash=silent noshell showopts ide0=noprobe ide1=noprobe hda=noprobe root=/dev/xvdc1 textmode=1 xencons=xvc0 noirqdebu\..." start_time = "1425314623.93" libxl = "" 1 = "" dm-version = "qemu_xen" -------------------------------- @ DomU dmesg -------------------------------- dmesg [ 0.000000] BRK [0x00c34000, 0x00c34fff] PUD [ 0.000000] BRK [0x00c35000, 0x00c35fff] PMD [ 0.000000] BRK [0x00c36000, 0x00c42fff] PTE [ 0.000000] Initializing cgroup subsys cpuset [ 0.000000] Initializing cgroup subsys cpu [ 0.000000] Initializing cgroup subsys cpuacct [ 0.000000] Linux version 3.19.0-4.g7f0e735-xen (geeko@buildhost) (gcc version 4.8.3 20141208 [gcc-4_8-branch revision 218481] (SUSE Linux) ) #1 SMP Sun Mar 1 22:25:41 UTC 2015 (7f0e735) [ 0.000000] Command line: root=/dev/xvdc1 resume=/dev/xvdb1 kbdtype=us headless text quiet nofb selinux=0 apparmor-0 edd=off splash=silent noshell showopts ide0=noprobe ide1=noprobe hda=noprobe root=/dev/xvdc1 textmode=1 xencons=xvc0 noirqdebug elevator=noop console=hvc0 [ 0.000000] e820: Xen-provided physical RAM map: [ 0.000000] Xen: [mem 0x0000000000000000-0x000000003fffffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] e820: last_pfn = 0x40000 max_arch_pfn = 0x80000000 [ 0.000000] init_memory_mapping: [mem 0x3fe00000-0x3fffffff] [ 0.000000] [mem 0x3fe00000-0x3fffffff] page 4k [ 0.000000] BRK [0x00c43000, 0x00c43fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x20000000-0x3fdfffff] [ 0.000000] [mem 0x20000000-0x3fdfffff] page 4k [ 0.000000] BRK [0x00c44000, 0x00c44fff] PGTABLE [ 0.000000] BRK [0x00c45000, 0x00c45fff] PGTABLE [ 0.000000] BRK [0x00c46000, 0x00c46fff] PGTABLE [ 0.000000] BRK [0x00c47000, 0x00c47fff] PGTABLE [ 0.000000] BRK [0x00c48000, 0x00c48fff] PGTABLE [ 0.000000] init_memory_mapping: [mem 0x00000000-0x1fffffff] [ 0.000000] [mem 0x00000000-0x1fffffff] page 4k [ 0.000000] RAMDISK: [mem 0x0105d000-0x01605fff] [ 0.000000] ACPI in unprivileged domain disabled [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x00000000-0x00ffffff] [ 0.000000] DMA32 [mem 0x01000000-0x3fffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x00000000-0x3fffffff] [ 0.000000] Initmem setup node 0 [mem 0x00000000-0x3fffffff] [ 0.000000] On node 0 totalpages: 262144 [ 0.000000] free_area_init_node: node 0, pgdat ffffffff80a9b440, node_mem_map ffff88003ee13000 [ 0.000000] DMA zone: 64 pages used for memmap [ 0.000000] DMA zone: 0 pages reserved [ 0.000000] DMA zone: 4096 pages, LIFO batch:0 [ 0.000000] DMA32 zone: 4032 pages used for memmap [ 0.000000] DMA32 zone: 258048 pages, LIFO batch:31 [ 0.000000] setup_percpu: NR_CPUS:512 nr_cpumask_bits:512 nr_cpu_ids:2 nr_node_ids:1 [ 0.000000] PERCPU: Embedded 22 pages/cpu @ffff88003ea00000 s51328 r8192 d30592 u1048576 [ 0.000000] pcpu-alloc: s51328 r8192 d30592 u1048576 alloc=1*2097152 [ 0.000000] pcpu-alloc: [0] 0 1 [ 0.000000] Swapping MFNs for PFN ac6 and 3ea08 (MFN 159d84 and 195582) [ 0.000000] Built 1 zonelists in Zone order, mobility grouping on. Total pages: 258048 [ 0.000000] Kernel command line: root=/dev/xvdc1 resume=/dev/xvdb1 kbdtype=us headless text quiet nofb selinux=0 apparmor-0 edd=off splash=silent noshell showopts ide0=noprobe ide1=noprobe hda=noprobe root=/dev/xvdc1 textmode=1 xencons=xvc0 noirqdebug elevator=noop console=hvc0 [ 0.000000] IRQ lockup detection disabled [ 0.000000] PID hash table entries: 4096 (order: 3, 32768 bytes) [ 0.000000] Dentry cache hash table entries: 131072 (order: 8, 1048576 bytes) [ 0.000000] Inode-cache hash table entries: 65536 (order: 7, 524288 bytes) [ 0.000000] Software IO TLB disabled [ 0.000000] Memory: 1007988K/1048576K available (5839K kernel code, 776K rwdata, 4300K rodata, 716K init, 748K bss, 40588K reserved, 0K cma-reserved) [ 0.000000] Hierarchical RCU implementation. [ 0.000000] RCU dyntick-idle grace-period acceleration is enabled. [ 0.000000] RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=2. [ 0.000000] RCU: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=2 [ 0.000000] nr_pirqs: 256 [ 0.000000] NR_IRQS:67328 nr_irqs:2624 16 [ 0.000000] Offload RCU callbacks from all CPUs [ 0.000000] Offload RCU callbacks from CPUs: 0-1. [ 0.000000] Xen reported: 2806.456 MHz processor. [ 0.000000] Console: colour dummy device 80x25 [ 0.000000] console [xvc0] enabled [ 0.080002] Calibrating delay using timer specific routine.. 5618.98 BogoMIPS (lpj=11237975) [ 0.080007] pid_max: default: 32768 minimum: 301 [ 0.080041] Security Framework initialized [ 0.080051] AppArmor: AppArmor initialized [ 0.080066] Mount-cache hash table entries: 2048 (order: 2, 16384 bytes) [ 0.080068] Mountpoint-cache hash table entries: 2048 (order: 2, 16384 bytes) [ 0.080251] Initializing cgroup subsys memory [ 0.080257] Initializing cgroup subsys devices [ 0.080260] Initializing cgroup subsys freezer [ 0.080263] Initializing cgroup subsys net_cls [ 0.080265] Initializing cgroup subsys blkio [ 0.080268] Initializing cgroup subsys perf_event [ 0.080271] Initializing cgroup subsys net_prio [ 0.080338] Last level iTLB entries: 4KB 512, 2MB 16, 4MB 8 Last level dTLB entries: 4KB 512, 2MB 128, 4MB 64, 1GB 0 [ 0.111418] ftrace: allocating 21933 entries in 86 pages [ 0.114787] ------------[ cut here ]------------ [ 0.114793] WARNING: CPU: 0 PID: 0 at ../kernel/trace/ftrace.c:1939 ftrace_bug+0x2a6/0x350() [ 0.114794] Modules linked in: [ 0.114798] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.19.0-4.g7f0e735-xen #1 [ 0.114799] Hardware name: Xen 4.5.0_03-350 PV guest [ 0.114801] 0000000000000000 ffffffff80019715 ffffffff805b9175 0000000000000000 [ 0.114803] ffffffff800488e1 0000000000000000 ffffffff80012000 ffff88003e540000 [ 0.114805] 0000000000000000 ffff88003e540000 ffffffff800eb6a6 ffffffff80b2ad20 [ 0.114808] Call Trace: [ 0.114820] [<ffffffff800171ea>] dump_trace+0x7a/0x1f0 [ 0.114824] [<ffffffff800173f3>] show_stack_log_lvl+0x93/0x170 [ 0.114828] [<ffffffff80019731>] show_stack+0x21/0x50 [ 0.114833] [<ffffffff805b9175>] dump_stack+0x40/0x50 [ 0.114837] [<ffffffff800488e1>] warn_slowpath_common+0x81/0xb0 [ 0.114841] [<ffffffff800eb6a6>] ftrace_bug+0x2a6/0x350 [ 0.114847] [<ffffffff800ebbc4>] ftrace_process_locs+0x3b4/0x690 [ 0.114853] [<ffffffff80adc352>] ftrace_init+0xb3/0x156 [ 0.114857] [<ffffffff80acbca1>] start_kernel+0x491/0x4a1 [ 0.114864] ---[ end trace e3751dd8ec96d5ab ]--- [ 0.114865] ftrace faulted on writing [<ffffffff80012000>] run_init_process+0x0/0x30 [ 0.114869] ftrace record flags: 0 [ 0.114870] (0) expected tramp: ffffffff805c08b0 [ 0.115404] Brought up 1 CPUs [ 0.115516] devtmpfs: initialized [ 0.115880] pinctrl core: initialized pinctrl subsystem [ 0.135726] RTC time: 165:165:165, date: 165/165/65 [ 0.135859] NET: Registered protocol family 16 [ 0.136156] SMP alternatives: switching to SMP code [ 0.168255] Brought up 2 CPUs [ 0.168522] PCI: Fatal: No config space access function found [ 0.168529] PCI: setting up Xen PCI frontend stub [ 0.184129] ACPI: Interpreter disabled. [ 0.184129] vgaarb: loaded [ 0.184129] suspend: event channel 9 [ 0.184129] xen_mem: Initialising balloon driver. [ 0.184129] Unable to read sysrq code in control/sysrq [ 0.184129] SCSI subsystem initialized [ 0.184129] libata version 3.00 loaded. [ 0.184129] PCI: System does not support PCI [ 0.184129] PCI: System does not support PCI [ 0.184129] NetLabel: Initializing [ 0.184129] NetLabel: domain hash size = 128 [ 0.184130] NetLabel: protocols = UNLABELED CIPSOv4 [ 0.184143] NetLabel: unlabeled traffic allowed by default [ 0.184205] Switched to clocksource xen [ 0.191859] AppArmor: AppArmor Filesystem Enabled [ 0.191937] pnp: PnP ACPI: disabled [ 0.196151] NET: Registered protocol family 2 [ 0.196382] TCP established hash table entries: 8192 (order: 4, 65536 bytes) [ 0.196413] TCP bind hash table entries: 8192 (order: 5, 131072 bytes) [ 0.196465] TCP: Hash tables configured (established 8192 bind 8192) [ 0.196515] TCP: reno registered [ 0.196519] UDP hash table entries: 512 (order: 3, 32768 bytes) [ 0.196534] UDP-Lite hash table entries: 512 (order: 3, 32768 bytes) [ 0.196609] NET: Registered protocol family 1 [ 0.196622] PCI: CLS 64 bytes [ 0.196705] Unpacking initramfs... [ 0.815728] Freeing initrd memory: 5796K (ffff88000105d000 - ffff880001606000) [ 0.818697] futex hash table entries: 512 (order: 3, 32768 bytes) [ 0.818716] Initialise system trusted keyring [ 0.818751] audit: initializing netlink subsys (disabled) [ 0.818774] audit: type=2000 audit(1425314625.604:1): initialized [ 0.819088] zpool: loaded [ 0.819092] zbud: loaded [ 0.819256] VFS: Disk quotas dquot_6.5.2 [ 0.819278] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.819551] Key type big_key registered [ 0.820010] Key type asymmetric registered [ 0.820014] Asymmetric key parser 'x509' registered [ 0.820039] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 252) [ 0.820099] io scheduler noop registered (default) [ 0.820103] io scheduler deadline registered [ 0.820120] io scheduler cfq registered [ 0.820188] pci_hotplug: PCI Hot Plug PCI Core version: 0.5 [ 0.820209] pciehp: PCI Express Hot Plug Controller Driver version: 0.4 [ 0.820328] Non-volatile memory driver v1.3 [ 0.820332] Linux agpgart interface v0.103 [ 0.820460] Xen virtual console successfully installed as xvc0 [ 0.823782] Console: switching to colour frame buffer device 100x37 [ 0.828987] input: Xen Virtual Keyboard as /devices/virtual/input/input0 [ 0.829053] input: Xen Virtual Pointer as /devices/virtual/input/input1 [ 0.832403] i8042: PNP: No PS/2 controller found. Probing ports directly. [ 1.832877] i8042: No controller found [ 1.833052] mousedev: PS/2 mouse device common for all mice [ 1.833242] ledtrig-cpu: registered to indicate activity on CPUs [ 1.833258] hidraw: raw HID events driver (C) Jiri Kosina [ 1.833319] TCP: cubic registered [ 1.833354] NET: Registered protocol family 10 [ 1.833531] mce: Unable to init device /dev/mcelog (rc: -5) [ 1.833787] Loading compiled-in X.509 certificates [ 1.834099] Loaded X.509 cert 'Kernel OBS Project: 1fb41512acbc8eebdf828d877e4367bf6c719af3' [ 1.834116] registered taskstats version 1 [ 1.834141] AppArmor: AppArmor sha1 policy hashing enabled [ 1.834223] Magic number: 1:252:3141 [ 1.932086] xenbus_probe: Device with no driver: device/vbd/51712 [ 1.932095] xenbus_probe: Device with no driver: device/vbd/51728 [ 1.932100] xenbus_probe: Device with no driver: device/vbd/51744 [ 1.932104] xenbus_probe: Device with no driver: device/vif/0 [ 1.932121] ../drivers/rtc/hctosys.c: unable to open rtc device (rtc0) [ 1.933449] Freeing unused kernel memory: 716K (ffff880000abe000 - ffff880000b71000) [ 1.933458] Write protecting the kernel read-only data: 10152k [ 1.947981] random: systemd urandom read with 21 bits of entropy available [ 2.068883] systemd-journald[97]: Failed to set file attributes: Inappropriate ioctl for device [ 2.365563] Initialising virtual ethernet driver. [ 2.367057] blkfront: device/vbd/51712: ring-pages=1 nr-ents=32 segs-per-req=64 [ 2.372571] blkfront: device/vbd/51728: ring-pages=1 nr-ents=32 segs-per-req=64 [ 2.377146] blkfront: device/vbd/51744: ring-pages=1 nr-ents=32 segs-per-req=64 [ 2.381806] xen-vbd: registered block device major 202 [ 2.381836] blkfront: xvda: flush diskcache: enabled [ 2.400228] xvda: xvda1 [ 2.400620] Setting capacity to 524288 [ 2.400625] xvda: detected capacity change from 0 to 268435456 [ 2.400977] blkfront: xvdb: flush diskcache: enabled [ 2.410490] xvdb: xvdb1 [ 2.412653] blkfront: xvdc: flush diskcache: enabled [ 2.427791] xvdc: xvdc1 [ 2.442712] Setting capacity to 1048576 [ 2.442729] xvdb: detected capacity change from 0 to 536870912 [ 2.443854] Setting capacity to 10485760 [ 2.443862] xvdc: detected capacity change from 0 to 5368709120 [ 2.880311] random: nonblocking pool is initialized [ 2.937095] EXT4-fs (xvdc1): mounted filesystem with ordered data mode. Opts: (null) [ 4.457219] device-mapper: uevent: version 1.0.3 [ 4.457383] device-mapper: ioctl: 4.29.0-ioctl (2014-10-28) initialised: dm-devel@redhat.com [ 5.306549] Program systemd-vconsol tried to access /dev/mem between 0->1. [ 5.603358] EXT4-fs (xvdc1): re-mounted. Opts: acl,user_xattr,barrier=1 [ 6.262139] Program systemd-vconsol tried to access /dev/mem between 0->1. [ 6.674617] Adding 523260k swap on /dev/xvdb1. Priority:-1 extents:1 across:523260k SSFS [ 6.697717] EXT4-fs (xvda1): mounted filesystem with ordered data mode. Opts: acl,user_xattr,barrier=1 [ 7.447261] NET: Registered protocol family 17 [ 7.708249] No iBFT detected. [ 12.532649] FS-Cache: Loaded [ 12.552821] RPC: Registered named UNIX socket transport module. [ 12.552831] RPC: Registered udp transport module. [ 12.552836] RPC: Registered tcp transport module. [ 12.552841] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 12.598385] FS-Cache: Netfs 'nfs' registered for caching [ 12.617909] Installing knfsd (copyright (C) 1996 okir@monad.swb.de). [ 12.673431] Key type dns_resolver registered [ 12.704211] NFS: Registering the id_resolver key type [ 12.704239] Key type id_resolver registered [ 12.704244] Key type id_legacy registered [ 13.073242] NFSD: starting 90-second grace period (net ffffffff80a8c980) [ 22.032179] nfs: server nfs.lan not responding, timed out ... [ 1609.906290] nfs: server nfs.lan not responding, timed out -------------------------------- -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 Neil Brown changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(nfbrown@suse.com) | --- Comment #10 from Neil Brown --- Looks more like a general networking problem than an NFS problem. I can see: Feb 26 08:34:27 domu01 kernel: Initialising virtual ethernet driver. and then Feb 26 08:34:37 domu01 wickedd[426]: eth0: Notified neighbours about IP address 10.1.10.1 Feb 26 08:34:40 domu01 wicked[428]: eth0 up so it appears that eth0 is configured as 10.1.10.1, but Feb 26 08:34:49 domu01 kernel: nfs: server nas01.lan not responding, timed out suggests that the server cannot be contacted, and

Booting to kernel 3.19, the guest is no longer accessible via network.

seems to confirm that. Has anything changed in the "Virtual ethernet driver" between 3.18 and 3.19?? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 --- Comment #12 from Jan Beulich --- (In reply to Neil F Brown from comment #10)

...

Booting to kernel 3.19, the guest is no longer accessible via network.

seems to confirm that. Has anything changed in the "Virtual ethernet driver" between 3.18 and 3.19??

Nothing at all (also not in the backend, for the avoidance of doubt). There was a change (only indirectly related to 3.19) in how stats are being collected (or more precisely the locking involved there), but an eventual problem introduced there should manifest quite obviously in e.g. soft lockup messages. And that change was also already carried through to our SLE12 kernel and hasn't caused any known problems there so far. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 grant k changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(jbeulich@suse.com | |) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 grant k changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED CC|grantksupport@operamail.com | Resolution|--- |WONTFIX --- Comment #15 from grant k --- (In reply to Jan Beulich from comment #14)

It's not clear to me why you set this to needinfo to me: The absence of the requested Dom0 kernel and hypervisor logs is quite obvious. And while it's simple here ("dmesg" and "xl dmesg" in Dom0 are all you should need), I also don't think bugzilla is the right forum for explaining how to obtain requested data - please use available information/resources on the internet to learn what it takes.

You asked a question, it wasn't clear, so I asked. When you failed to answer, I flagged it. Nobody's twisting your arm. If clarifying your own requests is too much of a hassle, consider different employment. Good luck figuring out your broken software on your own. When it boils up in your commercial product, I'm sure your attitude will help. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 James Fehlig changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|WONTFIX |--- --- Comment #17 from James Fehlig --- Seems "reopen" didn't work when adding the attachment. Reopening now. Let me know if anything else is needed. BTW, the domU booted fine after installing the latest kernel-xen from http://download.opensuse.org/repositories/Kernel:/stable/standard, but indeed networking has problems. The domU is configured with DHCP and on some boots I noticed it would get an IP addr, on other boots not. But even if it did get an IP addr, the network was not usable by the time I logged in.

From xenstore perspective, vif is connected and online (state 4).

-- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 James Fehlig changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mlatimer@suse.com --- Comment #19 from James Fehlig --- (In reply to James Fehlig from comment #18)

After rebooting the domU several times, the system is in a state where the domU will no longer boot. It stops at "A start job is running for /sysroot".

Perhaps just a red herring. The filesystem may have had some corruption. After running virt-rescue on the image, the domU boots again but still encounters the network problems. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 Thomas Blume changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |thomas.blume@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 --- Comment #22 from lynda t --- Created attachment 630792 --> http://bugzilla.suse.com/attachment.cgi?id=630792&action=edit serial console output booting to Xen HVM DomU with kernel 3.19 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 --- Comment #24 from Jan Beulich --- Created attachment 635405 --> http://bugzilla.suse.com/attachment.cgi?id=635405&action=edit tentative fix I identified upstream commit 6a6dc08ff6 as lacking a counterpart in xen3-patch-3.19. The (completely untested) patch here is the expected to be missing adjustment to our netfront incarnation. Could anyone give this a try? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 Jan Beulich changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(thomas.blume@suse | |.com) --- Comment #26 from Jan Beulich --- (In reply to Thomas Blume from comment #25)

Tried your patch with kernel version:

4.0.4-0.g4f5e0d5-xen

from factory. I still see the domU network getting disconnected. Just before it happens, I can see this in on the domU console:

[ 88.637324] vif vif-0 eth0: Too many frags

looks similar to the behaviour that is reported here:

http://lists.xen.org/archives/html/xen-devel/2013-03/msg00404.html

but I don't see a message that the device gets disabled. As reported before, the behaviour startet with kernel version 3.19.

Interesting - I don't think any of the logs we got to see for this issue had "Too many frags" in them. Are you sure you saw this also prior to applying that patch? I'm asking because it could well be that we have two distinct issues here. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 Thomas Blume changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(thomas.blume@suse | |.com) | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 http://bugzilla.suse.com/show_bug.cgi?id=919154#c29 boo35 boo35 <9b3e05a5@opayq.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |9b3e05a5@opayq.com --- Comment #29 from boo35 boo35 <9b3e05a5@opayq.com> --- I'm starting to hit this now - can't install up to date Opensuse guest kernels in online/hosted Xen VPS's. Looks like kernel 4.1 is close to upstream release; Probably Tumbleweed and 13.2 will soon offer it. Will 4.1 either as Xen host and/or guest fix this issue? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 http://bugzilla.suse.com/show_bug.cgi?id=919154#c31 --- Comment #31 from Jan Beulich --- Created attachment 646274 --> http://bugzilla.suse.com/attachment.cgi?id=646274&action=edit debugging patch This adds some debugging code as well as a simple (but not intended to be the final) solution to the backend, which for me fully eliminates all "Too many frags" messages. IOW this might help people here determine whether what they see is related to the "Too many frags" issue. A permanent solution will require quite a bit of code re-work, so will take some more time. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 http://bugzilla.suse.com/show_bug.cgi?id=919154#c33 --- Comment #33 from Thomas Blume --- (In reply to Jan Beulich from comment #30)

Now that I finally got around trying to repro this, there are two observations:

- I cannot reproduce any network hangs. Can anybody confirm these hangs still occur with kernel 4.2.x? If so, as detailed as possible a description under what kind of load they occur (ideally also enumerating cases where they don't occur) would be necessary. Without being able to repro the issue I don't see ways to reasonably debug this myself.

I couldn't reproduce the network hang anymore with: kernel-xen-4.2.0-2.1.gc4d41ea.x86_64

- I can trigger the "Too many frags" messages, but having looked through old logs I can chase them back to at least summer 2013, i.e. I'm not convinced that issue is really related to the one here. Still I am looking into how to eliminate those.

Yes, I can see them too: [ 51.113204] vif vif-0 eth0: Too many frags [ 51.155701] vif vif-0 eth0: Too many frags [ 51.678351] vif vif-0 eth0: Too many frags [ 51.774121] vif vif-0 eth0: Too many frags [ 52.388251] vif vif-0 eth0: Too many frags [ 53.849132] vif vif-0 eth0: Too many frags [ 53.850286] vif vif-0 eth0: Too many frags but apparently they don't have an impact on the network hang. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 http://bugzilla.suse.com/show_bug.cgi?id=919154#c35 --- Comment #35 from Jan Beulich --- (In reply to Thomas Blume from comment #34)

Are you still interested in the debugging output, considering the comment above?

No, that would be of interest only if hangs are still visible. Once I've got a patch ready to deal with the "Too many frags" thing I'd of course appreciate you or others to give this a try before I merge it into any git branch. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 http://bugzilla.suse.com/show_bug.cgi?id=919154#c37 --- Comment #37 from Thomas Blume --- (In reply to Jan Beulich from comment #36)

Created attachment 646780 [details] tentative fix

Please anyone interested give this a try.

I've tried the patch: xen107:~ # rpm -q --changelog kernel-xen | head * Fr Sep 11 2015 thomas.blume@suse.com - patches.xen/xen-bsc919153-tentative.patch: tentative fix for bsc#919154 and there is no more network hang. Still, on heavy network load, the serial console shows: -->-- xen107 login: Ä 22.397057Ü SFW2-INext-ACC-TCP IN=eth0 OUT= MAC=00:16:3e:54:89:bd:fe:ff:ff:ff:ff:ff:08:00 SRC=192.168.58.1 DST=192.168.58.107 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=60644 DF PROTO=TCP SPT=46117 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0 OPT (020405B40402080A00652F5A0000000001030307) Ä 32.843109Ü vif vif-0 eth0: Too many frags Ä 33.170557Ü vif vif-0 eth0: Too many frags Ä 33.204960Ü vif vif-0 eth0: Too many frags Ä 95.551688Ü vif vif-0 eth0: Too many frags Ä 97.864372Ü vif vif-0 eth0: Too many frags Ä 100.584173Ü vif vif-0 eth0: Too many frags Ä 100.651322Ü vif vif-0 eth0: Too many frags Ä 105.076181Ü vif vif-0 eth0: Too many frags Ä 105.085163Ü vif vif-0 eth0: Too many frags Ä 105.153476Ü vif vif-0 eth0: Too many frags Ä 105.326148Ü vif vif-0 eth0: Too many frags Ä 105.341968Ü vif vif-0 eth0: Too many frags Ä 105.371905Ü vif vif-0 eth0: Too many frags Ä 108.714202Ü vif vif-0 eth0: Too many frags Ä 108.802537Ü vif vif-0 eth0: Too many frags Ä 108.805242Ü vif vif-0 eth0: Too many frags Ä 118.147660Ü vif vif-0 eth0: Too many frags Ä 144.409872Ü vif vif-0 eth0: Too many frags Ä 144.461613Ü vif vif-0 eth0: Too many frags Ä 154.009564Ü vif vif-0 eth0: Too many frags Ä 154.034452Ü vif vif-0 eth0: Too many frags Ä 154.163568Ü vif vif-0 eth0: Too many frags Ä 154.341685Ü vif vif-0 eth0: Too many frags Ä 162.973648Ü vif vif-0 eth0: Too many frags Ä 167.532139Ü vif vif-0 eth0: Too many frags Ä 167.660511Ü vif vif-0 eth0: Too many frags Ä 167.763044Ü vif vif-0 eth0: Too many frags Ä 168.316072Ü vif vif-0 eth0: Too many frags Ä 170.331552Ü vif vif-0 eth0: Too many frags Ä 170.334015Ü vif vif-0 eth0: Too many frags Ä 170.815484Ü vif vif-0 eth0: Too many frags Ä 170.897829Ü vif vif-0 eth0: Too many frags Ä 170.941495Ü vif vif-0 eth0: Too many frags Ä 175.834838Ü net_ratelimit: 2 callbacks suppressed Ä 175.834849Ü vif vif-0 eth0: Too many frags Ä 175.945815Ü vif vif-0 eth0: Too many frags Ä 182.603568Ü vif vif-0 eth0: Too many frags Ä 202.746753Ü vif vif-0 eth0: Too many frags Ä 202.790950Ü vif vif-0 eth0: Too many frags Ä 202.828798Ü vif vif-0 eth0: Too many frags Ä 203.567032Ü vif vif-0 eth0: Too many frags Ä 203.753478Ü vif vif-0 eth0: Too many frags Ä 203.763813Ü vif vif-0 eth0: Too many frags Ä 203.938512Ü vif vif-0 eth0: Too many frags Ä 203.948050Ü vif vif-0 eth0: Too many frags Ä 205.098703Ü vif vif-0 eth0: Too many frags Ä 208.339193Ü net_ratelimit: 4 callbacks suppressed Ä 208.339210Ü vif vif-0 eth0: Too many frags Ä 208.545938Ü vif vif-0 eth0: Too many frags Ä 208.666983Ü vif vif-0 eth0: Too many frags Ä 208.765865Ü vif vif-0 eth0: Too many frags Ä 209.182452Ü vif vif-0 eth0: Too many frags Ä 209.236549Ü vif vif-0 eth0: Too many frags Ä 209.317220Ü vif vif-0 eth0: Too many frags Ä 209.727166Ü vif vif-0 eth0: Too many frags --<-- -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 http://bugzilla.suse.com/show_bug.cgi?id=919154#c39 --- Comment #39 from Thomas Blume --- (In reply to Jan Beulich from comment #38)

I would guess from what you provided that you updated the guest kernel only, while - the fix being to netback - it is the host kernel that needs updating.

Sorry, my test setup was trashed. But even after installing the kernel on both the guest and the host: -->-- linux-1yyf:~ # modinfo netbk filename: /lib/modules/3.16.7-27-xen/kernel/drivers/xen/netback/netbk.ko alias: xen-backend:vif license: Dual BSD/GPL srcversion: 236C86330FED3938FF28BB0 depends: xenbus_be intree: Y vermagic: 3.16.7-27-xen SMP mod_unload modversions Xen signer: home:tsaupe OBS Project sig_key: CA:A3:D5:88:38:85:EA:DF:01:2B:58:EA:D7:23:02:98:2F:41:1D:C8 sig_hashalgo: sha256 parm: queue_length:ulong parm: max_tx_slots:Maximum number of slots accepted in netfront TX requests (uint) parm: copy_skb:Copy data received from netfront without netloop (bool) parm: permute_returns:Randomly permute the order in which TX responses are sent to the frontend (bool) parm: groups:Specify the number of tasklet pairs/threads to use (uint) parm: tasklets:Use tasklets instead of kernel threads (invbool) parm: bind:Bind kernel threads to (v)CPUs (bool) --<-- I can see in the journal log of the guest: -->-- Sep 15 15:48:00 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:01 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:01 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:02 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:04 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:06 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:08 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:08 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:09 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:11 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:14 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:14 xen106 kernel: vif vif-0 eth0: Too many frags Sep 15 15:48:15 xen106 kernel: vif vif-0 eth0: Too many frags --<-- when retrieving a big file via http. What information would you need from guest and/or host? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=919154 http://bugzilla.suse.com/show_bug.cgi?id=919154#c41 --- Comment #41 from Thomas Blume --- (In reply to Jan Beulich from comment #40)

(In reply to Thomas Blume from comment #39)

...

But even after installing the kernel on both the guest and the host:

You quite clearly did not install a patched kernel in the host:

Sorry, I made a mistake building the patch. I've corrected it now: -->-- linux-1yyf:~ # modinfo netbk filename: /lib/modules/4.2.1-0.g9c7cacf-xen/kernel/drivers/xen/netback/netbk.ko alias: xen-backend:vif license: Dual BSD/GPL srcversion: 6F56DC12C8AC1A02648D9D0 depends: xenbus_be intree: Y vermagic: 4.2.1-0.g9c7cacf-xen SMP mod_unload modversions Xen parm: queue_length:ulong parm: max_tx_slots:Maximum number of slots accepted in netfront TX requests (uint) parm: copy_skb:Copy data received from netfront without netloop (bool) parm: always_coalesce:Always fully coalesce RX data (bint) parm: permute_returns:Randomly permute the order in which TX responses are sent to the frontend (bool) parm: groups:Specify the number of tasklet pairs/threads to use (uint) parm: tasklets:Use tasklets instead of kernel threads (invbool) parm: bind:Bind kernel threads to (v)CPUs (bool) --<-- and I can confirm that there are no more "Too many frags" visible. -- You are receiving this mail because: You are on the CC list for the bug.