...

...

On 07.10.14 at 08:27, wrote: Jan Beulich schreef op 6-10-2014 om 17:55: Whether the issue belongs here or on xen-devel (or xen-users) depends on what you expect, and by how much you're willing to help narrowing this down: From upstream's perspective, openSUSE and the Xen version it uses are of course only of secondary interest. It would be most interesting to know whether the -unstable tree (soon to become 4.5-rc) also exhibits the problem. The next best thing to try would be plain upstream 4.3.3 (or 4.4.1). No problem, tell me what is the fastest way for you towards on solution and I try to implement it. which ( if any ) repository it is on, or I should go back and compile the xen from source again.

Yes, of course from source. I don't think we have packaged up unstable yet (you might be able to pick something 4.4.1-ish from Factory). And in any event you need to be prepared that there's not going to be xend/xm anymore (i.e. other things may need updating too unless you're already using xl). Meanwhile I figured out that the issue is with NMI related state. Which raises the question: Are you also using PCI device pass- through? Is your system seeing NMIs for other reasons, or are you using the "watchdog" command line options? And in general I'm rather puzzled by guest state - you must be running some 16-bit OS inside KVM, which is quite unusual these days.

I do not know if it worked at all on opensuse 13.1

Then at least can you tell us which was the most recent version that worked for you without you adding additional patches and recompiling? Jan -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

...

...

On 07.10.14 at 09:56, wrote: 16 bit is correct, It is an appliance with 4 kvm guests. 2 times a sles10sp4 and 2 proprietary OS 16 bit. No watchdog options, and no pci-passthrough for this guest.

So do you know details of this proprietary OS? Is it making use of NMIs internally? I'm simply trying to understand where that NMI is having its origin... If you don't know, then maybe at least you can tell us whether the observed crash happens randomly or always at a deterministic point in the lifetime of the guest (the latter would hint towards guest-internal NMIs). Jan -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

...

...

On 07.10.14 at 15:24, wrote: Jan Beulich schreef op 7-10-2014 om 10:09: If you don't know, then maybe at least you can tell us whether the observed crash happens randomly or always at a deterministic point in the lifetime of the guest (the latter would hint towards guest-internal NMIs). It looks random to me, The time the guest lives is certainly not always the same. Sometimes it crashes after a few minutes sometimes it take 15 minutes, but rarely longer then that.

Yet the dumped state is roughly the same each time? I.e. the death of the guest is always upon an NMI injection going wrong? Anyway, I suppose you saw the mail I sent to VMX maintainer (and xen-devel) to understand where to place the apparently missing code. Jan -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

Oh, one more thing: Did you check hypervisor and kernel logs for instances of NMIs being reported there? What does /proc/interrupts in Dom0 say regarding the NMI count? CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 1: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi i8042 8: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi rtc0 9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi acpi 12: 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi i8042 16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi uhci_hcd:usb3, uhci_hcd:usb8 18: 278304 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi ehci_hcd:usb1, uhci_hcd:usb7, ata_piix, i801_smbus 19: 18301 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi uhci_hcd:usb6, ata_piix 21: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi uhci_hcd:usb4 23: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi ehci_hcd:usb2, uhci_hcd:usb5 64: 4438826 72423 4774 3459 3577 154438 4420 908388 85001 109580 28779 2863 51705 65838 30682 9314 3450 34471 113471 911724 12050 7378 72242 364746 Dynamic-percpu timer 65: 95232 45696 10328 3665 3353 2947 3392 42109 3107 8802 9726 2941 3059 6280 6341 14127 2936 3752 3766 36559 3130 3636 10725 27764 Dynamic-percpu ipi 66: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi pcpu 70: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi console 71: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi mce 72: 17 0 52 0 0 0 0 480 0 0 0 0 815 0 0 81 10 0 0 0 0 0 0 765 Phys-fasteoi ioat-msix 73: 782 0 0 52 0 0 0 0 480 0 0 0 0 815 81 0 0 10 0 0 0 0 0 0 Phys-fasteoi ioat-msix 74: 17 765 0 0 52 0 0 0 0 480 0 0 0 0 815 0 81 0 10 0 0 0 0 0 Phys-fasteoi ioat-msix 75: 17 0 765 0 0 52 0 0 0 0 480 0 0 0 0 815 0 81 0 10 0 0 0 0 Phys-fasteoi ioat-msix 76: 17 0 0 765 0 0 52 0 0 0 0 480 0 0 0 0 815 0 81 0 10 0 0 0 Phys-fasteoi ioat-msix 77: 17 0 0 0 765 0 0 52 0 0 0 0 480 0 0 0 0 825 0 81 0 0 0 0 Phys-fasteoi ioat-msix 78: 17 0 0 0 0 765 0 0 52 0 0 0 0 480 0 0 0 0 825 0 81 0 0 0 Phys-fasteoi ioat-msix 79: 17 0 0 0 0 0 765 0 0 52 0 0 0 0 480 0 0 0 0 825 0 81 0 0 Phys-fasteoi ioat-msix 80: 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 120 3248 0 1943 0 Phys-fasteoi eth0-rx-0 81: 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 Phys-fasteoi eth0-tx-0 82: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi eth0 83: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11782 0 0 0 1333 18877 Phys-fasteoi eth1-rx-0 84: 8354 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1550 11040 0 Phys-fasteoi eth1-tx-0 85: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi eth1 86: 196 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1422 0 Dynamic-fasteoi evtchn:xenstored[1691] 87: 52 0 0 223 0 0 0 0 0 0 0 0 0 0 0 908 0 0 0 0 0 556 0 863 Dynamic-fasteoi xenbus 88: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenstored[1691] 89: 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenstored[1691] 90: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenconsoled[1709] 91: 2454318 0 0 0 0 0 0 1080708 0 0 0 0 0 0 0 0 0 0 0 923076 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[4902] 92: 637843 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[4902] 93: 984134 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[4902] 94: 332001 0 0 0 0 0 0 0 0 0 51506 0 0 0 0 0 0 0 0 0 0 0 126405 0 Dynamic-fasteoi evtchn:qemu-system-i38[4902] 95: 39 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[4902] 96: 331 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenstored[1691] 97: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenconsoled[1709] 98: 483057 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5541] 99: 10506 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5541] 100: 135 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5541] 101: 5558 0 0 0 0 0 0 7 0 0 0 0 0 0 0 7899 0 0 0 1012 0 0 0 0 Dynamic-fasteoi blkif-backend 102: 26 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5541] 103: 2 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 717 0 0 0 204 0 0 Dynamic-fasteoi vif2.0 104: 313 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenstored[1691] 105: 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenconsoled[1709] 106: 3105 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5826] 107: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5826] 108: 3706 0 193 0 0 0 0 0 0 0 0 0 0 0 2487 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi blkif-backend 109: 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5826] 110: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi vif3.0 111: 324 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenstored[1691] 112: 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenconsoled[1709] 113: 3107 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5923] 114: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5923] 115: 2404 0 0 0 0 0 0 0 0 4514 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi blkif-backend 116: 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5923] 117: 2 0 0 0 0 0 0 0 0 0 0 243 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi vif4.0 NMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Non-maskable interrupts IWI: 471 498 126 48 82 122 47 50 110 501 129 149 160 52 82 295 50 145 173 121 55 62 51 366 IRQ work interrupts RES: 94089 39305 4443 2220 2213 1611 2098 36390 1739 2305 6399 1588 1304 1343 1911 1755 1762 2543 2171 31788 1621 1794 9668 25742 Rescheduling interrupts CAL: 697 5928 5776 1413 1068 1225 1257 5677 1271 6006 3210 1212 1604 4896 4361 12079 1131 1079 1432 4671 1465 1811 1020 1672 Function call interrupts LCK: 49 0 1 4 0 3 6 6 6 0 0 0 1 2 9 4 2 0 5 0 4 0 3 3 Spinlock wakeups MCE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Machine check polls

Jan

-- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

...

...

On 07.10.14 at 17:04, wrote: I installed xen-4.5.29823-1.xen_unstable.1.x86_64 from the http://download.opensuse.org/repositories/home:/olh:/xen-unstable/openSUSE_F... ctory/

After about 45 minutes the guests crashed again.

Good (in a sense). In which case we really should continue this on the thread on xen-devel. Could you repost this reply of yours as a reply there? Jan -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org

On Tue, Oct 07, Jan Beulich wrote:

Yes, of course from source. I don't think we have packaged up unstable yet (you might be able to pick something 4.4.1-ish from Factory). And in any event you need to be prepared that there's not going to be xend/xm anymore (i.e. other things may need updating too unless you're already using xl).

I maintain a package from xen unstable sources for some dists: zypper -v -v ar -c -f -K \ http://download.opensuse.org/repositories/home:/olh:/xen-unstable/openSUSE_1... \ remote-home_olh_xen_unstable Hopefully that works also for the OP. Olaf -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org