[opensuse-virtual] Crash of guest with nested vmx with Unknown nested vmexit reason 80000021.
Hi, Recently I updated my openSuse box from 12.3 to 13.1. On this box I run xen with several guests. One of these guests is an appliance that has 4 kvm guests running. When I start this appliance with the nested vmx feature the appliance crashes either immediately or after a few minutes. This same guest was running without a problem on opensuse releases 11.4 until 12.3 ( remark that about 3 year ago some patches were written by tim degan to make this appliance work ), I am not 100% sure what the right list to post this is soI also posted this problem on the xen-devel list. dom0 opensuse 13.1 x86 kernel 3.11.10-21-xen xen: 4.3..2_01-21.1 - domu (HVM) - sles11sp2 - mem: 8 GB - vcpu: 4 - kvm with 4 guest any body any thought on this? BR, Jeroen ==== outup xl demsg (XEN) vvmx.c:2459:d5 Unknown nested vmexit reason 80000021. (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0). (XEN) ************* VMCS Area ************** (XEN) *** Guest State *** (XEN) CR0: actual=0x0000000080000033, shadow=0x0000000000000011, gh_mask=ffffffffffffffff (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=ffffffffffffffff (XEN) CR3: actual=0x00000000feffc000, target_count=0 (XEN) target0=0000000000000000, target1=0000000000000000 (XEN) target2=0000000000000000, target3=0000000000000000 (XEN) RSP = 0x0000000000000398 (0x0000000000000398) RIP = 0x000000000000045f (0x000000000000045f) (XEN) RFLAGS=0x0000000000000006 (0x0000000000000006) DR7 = 0x0000000000000400 (XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000 (XEN) CS: sel=0x9dcc, attr=0x0009b, limit=0x00000ecb, base=0x0000000000689920 (XEN) DS: sel=0x9db4, attr=0x00093, limit=0x000038b9, base=0x00000000006778b0 (XEN) SS: sel=0x9e7c, attr=0x00093, limit=0x00000401, base=0x00000000006ba960 (XEN) ES: sel=0x3ec8, attr=0x00093, limit=0x00000099, base=0x000000000073c608 (XEN) FS: sel=0x0000, attr=0x1c000, limit=0xffffffff, base=0x0000000000000000 (XEN) GS: sel=0x0000, attr=0x1c000, limit=0xffffffff, base=0x0000000000000000 (XEN) GDTR: limit=0x0000ffff, base=0x0000000001000000 (XEN) LDTR: sel=0x0820, attr=0x00082, limit=0x0000ffff, base=0x0000000001010400 (XEN) IDTR: limit=0x000003ff, base=0x0000000001010000 (XEN) TR: sel=0x0690, attr=0x00083, limit=0x0000002b, base=0x0000000001213a30 (XEN) Guest PAT = 0x0000050100070406 (XEN) TSC Offset = fffd869460e8dc28 (XEN) DebugCtl=0000000000000000 DebugExceptions=0000000000000000 (XEN) Interruptibility=0008 ActivityState=0000 (XEN) *** Host State *** (XEN) RSP = 0xffff83083ff27f90 RIP = 0xffff82c4c01d9330 (XEN) CS=e008 DS=0000 ES=0000 FS=0000 GS=0000 SS=0000 TR=e040 (XEN) FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff83083ff2dc80 (XEN) GDTBase=ffff8318035ee000 IDTBase=ffff8318035fa000 (XEN) CR0=0000000080050033 CR3=0000000eefcc0000 CR4=00000000000026f0 (XEN) Sysenter RSP=ffff83083ff27fc0 CS:RIP=e008:ffff82c4c0216c60 (XEN) Host PAT = 0x0000050100070406 (XEN) *** Control State *** (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb (XEN) EntryControls=000011ff ExitControls=000fefff (XEN) ExceptionBitmap=00040042 (XEN) VMEntry: intr_info=80000202 errcode=5d021101 ilen=00000003 (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 (XEN) reason=80000021 qualification=00000000 (XEN) IDTVectoring: info=80000202 errcode=00000000 (XEN) TPR Threshold = 0x00 (XEN) EPT pointer = 0x0000000ef022501e (XEN) Virtual processor ID = 0xc0d0 (XEN) ************************************** (XEN) domain_crash called from vmx.c:2342 (XEN) Domain 5 (vcpu#3) crashed on cpu#20: (XEN) ----[ Xen-4.3.2_01-21.1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 20 (XEN) RIP: 9dcc:[<000000000000045f>] (XEN) RFLAGS: 0000000000000006 CONTEXT: hvm guest (XEN) rax: 0000000000000001 rbx: 00000000000026db rcx: 000000009dd40003 (XEN) rdx: 00000000248816a0 rsi: 0000000000000000 rdi: 0000000026b010e4 (XEN) rbp: 00000000000003ae rsp: 0000000000000398 r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000080000033 cr4: 0000000000002010 (XEN) cr3: 00000000feffc000 cr2: 0000000000000000 (XEN) ds: 9db4 es: 3ec8 fs: 0000 gs: 0000 ss: 9e7c cs: 9dcc -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On 01.10.14 at 09:27, <groen692@grosc.com> wrote: Recently I updated my openSuse box from 12.3 to 13.1. On this box I run xen with several guests. One of these guests is an appliance that has 4 kvm guests running. When I start this appliance with the nested vmx feature the appliance crashes either immediately or after a few minutes.
This same guest was running without a problem on opensuse releases 11.4 until 12.3
That's somewhat suspicious: Are you saying that even back then you already relied on nested virtualization (in order for KVM to work in a guest)?
( remark that about 3 year ago some patches were written by tim degan to make this appliance work ), I am not 100% sure what the right list to post this is soI also posted this problem on the xen-devel list.
Please be less vague for us to understand why you refer to. In particular we'd need to know whether you carried any patches in a private build of yours, which of course we can't really offer any help with. Whether the issue belongs here or on xen-devel (or xen-users) depends on what you expect, and by how much you're willing to help narrowing this down: From upstream's perspective, openSUSE and the Xen version it uses are of course only of secondary interest. It would be most interesting to know whether the -unstable tree (soon to become 4.5-rc) also exhibits the problem. The next best thing to try would be plain upstream 4.3.3 (or 4.4.1). Considering that you only recently switched, I suppose you don't really know whether this is a regression that got introduced during the lifetime of 13.1 (i.e. by one of the updates)?
==== outup xl demsg
I'll have to go through the dumped state in detail to see whether I can spot what caused the VM entry failure. That'll take some more time. Jan -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
Jan Beulich schreef op 6-10-2014 om 17:55:
On 01.10.14 at 09:27, <groen692@grosc.com> wrote: Recently I updated my openSuse box from 12.3 to 13.1. On this box I run xen with several guests. One of these guests is an appliance that has 4 kvm guests running. When I start this appliance with the nested vmx feature the appliance crashes either immediately or after a few minutes.
This same guest was running without a problem on opensuse releases 11.4 until 12.3 That's somewhat suspicious: Are you saying that even back then you already relied on nested virtualization (in order for KVM to work in a guest)? Yes, fyi on opensuse 11.4 I compiled the xen from source myself in order for nested vmx to work. I stopped running my on xen when the packeged xen incorperated the functionality I wanted.
( remark that about 3 year ago some patches were written by tim degan to make this appliance work ), I am not 100% sure what the right list to post this is soI also posted this problem on the xen-devel list. Please be less vague for us to understand why you refer to. In particular we'd need to know whether you carried any patches in a private build of yours, which of course we can't really offer any help with. See attached email for the patch back then, it was on the xen-devel list with topic "n2 MSR handling and capability exposure"
Whether the issue belongs here or on xen-devel (or xen-users) depends on what you expect, and by how much you're willing to help narrowing this down: From upstream's perspective, openSUSE and the Xen version it uses are of course only of secondary interest. It would be most interesting to know whether the -unstable tree (soon to become 4.5-rc) also exhibits the problem. The next best thing to try would be plain upstream 4.3.3 (or 4.4.1). No problem, tell me what is the fastest way for you towards on solution and I try to implement it. which ( if any ) repository it is on, or I should go back and compile the xen from source again.
Considering that you only recently switched, I suppose you don't really know whether this is a regression that got introduced during the lifetime of 13.1 (i.e. by one of the updates)? I do not know if it worked at all on opensuse 13.1
==== outup xl demsg I'll have to go through the dumped state in detail to see whether I can spot what caused the VM entry failure. That'll take some more time.
Jan
On 07.10.14 at 08:27, <groen692@grosc.com> wrote: Jan Beulich schreef op 6-10-2014 om 17:55: Whether the issue belongs here or on xen-devel (or xen-users) depends on what you expect, and by how much you're willing to help narrowing this down: From upstream's perspective, openSUSE and the Xen version it uses are of course only of secondary interest. It would be most interesting to know whether the -unstable tree (soon to become 4.5-rc) also exhibits the problem. The next best thing to try would be plain upstream 4.3.3 (or 4.4.1). No problem, tell me what is the fastest way for you towards on solution and I try to implement it. which ( if any ) repository it is on, or I should go back and compile the xen from source again.
Yes, of course from source. I don't think we have packaged up unstable yet (you might be able to pick something 4.4.1-ish from Factory). And in any event you need to be prepared that there's not going to be xend/xm anymore (i.e. other things may need updating too unless you're already using xl). Meanwhile I figured out that the issue is with NMI related state. Which raises the question: Are you also using PCI device pass- through? Is your system seeing NMIs for other reasons, or are you using the "watchdog" command line options? And in general I'm rather puzzled by guest state - you must be running some 16-bit OS inside KVM, which is quite unusual these days.
I do not know if it worked at all on opensuse 13.1
Then at least can you tell us which was the most recent version that worked for you without you adding additional patches and recompiling? Jan -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
Jan Beulich schreef op 7-10-2014 om 09:09:
On 07.10.14 at 08:27, <groen692@grosc.com> wrote: Jan Beulich schreef op 6-10-2014 om 17:55: Whether the issue belongs here or on xen-devel (or xen-users) depends on what you expect, and by how much you're willing to help narrowing this down: From upstream's perspective, openSUSE and the Xen version it uses are of course only of secondary interest. It would be most interesting to know whether the -unstable tree (soon to become 4.5-rc) also exhibits the problem. The next best thing to try would be plain upstream 4.3.3 (or 4.4.1). No problem, tell me what is the fastest way for you towards on solution and I try to implement it. which ( if any ) repository it is on, or I should go back and compile the xen from source again. Yes, of course from source. I don't think we have packaged up unstable yet (you might be able to pick something 4.4.1-ish from Factory). And in any event you need to be prepared that there's not going to be xend/xm anymore (i.e. other things may need updating too unless you're already using xl). I start this up again, It has been awhile since I have done that so it may take a while. And I am using xl toolstack already.
Meanwhile I figured out that the issue is with NMI related state. Which raises the question: Are you also using PCI device pass- through? Is your system seeing NMIs for other reasons, or are you using the "watchdog" command line options? And in general I'm rather puzzled by guest state - you must be running some 16-bit OS inside KVM, which is quite unusual these days. 16 bit is correct, It is an appliance with 4 kvm guests. 2 times a sles10sp4 and 2 proprietary OS 16 bit. No watchdog options, and no pci-passthrough for this guest. You need any traces? I had it crashed with state=preserved.
I do not know if it worked at all on opensuse 13.1 Then at least can you tell us which was the most recent version that worked for you without you adding additional patches and recompiling? It was working on an up to date version of opensuse 12.3 ( standard repositories )
Jan
-- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On 07.10.14 at 09:56, <groen692@grosc.com> wrote: 16 bit is correct, It is an appliance with 4 kvm guests. 2 times a sles10sp4 and 2 proprietary OS 16 bit. No watchdog options, and no pci-passthrough for this guest.
So do you know details of this proprietary OS? Is it making use of NMIs internally? I'm simply trying to understand where that NMI is having its origin... If you don't know, then maybe at least you can tell us whether the observed crash happens randomly or always at a deterministic point in the lifetime of the guest (the latter would hint towards guest-internal NMIs). Jan -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On 07.10.14 at 09:56, <groen692@grosc.com> wrote: 16 bit is correct, It is an appliance with 4 kvm guests. 2 times a sles10sp4 and 2 proprietary OS 16 bit. No watchdog options, and no pci-passthrough for this guest. So do you know details of this proprietary OS? Is it making use of NMIs internally? I'm simply trying to understand where that NMI is having its origin... I do not know anything about the internal NMI's, I think the base of the
If you don't know, then maybe at least you can tell us whether the observed crash happens randomly or always at a deterministic point in the lifetime of the guest (the latter would hint towards guest-internal NMIs). It looks random to me, The time the guest lives is certainly not always
Jan Beulich schreef op 7-10-2014 om 10:09: proprietary OS is BS1000/BS2000. the same. Sometimes it crashes after a few minutes sometimes it take 15 minutes, but rarely longer then that.
Jan
-- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On 07.10.14 at 15:24, <groen692@grosc.com> wrote: Jan Beulich schreef op 7-10-2014 om 10:09: If you don't know, then maybe at least you can tell us whether the observed crash happens randomly or always at a deterministic point in the lifetime of the guest (the latter would hint towards guest-internal NMIs). It looks random to me, The time the guest lives is certainly not always the same. Sometimes it crashes after a few minutes sometimes it take 15 minutes, but rarely longer then that.
Yet the dumped state is roughly the same each time? I.e. the death of the guest is always upon an NMI injection going wrong? Anyway, I suppose you saw the mail I sent to VMX maintainer (and xen-devel) to understand where to place the apparently missing code. Jan -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On 07.10.14 at 09:56, <groen692@grosc.com> wrote: 16 bit is correct, It is an appliance with 4 kvm guests. 2 times a sles10sp4 and 2 proprietary OS 16 bit. No watchdog options, and no pci-passthrough for this guest. So do you know details of this proprietary OS? Is it making use of NMIs internally? I'm simply trying to understand where that NMI is having its origin... I do not know anything about the internal NMI's, I think the base of the
On 07.10.14 at 15:24, <groen692@grosc.com> wrote: Jan Beulich schreef op 7-10-2014 om 10:09: proprietary OS is BS1000/BS2000.
Oh, one more thing: Did you check hypervisor and kernel logs for instances of NMIs being reported there? What does /proc/interrupts in Dom0 say regarding the NMI count? Jan -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
Oh, one more thing: Did you check hypervisor and kernel logs for instances of NMIs being reported there? What does /proc/interrupts in Dom0 say regarding the NMI count? CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 CPU8 CPU9 CPU10 CPU11 CPU12 CPU13 CPU14 CPU15 CPU16 CPU17 CPU18 CPU19 CPU20 CPU21 CPU22 CPU23 1: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi i8042 8: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi rtc0 9: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi acpi 12: 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi i8042 16: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi uhci_hcd:usb3, uhci_hcd:usb8 18: 278304 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi ehci_hcd:usb1, uhci_hcd:usb7, ata_piix, i801_smbus 19: 18301 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi uhci_hcd:usb6, ata_piix 21: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi uhci_hcd:usb4 23: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi ehci_hcd:usb2, uhci_hcd:usb5 64: 4438826 72423 4774 3459 3577 154438 4420 908388 85001 109580 28779 2863 51705 65838 30682 9314 3450 34471 113471 911724 12050 7378 72242 364746 Dynamic-percpu timer 65: 95232 45696 10328 3665 3353 2947 3392 42109 3107 8802 9726 2941 3059 6280 6341 14127 2936 3752 3766 36559 3130 3636 10725 27764 Dynamic-percpu ipi 66: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi pcpu 70: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi console 71: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi mce 72: 17 0 52 0 0 0 0 480 0 0 0 0 815 0 0 81 10 0 0 0 0 0 0 765 Phys-fasteoi ioat-msix 73: 782 0 0 52 0 0 0 0 480 0 0 0 0 815 81 0 0 10 0 0 0 0 0 0 Phys-fasteoi ioat-msix 74: 17 765 0 0 52 0 0 0 0 480 0 0 0 0 815 0 81 0 10 0 0 0 0 0 Phys-fasteoi ioat-msix 75: 17 0 765 0 0 52 0 0 0 0 480 0 0 0 0 815 0 81 0 10 0 0 0 0 Phys-fasteoi ioat-msix 76: 17 0 0 765 0 0 52 0 0 0 0 480 0 0 0 0 815 0 81 0 10 0 0 0 Phys-fasteoi ioat-msix 77: 17 0 0 0 765 0 0 52 0 0 0 0 480 0 0 0 0 825 0 81 0 0 0 0 Phys-fasteoi ioat-msix 78: 17 0 0 0 0 765 0 0 52 0 0 0 0 480 0 0 0 0 825 0 81 0 0 0 Phys-fasteoi ioat-msix 79: 17 0 0 0 0 0 765 0 0 52 0 0 0 0 480 0 0 0 0 825 0 81 0 0 Phys-fasteoi ioat-msix 80: 14 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 120 3248 0 1943 0 Phys-fasteoi eth0-rx-0 81: 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 0 0 0 0 Phys-fasteoi eth0-tx-0 82: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi eth0 83: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 11782 0 0 0 1333 18877 Phys-fasteoi eth1-rx-0 84: 8354 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1550 11040 0 Phys-fasteoi eth1-tx-0 85: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Phys-fasteoi eth1 86: 196 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1422 0 Dynamic-fasteoi evtchn:xenstored[1691] 87: 52 0 0 223 0 0 0 0 0 0 0 0 0 0 0 908 0 0 0 0 0 556 0 863 Dynamic-fasteoi xenbus 88: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenstored[1691] 89: 19 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenstored[1691] 90: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenconsoled[1709] 91: 2454318 0 0 0 0 0 0 1080708 0 0 0 0 0 0 0 0 0 0 0 923076 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[4902] 92: 637843 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[4902] 93: 984134 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[4902] 94: 332001 0 0 0 0 0 0 0 0 0 51506 0 0 0 0 0 0 0 0 0 0 0 126405 0 Dynamic-fasteoi evtchn:qemu-system-i38[4902] 95: 39 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[4902] 96: 331 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenstored[1691] 97: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenconsoled[1709] 98: 483057 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5541] 99: 10506 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5541] 100: 135 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5541] 101: 5558 0 0 0 0 0 0 7 0 0 0 0 0 0 0 7899 0 0 0 1012 0 0 0 0 Dynamic-fasteoi blkif-backend 102: 26 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5541] 103: 2 0 0 0 0 0 0 0 0 6 0 0 0 0 0 0 0 717 0 0 0 204 0 0 Dynamic-fasteoi vif2.0 104: 313 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenstored[1691] 105: 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenconsoled[1709] 106: 3105 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5826] 107: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5826] 108: 3706 0 193 0 0 0 0 0 0 0 0 0 0 0 2487 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi blkif-backend 109: 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5826] 110: 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi vif3.0 111: 324 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenstored[1691] 112: 128 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:xenconsoled[1709] 113: 3107 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5923] 114: 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5923] 115: 2404 0 0 0 0 0 0 0 0 4514 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi blkif-backend 116: 46 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi evtchn:qemu-system-i38[5923] 117: 2 0 0 0 0 0 0 0 0 0 0 243 0 0 0 0 0 0 0 0 0 0 0 0 Dynamic-fasteoi vif4.0 NMI: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Non-maskable interrupts IWI: 471 498 126 48 82 122 47 50 110 501 129 149 160 52 82 295 50 145 173 121 55 62 51 366 IRQ work interrupts RES: 94089 39305 4443 2220 2213 1611 2098 36390 1739 2305 6399 1588 1304 1343 1911 1755 1762 2543 2171 31788 1621 1794 9668 25742 Rescheduling interrupts CAL: 697 5928 5776 1413 1068 1225 1257 5677 1271 6006 3210 1212 1604 4896 4361 12079 1131 1079 1432 4671 1465 1811 1020 1672 Function call interrupts LCK: 49 0 1 4 0 3 6 6 6 0 0 0 1 2 9 4 2 0 5 0 4 0 3 3 Spinlock wakeups MCE: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Machine check exceptions MCP: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Machine check polls
Jan
-- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
Hi Jan, I installed xen-4.5.29823-1.xen_unstable.1.x86_64 from the http://download.opensuse.org/repositories/home:/olh:/xen-unstable/openSUSE_F... After about 45 minutes the guests crashed again. mfg, Jeroen (XEN) vvmx.c:2476:d1v0 Unknown nested vmexit reason 80000021. (XEN) Failed vm entry (exit reason 0x80000021) caused by invalid guest state (0). (XEN) ************* VMCS Area ************** (XEN) *** Guest State *** (XEN) CR0: actual=0x0000000000000033, shadow=0x0000000000000011, gh_mask=ffffffffffffffff (XEN) CR4: actual=0x0000000000002050, shadow=0x0000000000000000, gh_mask=ffffffffffffffff (XEN) CR3: actual=0x00000000feffc000, target_count=0 (XEN) target0=0000000000000000, target1=0000000000000000 (XEN) target2=0000000000000000, target3=0000000000000000 (XEN) RSP = 0x00000000000006be (0x00000000000006be) RIP = 0x000000000000017d (0x000000000000017d) (XEN) RFLAGS=0x0000000000000002 (0x0000000000000002) DR7 = 0x0000000000000400 (XEN) Sysenter RSP=0000000000000000 CS:RIP=0000:0000000000000000 (XEN) CS: sel=0xbf70, attr=0x0009b, limit=0x0000075a, base=0x000000000194fec0 (XEN) DS: sel=0x3cec, attr=0x00093, limit=0x00000110, base=0x000000000bcbe276 (XEN) SS: sel=0x3cf4, attr=0x00093, limit=0x000007ff, base=0x000000000bcbe3aa (XEN) ES: sel=0x3cfc, attr=0x00093, limit=0x00000099, base=0x000000000bcbebac (XEN) FS: sel=0x0000, attr=0x1c000, limit=0xffffffff, base=0x0000000000000000 (XEN) GS: sel=0x0000, attr=0x1c000, limit=0xffffffff, base=0x0000000000000000 (XEN) GDTR: limit=0x0000ffff, base=0x0000000001000000 (XEN) LDTR: sel=0x0820, attr=0x00082, limit=0x0000ffff, base=0x0000000001010400 (XEN) IDTR: limit=0x000003ff, base=0x0000000001010000 (XEN) TR: sel=0x0690, attr=0x00083, limit=0x0000002b, base=0x0000000001213a30 (XEN) Guest PAT = 0x0000050100070406 (XEN) TSC Offset = fffffe8a52350fd4 (XEN) DebugCtl=0000000000000000 DebugExceptions=0000000000000000 (XEN) Interruptibility=0008 ActivityState=0000 (XEN) *** Host State *** (XEN) RSP = 0xffff830836b5ff90 RIP = 0xffff82d0801e3ed0 (XEN) CS=e008 DS=0000 ES=0000 FS=0000 GS=0000 SS=0000 TR=e040 (XEN) FSBase=0000000000000000 GSBase=0000000000000000 TRBase=ffff830836b64c80 (XEN) GDTBase=ffff830836b55000 IDTBase=ffff830836b61000 (XEN) CR0=0000000080050033 CR3=000000082ffdd000 CR4=00000000000026f0 (XEN) Sysenter RSP=ffff830836b5ffc0 CS:RIP=e008:ffff82d0802220a0 (XEN) Host PAT = 0x0000050100070406 (XEN) *** Control State *** (XEN) PinBased=0000003f CPUBased=b6b9e5fa SecondaryExec=000004eb (XEN) EntryControls=000011ff ExitControls=000fefff (XEN) ExceptionBitmap=00040042 (XEN) VMEntry: intr_info=80000202 errcode=00000000 ilen=00000003 (XEN) VMExit: intr_info=00000000 errcode=00000000 ilen=00000003 (XEN) reason=80000021 qualification=00000000 (XEN) IDTVectoring: info=80000202 errcode=00000000 (XEN) TPR Threshold = 0x00 (XEN) EPT pointer = 0x000000082ffff01e (XEN) Virtual processor ID = 0xe6a3 (XEN) ************************************** (XEN) domain_crash called from vmx.c:2483 (XEN) Domain 1 (vcpu#0) crashed on cpu#9: (XEN) ----[ Xen-4.5.29823-1.xen_unstable.1 x86_64 debug=n Not tainted ]---- (XEN) CPU: 9 (XEN) RIP: bf70:[<000000000000017d>] (XEN) RFLAGS: 0000000000000002 CONTEXT: hvm guest (XEN) rax: 0000000000040002 rbx: 0000000000042701 rcx: 00000000bf480003 (XEN) rdx: 00000000248816a0 rsi: 0000000000000000 rdi: 0000000026b00096 (XEN) rbp: 00000000000006d4 rsp: 00000000000006be r8: 0000000000000000 (XEN) r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) r15: 0000000000000000 cr0: 0000000000000033 cr4: 0000000000002010 (XEN) cr3: 00000000feffc000 cr2: 0000000000000000 (XEN) ds: 3cec es: 3cfc fs: 0000 gs: 0000 ss: 3cf4 cs: bf70 Jan Beulich schreef op 7-10-2014 om 10:09:
On 07.10.14 at 09:56, <groen692@grosc.com> wrote: 16 bit is correct, It is an appliance with 4 kvm guests. 2 times a sles10sp4 and 2 proprietary OS 16 bit. No watchdog options, and no pci-passthrough for this guest. So do you know details of this proprietary OS? Is it making use of NMIs internally? I'm simply trying to understand where that NMI is having its origin... If you don't know, then maybe at least you can tell us whether the observed crash happens randomly or always at a deterministic point in the lifetime of the guest (the latter would hint towards guest-internal NMIs).
Jan
-- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On 07.10.14 at 17:04, <groen692@grosc.com> wrote: I installed xen-4.5.29823-1.xen_unstable.1.x86_64 from the http://download.opensuse.org/repositories/home:/olh:/xen-unstable/openSUSE_F... ctory/
After about 45 minutes the guests crashed again.
Good (in a sense). In which case we really should continue this on the thread on xen-devel. Could you repost this reply of yours as a reply there? Jan -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On Tue, Oct 07, Jeroen Groenewegen van der Weyden wrote:
I installed xen-4.5.29823-1.xen_unstable.1.x86_64
For the record, the number is the hg revision from http://xenbits.xen.org/hg/staging/xen-unstable.hg/ Olaf -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
On Tue, Oct 07, Jan Beulich wrote:
Yes, of course from source. I don't think we have packaged up unstable yet (you might be able to pick something 4.4.1-ish from Factory). And in any event you need to be prepared that there's not going to be xend/xm anymore (i.e. other things may need updating too unless you're already using xl).
I maintain a package from xen unstable sources for some dists: zypper -v -v ar -c -f -K \ http://download.opensuse.org/repositories/home:/olh:/xen-unstable/openSUSE_1... \ remote-home_olh_xen_unstable Hopefully that works also for the OP. Olaf -- To unsubscribe, e-mail: opensuse-virtual+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-virtual+owner@opensuse.org
participants (3)
-
Jan Beulich -
Jeroen Groenewegen van der Weyden -
Olaf Hering