[Bug 631194] New: xen unexpectedly fails with out-of-memory situation
http://bugzilla.novell.com/show_bug.cgi?id=631194 http://bugzilla.novell.com/show_bug.cgi?id=631194#c0 Summary: xen unexpectedly fails with out-of-memory situation Classification: openSUSE Product: openSUSE 11.3 Version: Final Platform: x86-64 OS/Version: Linux Status: NEW Severity: Critical Priority: P5 - None Component: Xen AssignedTo: jdouglas@novell.com ReportedBy: jengelh@medozas.de QAContact: qa@suse.de Found By: Beta-Customer Blocker: --- It is possible to force Xen into a situation where it believes it has no more memory. I seem to have hit this today pretty often, and isolated one test case. ---8<--- 22:40 xen35:~ # vm-install Gathering settings... Please specify the type of operating system that will run within the virtual machine. This defines many defaults, and helps decide how to start paravirtualized operating systems. Press 'q' or the Escape key to exit. 1: Novell Open Enterprise Server 2 (Linux) 2: Novell Open Enterprise Server 2 (NetWare) 3: Other operating system 4: PXE 5: RedHat (other) 6: RedHat Enterprise Linux 3 7: RedHat Enterprise Linux 4 8: RedHat Enterprise Linux 5 9: SUSE (other) 10: SUSE Linux Enterprise Desktop 10 11: SUSE Linux Enterprise Desktop 11 12: SUSE Linux Enterprise Server 8 13: SUSE Linux Enterprise Server 9 14: SUSE Linux Enterprise Server 10 15: SUSE Linux Enterprise Server 11 16: Solaris 9 and older 17: Solaris 10 18: Windows (other) 19: Windows (other, x64) 20: Windows NT 21: Windows Server 2008 22: Windows Server 2008 (x64) 23: Windows Vista, Windows 7 24: Windows Vista, Windows 7 (x64) 25: Windows XP, 2000, 2003 26: Windows XP, 2003 (x64) 27: openSUSE 28: openSUSE 11 [28] > Please choose a name for the virtual machine. [opensuse11] > Description > Specify the amount of memory and number of processors to allocate for the VM. Initial Memory [512] > Maximum Memory [512] > Virtual Processors [1] > Please specify the type of virtualized graphics hardware. 1: No Graphics Support 2: Paravirtualized Graphics Adapter [2] > 1 Virtual Disks: (None) Do you want to add another virtual disk? (Y / N) [Y] > y Create a virtual disk based on a device (CD or other block device), an existing image file (ISO), or a new file. Specify a device by its device node, such as /dev/cdrom, not its mount point. What type of virtual disk do you want to add? 1: CD-ROM or DVD 2: Hard Disk [2] > 2 Where will the virtual disk physically reside? [/var/lib/xen/images/opensuse11/xvda] > Size (GB) [8.0] > 1 Create a sparse image file for the virtual disk? (Y / N) [Y] > n Virtual Disks: 1.0 GB Hard Disk (file:/var/lib/xen/images/opensuse11/xvda) Do you want to add another virtual disk? (Y / N) [N] > y Create a virtual disk based on a device (CD or other block device), an existing image file (ISO), or a new file. Specify a device by its device node, such as /dev/cdrom, not its mount point. What type of virtual disk do you want to add? 1: CD-ROM or DVD 2: Hard Disk [2] > 1 Where will the virtual disk physically reside? [/var/lib/xen/images/opensuse11/xvdb] > /mnt/iso/openSUSE-11.3-DVD-x86_64.iso Virtual Disks: 1.0 GB Hard Disk (file:/var/lib/xen/images/opensuse11/xvda) 4.2 GB CD-ROM or DVD (file:/mnt/iso/openSUSE-11.3-DVD-x86_64.iso) Do you want to add another virtual disk? (Y / N) [N] > Network Adapters (None) Do you want to add another virtual network adapter? (Y / N) [Y] > Network Adapters Paravirtualized; Randomly generated MAC address Do you want to add another virtual network adapter? (Y / N) [N] > Specify the bootable virtual disk (often labeled as Disk 1) or the network installation source URL. Each CD, DVD, or ISO image required for installation must be added as a virtual disk. 1: Network URL (ftp://, http://, nfs://, smb://) 2: Virtual Disk (4.2 GB CD-ROM or DVD (file:/mnt/iso/openSUSE-11.3-DVD-x86_64.iso)) [1] > 2 Some operating systems support automating the installation by specifying a URL or file(s). Select a directory to include multiple files. AutoYaST file > Some operating systems accept additional arguments, used to customize the installation or boot process. Additional Arguments > Preparing to start the installation... /dev/sdb: open failed: No medium found Installing... (some kernel messages, eventually yast2) --->8--- Once the kernel starts, hit ^] anytime (directly or when the first yast2 dialog is onscreen). With ^] installation aborts as desired. Rerun vm-install now. Xen fails to launch then: --->8--- Installing... Error: Failed to start the VM. (Error: Not enough memory is available, and dom0 cannot be shrunk any further Usage: xm start <DomainName> Start a Xend managed domain -p, --paused Do not unpause domain after starting it -c, --console_autoconnect Connect to the console after the domain is created --vncviewer Connect to display via VNC after the domain is created --vncviewer-autopass Pass VNC password to viewer via stdin and -autopass ) ---8<--- 22:46 xen35:/var/log/xen # free total used free shared buffers cached Mem: 5562368 421836 5140532 0 13724 121280 -/+ buffers/cache: 286832 5275536 Swap: 522108 0 522108 -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=631194 http://bugzilla.novell.com/show_bug.cgi?id=631194#c1 --- Comment #1 from Jan Engelhardt <jengelh@medozas.de> 2010-08-13 20:53:32 UTC --- Created an attachment (id=382978) --> (http://bugzilla.novell.com/attachment.cgi?id=382978) domain-builder-ng.log.xz -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=631194 http://bugzilla.novell.com/show_bug.cgi?id=631194#c2 --- Comment #2 from Jan Engelhardt <jengelh@medozas.de> 2010-08-13 20:53:50 UTC --- Created an attachment (id=382979) --> (http://bugzilla.novell.com/attachment.cgi?id=382979) xend-debug.log.xz -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=631194 http://bugzilla.novell.com/show_bug.cgi?id=631194#c3 --- Comment #3 from Jan Engelhardt <jengelh@medozas.de> 2010-08-13 20:54:12 UTC --- Created an attachment (id=382980) --> (http://bugzilla.novell.com/attachment.cgi?id=382980) xend.log.xz -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=631194 http://bugzilla.novell.com/show_bug.cgi?id=631194#c Charles Arnold <carnold@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |carnold@novell.com AssignedTo|jdouglas@novell.com |jfehlig@novell.com QAContact|qa@suse.de |jdouglas@novell.com -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c4 James Fehlig <jfehlig@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |jengelh@medozas.de --- Comment #4 from James Fehlig <jfehlig@novell.com> 2010-08-24 23:29:15 UTC --- Hmm, I'm not able to reproduce this on 11.3 machine running 11.3 KOTD. How much physical memory does your machine have? Could you provide output of 'xm info'? Also, I'd suggest disabling dom0 ballooning anyhow and give dom0 an explicit amount of memory. E.g /etc/xen/xend-config.sxp (enable-dom0-ballooning no) /boot/grub/menu.lst of host title Xen -- openSUSE 11.3 - 2.6.34.4-0.0.25.5dc3ffe root (hd0,4) kernel /boot/xen.gz dom0_mem=1024M ... ... Is there any reason you need dom0 ballooning enabled? If not, can you try with it disabled? Thanks! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c Ihno Krumreich <ihno@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ihno@novell.com OS/Version|Linux |openSUSE 11.3 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c5 --- Comment #5 from Jan Engelhardt <jengelh@medozas.de> 2010-08-30 21:50:05 UTC ---
running 11.3 KOTD.
I am on openSUSE's 11.3/ 2.6.34-12-xen. No KOTD. 6 GB of memory.
Is there any reason you need dom0 ballooning enabled?
It's the openSUSE default, I did not decide anything in that regard. More on that tomorrow. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c6 --- Comment #6 from James Fehlig <jfehlig@novell.com> 2010-08-30 22:07:17 UTC --- (In reply to comment #5)
running 11.3 KOTD.
I am on openSUSE's 11.3/ 2.6.34-12-xen. No KOTD. 6 GB of memory.
Ah, there is a serious netback deadlock issue in the shipping 11.3 xen kernel that manifests itself in all sorts of strange bugs. I'm not sure if it is worth investigating this issue with shipping 11.3 kernel-xen. Unfortunately I can't ask you to apply an updated kernel since one hasn't been released yet :-(. A kernel update is being prepared and hopefully it will make it through the maintenance process here soon. In the meantime I can only as that you try the 11.3 KOTD.
Is there any reason you need dom0 ballooning enabled?
It's the openSUSE default, I did not decide anything in that regard.
Yep, it is the default and is generally a sane and safe one (also default upstream). But IMO best-practices dictate turning dom0 ballooning off and giving dom0 an explicit amount of memory. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c7 Jan Engelhardt <jengelh@medozas.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|jengelh@medozas.de | --- Comment #7 from Jan Engelhardt <jengelh@medozas.de> 2010-08-30 22:22:37 UTC ---
manifests itself in all sorts of strange bugs.
Including "Error: Device 0 (vif)...Hotplug scripts not working." when trying to start a new VM?
that you try the 11.3 KOTD.
Already tried KOTD at one point. What it gave me was a crash-at-boot. I prefer not to retry that with servers that don't have KVM or a physical access . . .
But IMO best-practices dictate turning dom0 ballooning off
Though I would love to retain ballooning in at least one direction. Is there like dom0_maxmem too, such that dom0 can always go lower, but never above maxmem? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c8 --- Comment #8 from James Fehlig <jfehlig@novell.com> 2010-08-30 22:50:48 UTC --- (In reply to comment #7)
Including "Error: Device 0 (vif)...Hotplug scripts not working." when trying to start a new VM?
Heh, yeah that's the most common error.
Already tried KOTD at one point. What it gave me was a crash-at-boot. I prefer not to retry that with servers that don't have KVM or a physical access . . .
Yep, understood. Unfortunately you'll have to wait for the (overdue) update.
Though I would love to retain ballooning in at least one direction. Is there like dom0_maxmem too, such that dom0 can always go lower, but never above maxmem?
Memory is never automatically given back to dom0 once it has been ballooned down. Automatically, you only go one direction anyway - down. To give dom0 more memory you have to explicitly do it with 'xm mem-set 0 <new-val>'. By disabling ballooning I mean the automatic dom0 ballooning done by the tools when starting new domUs. You could still manually balloon dom0. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c9 --- Comment #9 from Jan Engelhardt <jengelh@medozas.de> 2010-08-31 19:03:01 UTC --- There isn't even a kernel-xen in http://download.opensuse.org/repositories/Kernel:/HEAD/openSUSE_11.3/x86_64/ to test right now.. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c10 --- Comment #10 from James Fehlig <jfehlig@novell.com> 2010-08-31 20:04:03 UTC --- That's the head kernel (2.6.36), which doesn't have the xen forward ports yet. If you want to use a test kernel until the update is released, I'd suggest the 11.3 KOTD (2.6.34) ftp://ftp.suse.com/pub/projects/kernel/kotd/openSUSE-11.3/x86_64/ -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c11 --- Comment #11 from Jan Engelhardt <jengelh@medozas.de> 2010-09-01 13:54:35 UTC --- That 2.6.34.4 seems to work well. (Note: the kernel that crashed was a 2.6.35rc/35 KOTD) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c12 James Fehlig <jfehlig@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |jengelh@medozas.de --- Comment #12 from James Fehlig <jfehlig@novell.com> 2010-09-01 14:15:34 UTC --- The 11.3 KOTD is usually stable. Actually, this can be said for any shipping product KOTD. A product's KOTD is the kernel that shipped with the product + all the latest bug fixes, backports, etc. The head KOTD, on the other hand, is often unstable. It's the kernel under development for next openSUSE release and may not work at all on a given day. Anyhow, with an *11.3* KOTD, are you still seeing the reported issue? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c13 Jan Engelhardt <jengelh@medozas.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|jengelh@medozas.de | --- Comment #13 from Jan Engelhardt <jengelh@medozas.de> 2010-09-01 14:21:57 UTC --- The 11.3 KOTD, that is, 2.6.34.4-0.0.25.5dc3ffe-xen, works for me. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c14 James Fehlig <jfehlig@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |FIXED --- Comment #14 from James Fehlig <jfehlig@novell.com> 2010-09-01 14:30:13 UTC --- Ok, thanks. I'm closing this bug. BTW, a kernel update for 11.3 is in the works. AFAICT, it is in QA so will be released shortly. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=631194 https://bugzilla.novell.com/show_bug.cgi?id=631194#c Jan Engelhardt <jengelh@medozas.de> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |VERIFIED -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply@novell.com