http://bugzilla.novell.com/show_bug.cgi?id=589788 http://bugzilla.novell.com/show_bug.cgi?id=589788#c0 Summary: Xen hypervisor memory utilization does not reflect sum of domain allocations (leaks over time) Classification: openSUSE Product: openSUSE 11.2 Version: Final Platform: x86-64 OS/Version: openSUSE 11.2 Status: NEW Severity: Normal Priority: P5 - None Component: Xen AssignedTo: jdouglas@novell.com ReportedBy: ffejes@searshc.com QAContact: qa@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2) Gecko/20100218 Fedora/3.6.1-2.fc14 Firefox/3.6 Over the past couple months I've noticed that a few of our Xen servers are reporting (via xentop) hypervisor memory utilization discrepancies between the sum of VM memory allocations and the actual memory used. In time, the free memory drops to a point where no new VMs can be started. Below is recent sample xentop output with extra columns chopped. The hypervisor reports that it is using 32gb of ram, however the sum of the running VMs is 16gb. Ballooning is disabled and dom0_mem is set to 4gb. xentop - 15:38:20 Xen 3.4.1_19718_04-2.1 12 domains: 2 running, 10 blocked, 0 paused, 0 crashed, 0 dying, 0 shutdown Mem: 33549248k total, 33241552k used, 307696k free CPUs: 4 @ 2992MHz NAME STATE CPU(sec) CPU(%) MEM(k) MEM(%) MAXMEM(k) MAXMEM(%) xxxxxxxxxx --b--- 1584 0.2 547840 1.6 547840 1.6 Domain-0 -----r 165894 34.3 4192768 12.5 no limit n/a xxxxxxxxxx --b--- 15166 22.3 3071916 9.2 3076096 9.2 xxxxxxxxxx -----r 1328 77.8 1028012 3.1 1028096 3.1 xxxxxxxxxx --b--- 55433 4.7 1027880 3.1 1028096 3.1 xxxxxxxxxx --b--- 67527 4.8 1027880 3.1 1028096 3.1 xxxxxxxxxx --b--- 73019 6.8 1027880 3.1 1028096 3.1 xxxxxxxxxx --b--- 61266 6.8 1027880 3.1 1028096 3.1 xxxxxxxxxx --b--- 78880 6.2 1028012 3.1 1028096 3.1 xxxxxxxxxx --b--- 969 7.7 516012 1.5 516096 1.5 xxxxxxxxxx --b--- 10544 1.5 1027880 3.1 1028096 3.1 xxxxxxxxxx --b--- 21001 5.8 1027880 3.1 1028096 3.1 This behavior is only seen on our "development" Xen servers where we routinely create/clone/destroy a large number of (mostly HVM) VMs, however when doing any of those functions I cannot reproduce the "leak". That is to say, when I create a VM the hypervisor memory free statistic drops in an amount corresponding to the size of the VM. When the VM is destroyed, the free statistic increases back to where it was previously. I have not been able to find any way to get the hypervisor to reclaim some of the lost memory and, as a result, I have been forced to reboot the server. On systems with 20+ VMs where we don't have the option for live migration this can be rather traumatic. Thank you! Reproducible: Couldn't Reproduce Steps to Reproduce: 1. 2. 3. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.