http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 Guillaume GARDET changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |afaerber@suse.com, | |agraf@suse.com, | |dmueller@suse.com, | |mbrugger@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 Guillaume GARDET changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |okurz@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 Stefan Brüns changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |stefan.bruens@rwth-aachen.d | |e -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 Oliver Kurz changed: What |Removed |Added ---------------------------------------------------------------------------- Blocker|--- |Yes Severity|Normal |Major -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c4 Oliver Kurz changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |richard.palethorpe@suse.com --- Comment #4 from Oliver Kurz --- That's good news. Thanks for even coming up with a suggested mitigation in the os-autoinst code -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c6 --- Comment #6 from Guillaume GARDET --- Dirk deployed his workaround to aarch64 openQA worker. Things are better, but we still see some failures, especially on 'kde' and 'create_hdd_kde' tests. As a workaround, I added 'EXTRABOOTPARAMS= rcupdate.rcu_task_stall_timeout=0' to aarch64 machine settings to disable RCU task stall messages, using grub. It has been used for Leap 15.1 build 70.2 and worked fine. Let's see how it goes for Tumbleweed build 20180829. This workaround only applies for boot from ISO, not boot from HDD, but long VM snapshots are only after installation, AFAICS. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c8 Guillaume GARDET changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |IN_PROGRESS --- Comment #8 from Guillaume GARDET --- (In reply to Guillaume GARDET from comment #6)

As a workaround, I added 'EXTRABOOTPARAMS= rcupdate.rcu_task_stall_timeout=0' to aarch64 machine settings to disable RCU task stall messages, using grub.

It is not working since there is a reboot after the installation. It seems GRUB_KERNEL_OPTION_APPEND would be a better candidate as it is used in 'grub_test', after the reboot of the installation. GRUB_KERNEL_OPTION_APPEND should probably also be added to 'boot_to_desktop' test. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c10 --- Comment #10 from Richard Palethorpe --- Apparently there is also GRUB_CMDLINE_LINUX_DEFAULT -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c12 --- Comment #12 from Richard Palethorpe --- It seems that on a Centriq 2400 44-core machine with a single worker with 4 cores assigned (to QEMU) and compression level 6 it takes approx 40 seconds and the dump is about 1.7GB With compress level 1 it takes approx 30 seconds and the dump is about 1.6GB (this depends where it is in the installation though, it goes up to ~2.1GB). So we get a 25% reduction in time for a 7% increase in space. However if we give the worker 8 cores, then this comes down to approx. 15 seconds. It seems that at around 14-15 cores it stops being CPU bound. On a contended machine we will probably find that the load stops being CPU bound sooner, so 8-10 cores seems like a reasonable figure for each worker. I'm not entirely sure what kind of workers we have at the moment on o3, but they are probably too over-contended to deal with snapshots. Although this is not the most rigorous profiling, I would conclude from this that we need to reduce the compression to 1 on ARM and give each worker 8-10 cores. Possibly we can get away with slightly over provisioning the number of cores, but not by much. Differential snapshots (if they are implemented in QEMU yet) might help to some degree, but the chances are most the RAM pages will be dirty during the first snapshot after installation, so they will all need to be written. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c14 --- Comment #14 from Richard Palethorpe --- (In reply to Stefan Brüns from comment #13)

Just another idea - the snapshotted RAM also includes buffers and caches, correct? Can we just tell the kernel to drop these prior to the snapshot?

Interestingly the first snapshot, which has the most problems, is taken directly after restarting the machine. So the guest kernel will have dropped most the RAM before taking the snapshot. However QEMU does not seem to be aware of this, I am not sure if there is a way for QEMU to know (without the guest agent) if the guest kernel has stopped using a page of RAM. Maybe we could use -no-reboot which should cause QEMU to exit instead of reboot and then restart QEMU. This will certainly drop all the RAM... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c16 --- Comment #16 from Stefan Brüns --- Dropping caches indeed needs some additional steps to make the host aware of the no longer used memory. Just dropping caches etc. will not change the pages contents, and we will still end up with incompressible noise. The balloon driver should be able to give back memory to the host, but AFAIK ballooning is completely manual ATM, so the sequence would be - 1. drop caches, 2. trigger ballooning using QMP, 3. snapshot. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c18 --- Comment #18 from Richard Palethorpe --- I tried ballooning with the following method before taking a snapshot: 1) Call balloon with a target guest RAM usage of 1GB 2) Wait for balloon to stop recovering RAM 3) Take the snapshot 4) Call balloon again with a target of QEMURAM (4GB in this case) This resulted in a significant snapshot time reduction of approx. 50%, but the actual snapshot size (after installation and a reboot) remained constant at around 1.6-1.7GB. It should be less than 600MB I can only speculate as to why that is, but it seems like a reasonable improvement. https://github.com/os-autoinst/os-autoinst/pull/1020 Unfortunately ballooning is not totally safe, setting the target usage very low will kill the guest kernel. However I think with the given speed improvement it may be worth using. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c20 --- Comment #20 from Richard Palethorpe --- (In reply to Alexander Graf from comment #19)

QEMU's RAM migration code treats all-0 pages special, so maybe it's enough to just make as many pages as we can contain all zeros?

This looks similar to a balloon, but where the pages are zeroed instead of just locked inside the balloon :-p -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c22 --- Comment #22 from Guillaume GARDET --- (In reply to Guillaume GARDET from comment #6)

rcupdate.rcu_task_stall_timeout=0' to aarch64 machine settings to disable RCU task stall messages, using grub.

'rcupdate.rcu_task_stall_timeout=0' did not disabled RCU messages, let's try 'rcupdate.rcu_cpu_stall_suppress=1' for next snapshot. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1105302 http://bugzilla.opensuse.org/show_bug.cgi?id=1105302#c27 Guillaume GARDET changed: What |Removed |Added ---------------------------------------------------------------------------- Status|IN_PROGRESS |RESOLVED Resolution|--- |FIXED --- Comment #27 from Guillaume GARDET --- With listed patches and the new machine for openQA, this bug is fixed. -- You are receiving this mail because: You are on the CC list for the bug.