http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c1 Michal Suchanek changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |marius.kittler@suse.com Flags| |needinfo?(marius.kittler@su | |se.com) --- Comment #1 from Michal Suchanek --- What is the firmware version of the machines? -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c3 --- Comment #3 from Michal Suchanek --- Not sure what that firmware revision relates to. Typically the POWER firmware revisions shown look something like SV840_FW840.00 (56) FW840.00 (SV840_056) -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c5 --- Comment #5 from Michal Suchanek --- The ASMI (web management interface for the machine), and some random tools output it as well (it's seen multiple times in supportconfig). -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c9 --- Comment #9 from Michal Suchanek --- It look like the these machine are Opal-only so the the firmware version is not comparable with the PowerVM firmware versions The latest firmware available from https://www.ibm.com/support/fixcentral 8335-GCA: 8335_820.1923.20190604n_update.hpm OP8_v1.12_2.98.pd.sdd S822LC-8335-GCA-GTA-OpenPowerReadme_op820.30.xhtml 8348-21C: 8348_820.1923.20190613n.html 8348_820.1923.20190613n_update.hpm OP8_v1.12_2.96_H.dd.xml OP8_v1.12_2.96_H.pd.sdd The machines use different firmware build so the results between them are not really comparable. Please update the firmware on the problematic machine. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c11 --- Comment #11 from Michal Suchanek --- Can you set up kdump on the machine? -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c13 --- Comment #13 from Marius Kittler --- Ok, the kdump config can be adjusted. I was just to stupid to use the ncurses-based UI. I now set the memory to the value recommended by the UI and also enabled "Use Firmware-Assisted Dump". Maybe these settings make a difference? Tomorrow I can try rebooting on Leap 15.4 although I'll have to clarify whether that's actually ok (as the worker is used in production). -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c15 --- Comment #15 from Marius Kittler --- I disabled it again. Unfortunately I had too many other things to do today so I haven't been able to proceed with the investigation. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c17 --- Comment #17 from Marius Kittler --- While qa-power8-5-kvm at least doesn't crash anymore under Linux 5.8 it still doesn't operate normally. It appears that QEMU is getting stuck (and therefore openQA jobs timeout). The error message ``` ��� QA-Power8-5-kvm kernel: cma: cma_alloc: alloc failed, req-size: 512 pages, ret: -4 ``` is showing up quite often in the logs, see https://progress.opensuse.org/issues/117229#note-4. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c19 --- Comment #19 from Marius Kittler --- I'm actually wondering whether the problem still occurs on the latest kernel version. Maybe it has already been fixed so one can simply update to the latest release. However, the newest version I could find in home:tiwai:kernel:X projects is 5.18. Maybe I'll try that one unless there's a repo providing e.g. the current kernel version of TW for Leap 15.4 on PPC. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c21 --- Comment #21 from Marius Kittler --- Unfortunately the newer kernel versions from home:tiwai:kernel:X projects aren't even installable at all on Leap 15.4 because they're conflicting with 'filesystem < 16' provided by filesystem-15.0-11.8.1.ppc64le. The kernel from the backport repo is installable so I've installed and booted into it. Let's see how well that one works. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c23 --- Comment #23 from Marius Kittler --- It crashes in the same way on vanilla Linux 6.0.0, see https://progress.opensuse.org/issues/114565#note-62 and subsequent comments. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c25 --- Comment #25 from Michal Suchanek --- Reproducibility once a day is not all that great. I could try to bisect building kernels locally but with this reproducibility rate it's difficult to get reliable results. -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c27 --- Comment #27 from Marius Kittler --- I've tried some stress testing. It is definitely not hard to produce *some* crashes/hang-ups. However, I'm not sure whether these tests are very meaningful to tackle the problems we see under actual production load. You can read my comments around https://progress.opensuse.org/issues/114565#note-73 for more details. We could also still give you access to the machine (ipmi and ssh) if you want to run some tests for yourself. (If you have better ideas for stress testing that would also already be helpful.) -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c29 --- Comment #29 from Marius Kittler --- Now the machine powerqaworker-qam-1.qa.suse.de has started to show the same symptoms. To my knowledge it ran just fine on Leap 15.4 with the standard kernel but for some reason it is unstable now as well. It has already crashed 2 times this week. Like on the other hosts there's no useful information in the journal (see https://progress.opensuse.org/issues/120004 for the related ticket on our side). I find it quite weird that not all machines showed the symptoms directly after the upgrade to Leap 15.4. As stated in the bug description, only qa-power8-4-kvm.qa.suse.de appeared to be problematic at the beginning while qa-power8-5-kvm.qa.suse.de seemed to be stable and only showed the problem a bit later. Now a 3rd machine shows the problem. Maybe the workload has changed. (I'll investigate that.) -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c32 Oliver Kurz changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(marius.kittler@su | |se.com) | --- Comment #32 from Oliver Kurz --- (In reply to Michal Suchanek from comment #31)

What kind of hardware is powerqaworker-qam-1.qa.suse.de?

Yes, this is of course also Power8. https://racktables.nue.suse.com/index.php?page=object&tab=default&object_id=9710 says it's in particular IBM Power System S822LC

So far the problematic workers were POWER8 which use the old KVM code which gets very little testing upstream.

Further, the problematic workers use 'openpower' firmware. That's some specific firmware different from what most machines use, and the little testing that is done by upstream on POWER8 likely happens on different firmware that also supports PowerVM.

Finally the virtualization team does not provide any support for KVM on Power at all. Any support we have is best-effort. I don't have any POWER8 hardware capable of running KVM available.

Let me try to use this channel. Maybe we can find an answer here to a question that was never properly answered elsewhere: What do you consider the best way to run virtual machine based tests on PowerPC and how to implement that? So far we always preferred qemu based tests because we can use the same on x86_64, aarch64, ppc64le (so far) as well as even s390x. Also this way we can scale best because machines can be created on the fly based on test parameters, e.g. RAM and storage size as needed for testing. For PowerVM we have a testing "backend" but it has very limited capabilities compared to bare-metal tests, e.g. interact with pre-configured LPARs and no support to save/load virtual machine images. And implementing that would require very specific PowerVM knowledge and any solution would only be valid for PowerPC. So, what is your take on it? -- You are receiving this mail because: You are the assignee for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1202138 http://bugzilla.opensuse.org/show_bug.cgi?id=1202138#c34 --- Comment #34 from Oliver Kurz --- (In reply to Michal Suchanek from comment #33)

If you want to use KVM on Power it's more likely to work on POWER9. It uses the new KVM code that upstream tests and actively develops.

Nonetheless, the virtualization team does not support it, and it may be broken from time to time, and take some time until fixed. Better chances than POWR8, though.

thank you. That is helpful information.

For PowerVM both HMC and Novalink is scriptable so it's very much possible to create VMs on the fly but it requires platform-specific implementation.

For storage and snapshots FC storage and iSCSI storage is supported by PowerVM making it possible to move the save/restore/... functionality outside of the test machine. Either requires extra hardware, the default 4x1Gbit NIC is not great for iSCSI, and FC is not available on most machines.

Yes. That's right. All those ideas are good but platform-specific so an expensive investment if we would continue with that.

It's also possible to use some solution that saves/restores the system over network in a platform-independent way, and it may be of use for testing baremetal as well if it ever gets implemented.

do you mean something like calling "dd" piped to "netcat" from a live-system? Because that's what I was commonly using to deploy systems easily including Microsoft Windows XP :D

Also iSCSI can be used on most hardware.

Management tools that make creating LPARs easier do exist but we did not make use of any so far, mostly because Orthos and openQA do the same thing in a crosss-platform way. It is failing on Power, though.

And s390 KVM is likely not making full use of the hardware capabilities, either.

Well, it's working fine for years and supports saving/loading VM images which is the important factor. -- You are receiving this mail because: You are the assignee for the bug.