[Bug 1160707] New: qemu VNC console corruption
http://bugzilla.opensuse.org/show_bug.cgi?id=1160707 Bug ID: 1160707 Summary: qemu VNC console corruption Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.1 Hardware: aarch64 URL: https://openqa.suse.de/tests/3773959/modules/update_ke rnel/steps/6 OS: Other Status: NEW Severity: Normal Priority: P5 - None Component: Virtualization:Other Assignee: virt-bugs@suse.de Reporter: okurz@suse.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- ## Observation openQA test in scenario sle-12-SP4-Server-DVD-Incidents-Kernel-aarch64-install_ltp+sle+Server-DVD-Incidents-Kernel@aarch64-virtio fails in [update_kernel](https://openqa.suse.de/tests/3773959/modules/update_kernel/steps/6) as a random problem sometimes appears on aarch64 test machines where the VM screen isn't properly cleared after boot and console output gets drawn over remnants of boot splash screen. Then the job fails because needles don't match. The problem appears less than once a week and job restart usually fixes it but it might be worth investigating further. ## Reproducible Sporadic ## Expected result Last good: [4.12.14-449.1.g4163e5c](https://openqa.suse.de/tests/3768894) (or more recent) ## Further details Always latest result in this scenario: [latest](https://openqa.suse.de/tests/latest?arch=aarch64&distri=sle&flavor=Server-DVD-Incidents-Kernel&machine=aarch64-virtio&test=install_ltp%2Bsle%2BServer-DVD-Incidents-Kernel&version=12-SP4) The problem appears on the osd (openqa.suse.de) aarch64 workers. OS is openSUSE Leap 15.1, qemu package versions: ``` $ sudo rpm -qa | grep qemu qemu-seabios-1.12.0-lp151.7.9.1.noarch qemu-arm-3.1.1.1-lp151.7.9.1.aarch64 qemu-ui-gtk-3.1.1.1-lp151.7.9.1.aarch64 qemu-ovmf-x86_64-0.1+svn17700-3.2.noarch qemu-3.1.1.1-lp151.7.9.1.aarch64 qemu-ipxe-1.0.0+-lp151.7.9.1.noarch qemu-block-rbd-3.1.1.1-lp151.7.9.1.aarch64 qemu-block-curl-3.1.1.1-lp151.7.9.1.aarch64 qemu-sgabios-8-lp151.7.9.1.noarch qemu-ui-sdl-3.1.1.1-lp151.7.9.1.aarch64 qemu-uefi-aarch64-2017+git1510945757.b2662641d5-lp151.10.1.noarch qemu-ui-curses-3.1.1.1-lp151.7.9.1.aarch64 qemu-ksm-3.1.1.1-lp151.7.9.1.aarch64 qemu-vgabios-1.12.0-lp151.7.9.1.noarch qemu-x86-3.1.1.1-lp151.7.9.1.aarch64 qemu-tools-3.1.1.1-lp151.7.9.1.aarch64 ``` ## Workaround Retrigger tests with new qemu instances -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160707 http://bugzilla.opensuse.org/show_bug.cgi?id=1160707#c6 Sergio Lindo Mansilla <slindomansilla@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |slindomansilla@suse.com --- Comment #6 from Sergio Lindo Mansilla <slindomansilla@suse.com> --- Related? https://progress.opensuse.org/issues/88203 visual artifacts from previous tty (user-console) when switching from serial terminal to tty (root-console) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160707 http://bugzilla.opensuse.org/show_bug.cgi?id=1160707#c7 Santiago Zarate <santiago.zarate@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lyan@suse.com, | |santiago.zarate@suse.com Flags| |needinfo?(lyan@suse.com) Severity|Normal |Major --- Comment #7 from Santiago Zarate <santiago.zarate@suse.com> --- (In reply to Sergio Lindo Mansilla from comment #6)
Related? https://progress.opensuse.org/issues/88203
visual artifacts from previous tty (user-console) when switching from serial terminal to tty (root-console)
(In reply to Liang Yan from comment #2)
Hi, Oliver, could you still hit it? I try to reproduce it but no luck.
It also happens somewhat often in SLE as well, https://openqa.suse.de/tests/latest?arch=aarch64&distri=sle&flavor=Online&machine=aarch64&test=minimal_x%2Buefi&version=15-SP3#step/consoletest_setup/63 Liang, have you had the chance to give it a look? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160707 Liang Yan <lyan@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|lyan@suse.com |virt-bugs@suse.de Flags|needinfo?(lyan@suse.com) | -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160707 Bruce Rogers <brogers@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|lyan@suse.com |brogers@suse.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160707 http://bugzilla.opensuse.org/show_bug.cgi?id=1160707#c21 Oliver Kurz <okurz@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Blocker|--- |Yes Flags|needinfo?(okurz@suse.com) | --- Comment #21 from Oliver Kurz <okurz@suse.com> --- Setting "Blocker" flag as this is actually impacting and aborting tests, e.g. https://openqa.suse.de/tests/6979012#step/prepare_test_data/6 Forgot to clear the needinfo flag as szarate already provided relevant information. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160707 http://bugzilla.opensuse.org/show_bug.cgi?id=1160707#c22 --- Comment #22 from Jos� Ricardo Ziviani <jose.ziviani@suse.com> --- (In reply to Oliver Kurz from comment #21)
Setting "Blocker" flag as this is actually impacting and aborting tests, e.g. https://openqa.suse.de/tests/6979012#step/prepare_test_data/6
Forgot to clear the needinfo flag as szarate already provided relevant information.
Hello Oliver, Looking at the screenshot, are you sure it's a QEMU error and not a VNC client issue? I'm telling it because I cannot reproduce it here (using QEMU only). So, do you have a QEMU reproducer (or at least the QEMU command line + the vnc client in use)? Thank you VERY much!! Jose R Ziviani -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160707 http://bugzilla.opensuse.org/show_bug.cgi?id=1160707#c23 Oliver Kurz <okurz@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(jose.ziviani@suse | |.com) | --- Comment #23 from Oliver Kurz <okurz@suse.com> --- (In reply to Jos� Ricardo Ziviani from comment #22)
(In reply to Oliver Kurz from comment #21)
Setting "Blocker" flag as this is actually impacting and aborting tests, e.g. https://openqa.suse.de/tests/6979012#step/prepare_test_data/6
Forgot to clear the needinfo flag as szarate already provided relevant information.
Hello Oliver,
Looking at the screenshot, are you sure it's a QEMU error and not a VNC client issue?
I'm telling it because I cannot reproduce it here (using QEMU only). So, do you have a QEMU reproducer (or at least the QEMU command line + the vnc client in use)?
Sure, you are right. It can very well be a VNC client issue. I unfortunately do not have a QEMU-only reproducer. The complete command line is available in every openQA job in the "downloads" section in the autoinst-log.txt file, e.g. from https://openqa.opensuse.org/tests/1941411/logfile?filename=autoinst-log.txt the complete command line would be: ``` /usr/bin/qemu-system-x86_64 -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.fdtypeA=none -m 2048 -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot once=d -device usb-ehci -device usb-tablet -smp 1 -enable-kvm -no-shutdown -vnc :109,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log,logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev pipe,id=virtio_console1,path=virtio_console1,logfile=virtio_console1.log,logappend=on -device virtconsole,chardev=virtio_console1,name=org.openqa.console.virtio_console1 -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log,logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-overlay0-file,filename=/var/lib/openqa/pool/19/raid/hd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0-overlay0,file=hd0-overlay0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0-overlay0,serial=hd0 ``` running qemu version 4.2.1 from openSUSE Leap 15.2, the VNC client by default is the os-autoinst builtin VNC-Console, see https://github.com/os-autoinst/os-autoinst/blob/39457352742edc264c5750795973... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1160707 http://bugzilla.opensuse.org/show_bug.cgi?id=1160707#c24 Claudio Fontana <claudio.fontana@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |claudio.fontana@suse.com --- Comment #24 from Claudio Fontana <claudio.fontana@suse.com> --- (In reply to Oliver Kurz from comment #23)
(In reply to Jos� Ricardo Ziviani from comment #22)
(In reply to Oliver Kurz from comment #21)
Setting "Blocker" flag as this is actually impacting and aborting tests, e.g. https://openqa.suse.de/tests/6979012#step/prepare_test_data/6
Forgot to clear the needinfo flag as szarate already provided relevant information.
Hello Oliver,
Looking at the screenshot, are you sure it's a QEMU error and not a VNC client issue?
I'm telling it because I cannot reproduce it here (using QEMU only). So, do you have a QEMU reproducer (or at least the QEMU command line + the vnc client in use)?
Sure, you are right. It can very well be a VNC client issue. I unfortunately do not have a QEMU-only reproducer. The complete command line is available in every openQA job in the "downloads" section in the autoinst-log.txt file, e.g. from https://openqa.opensuse.org/tests/1941411/logfile?filename=autoinst-log.txt the complete command line would be:
``` /usr/bin/qemu-system-x86_64 -only-migratable -chardev ringbuf,id=serial0,logfile=serial0,logappend=on -serial chardev:serial0 -audiodev none,id=snd0 -device intel-hda -device hda-output,audiodev=snd0 -global isa-fdc.fdtypeA=none -m 2048 -cpu host -netdev user,id=qanet0 -device virtio-net,netdev=qanet0,mac=52:54:00:12:34:56 -boot once=d -device usb-ehci -device usb-tablet -smp 1 -enable-kvm -no-shutdown -vnc :109,share=force-shared -device virtio-serial -chardev pipe,id=virtio_console,path=virtio_console,logfile=virtio_console.log, logappend=on -device virtconsole,chardev=virtio_console,name=org.openqa.console.virtio_console -chardev pipe,id=virtio_console1,path=virtio_console1,logfile=virtio_console1.log, logappend=on -device virtconsole,chardev=virtio_console1,name=org.openqa.console.virtio_console1 -chardev socket,path=qmp_socket,server,nowait,id=qmp_socket,logfile=qmp_socket.log, logappend=on -qmp chardev:qmp_socket -S -device virtio-scsi-pci,id=scsi0 -blockdev driver=file,node-name=hd0-overlay0-file,filename=/var/lib/openqa/pool/19/ raid/hd0-overlay0,cache.no-flush=on -blockdev driver=qcow2,node-name=hd0-overlay0,file=hd0-overlay0-file,cache.no-flush=on -device virtio-blk,id=hd0-device,drive=hd0-overlay0,serial=hd0 ```
running qemu version 4.2.1 from openSUSE Leap 15.2, the VNC client by default is the os-autoinst builtin VNC-Console, see https://github.com/os-autoinst/os-autoinst/blob/ 39457352742edc264c57507959734751aeb95aff/consoles/VNC.pm
This is a recurring pattern with problems detected with OpenQA: they are hard to reproduce, and the expectation seems to be to just "run OpenQA", and fetch artifacts from there. As discussed previously also with QA, we are not resourced for this kind of operation. Please provide a minimal local reproducer for QEMU to Jose if you suspect qemu to be the issue, otherwise it might be helpful to start exploring other components (vnc client, X11 stack etc). Thanks -- You are receiving this mail because: You are on the CC list for the bug.
participants (2)
-
bugzilla_noreply@novell.com
-
bugzilla_noreply@suse.com