[Bug 1193019] New: BTRFS i/o error on Tumbleweed VM on Leap 15.3
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 Bug ID: 1193019 Summary: BTRFS i/o error on Tumbleweed VM on Leap 15.3 Classification: openSUSE Product: openSUSE Distribution Version: Leap 15.3 Hardware: x86-64 OS: openSUSE Leap 15.3 Status: NEW Severity: Major Priority: P5 - None Component: Virtualization:Tools Assignee: virt-bugs@suse.de Reporter: kokeko@mac.com QA Contact: qa-bugs@suse.de Found By: --- Blocker: --- Hi I'm running Tumbleweed/MicroOS VM on Leap 15.3 Physical Machine. Qemu version is 5.2.0-106.4. On transactional-update, VM console report following message, and abort update. [ 1979.965517][ C1] BTRFS error (device vda2): bdev /dev/vda2 errs: wr 897, rd 0, flush 0, corrupt 0, gen 0 [ 1979.968185][ C1] BTRFS error (device vda2): bdev /dev/vda2 errs: wr 898, rd 0, flush 0, corrupt 0, gen 0 [ 1979.970776][ C1] blk_update_request: I/O error, dev vda, sector 75757672 op 0x1:(WRITE) flags 0x104000 phys_seg 137 prio class 0 [ 1979.973873][ C1] blk_update_request: I/O error, dev vda, sector 75760232 op 0x1:(WRITE) flags 0x100000 phys_seg 120 prio class 0 [ 1979.976937][ C1] BTRFS error (device vda2): bdev /dev/vda2 errs: wr 899, rd 0, flush 0, corrupt 0, gen 0 [ 1979.979682][ C1] blk_update_request: I/O error, dev vda, sector 75761736 op 0x1:(WRITE) flags 0x104000 phys_seg 254 prio class 0 [ 1979.982415][ C1] BTRFS error (device vda2): bdev /dev/vda2 errs: wr 900, rd 0, flush 0, corrupt 0, gen 0 [ 1979.985079][ C1] BTRFS error (device vda2): bdev /dev/vda2 errs: wr 901, rd 0, flush 0, corrupt 0, gen 0 [ 1979.987830][ C1] BTRFS error (device vda2): bdev /dev/vda2 errs: wr 902, rd 0, flush 0, corrupt 0, gen 0 [ 1979.990679][ C1] BTRFS error (device vda2): bdev /dev/vda2 errs: wr 903, rd 0, flush 0, corrupt 0, gen 0 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 Michiya Hagimoto <kokeko@mac.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kokeko@mac.com -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c1 --- Comment #1 from Michiya Hagimoto <kokeko@mac.com> --- On VM transaction-update message. ( 768/1526) Installing: kernel-source-5.15.3-1.3.noarch [...........................error] Installation of kernel-source-5.15.3-1.3.noarch failed: Error: Subprocess failed. Error: RPM failed: error: unpacking of archive failed on file /usr/src/linux-5.15.3-1/drivers/scsi/aic7xxx/aic79xx.seq;619dc01e: cpio: read failed - Directory not empty error: kernel-source-5.15.3-1.3.noarch: install failed error: kernel-source-5.14.14-1.1.noarch: erase skipped Abort, retry, ignore? [a/r/i] (a): a Warning: %posttrans scripts skipped while aborting: login_defs-4.9-2.1.noarch.rpm coreutils-9.0-2.1.x86_64.rpm grub2-i386-pc-2.06-10.1.noarch.rpm ucode-amd-20211115-1.1.noarch.rpm sg3_utils-1.47-1.1.x86_64.rpm kernel-firmware-usb-network-20211115-1.1.noarch.rpm grub2-x86_64-efi-2.06-10.1.noarch.rpm grub2-branding-openSUSE-84.87.20200106-4.46.noarch.rpm kernel-firmware-ueagle-20211115-1.1.noarch.rpm kernel-firmware-ti-20211115-1.1.noarch.rpm java-11-openjdk-headless-11.0.13.0-2.1.x86_64.rpm kernel-firmware-sound-20211115-1.1.noarch.rpm kernel-firmware-serial-20211115-1.1.noarch.rpm texlive-2021.20210325-77.2.x86_64.rpm kernel-firmware-realtek-20211115-1.1.noarch.rpm kernel-firmware-radeon-20211115-1.1.noarch.rpm pam-1.5.2-3.1.x86_64.rpm Problem occurred during or after installation or removal of packages: Installation has been aborted as directed. Please see the above error message for a hint. 2021-11-24 13:32:14 Application returned with exit status 8. ERROR: zypper --no-cd dup on /.snapshots/255/snapshot failed with exit code 8! Use '--interactive' for manual problem resolution. Warning: The following files were changed in the snapshot, but are shadowed by other mounts and will not be visible to the system: /.snapshots/255/snapshot/var/lib/misc/Makefile Removing snapshot #255... 2021-11-24 13:32:15 tukit 3.6.2 started 2021-11-24 13:32:15 Options: abort 255 2021-11-24 13:32:15 Discarding snapshot 255. 2021-11-24 13:32:15 Transaction completed. transactional-update finished -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c4 Dario Faggioli <dfaggioli@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dfaggioli@suse.com Flags| |needinfo?(kokeko@mac.com) --- Comment #4 from Dario Faggioli <dfaggioli@suse.com> --- (In reply to Takashi Iwai from comment #3)
Sounds rather like a VM issue than a filesystem problem. Let's try to reassign to our KVM people at first.
Well, perhaps. FWIWI, I've just created a MicroOS VM on a SLE 15-SP3 host (which shouldn't be too different from a Leap 15.3, which I currently don't have handy), and things work fine for me. In any case: - how reproducible is the problem? - Does it happen all the times that a specific command is invoked in the guest? - If yes, what command? - The description mentions transactional-update, does this mean that the problem occurs all the time transactional-update is used? Or does it only happen when doing something specific with it? Also, can we see: - some logs from the host (e.g., dmesg, lscpu, lsblk) - the VM config file - the content of /var/log/libvirt/qemu/<guest_name>.log -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c5 --- Comment #5 from Michiya Hagimoto <kokeko@mac.com> --- Hi - how reproducible is the problem? This is a probability and can be very rare. Large updates such as kernel updates will fail, but lightweight (MB order) will usually succeed. - Does it happen all the times that a specific command is invoked in the guest? Updates with the dup command usually fail. There is no 100% reproducibility, and will it fail at about 90% sensuously? - If yes, what command? Repeating small updates with the pkg in command may work, but kernel updates are prone to failure. - The description mentions transactional-update, does this mean that the problem occurs all the time transactional-update is used? Or does it only happen when doing something specific with it? I didn't use a disk image in my environment and didn't mention that I'm using ZVOL (ZFS volume feature) for my block device, I'm sorry. However, even in this environment, this error started to appear after updating QEMU. Check if can I log after applying today's update. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c6 --- Comment #6 from Michiya Hagimoto <kokeko@mac.com> --- Host's dmesg, No kvm, libvirt message. [ 1469.025818] systemd-sysv-generator[19755]: SysV service '/etc/init.d/xfs' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. [ 1470.268921] systemd-sysv-generator[19970]: SysV service '/etc/init.d/xfs' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. [ 1471.576149] systemd-sysv-generator[20200]: SysV service '/etc/init.d/xfs' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. [ 1472.836190] systemd-sysv-generator[20467]: SysV service '/etc/init.d/xfs' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. [ 1473.901906] systemd-sysv-generator[20750]: SysV service '/etc/init.d/xfs' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. [ 1474.898315] systemd-sysv-generator[21119]: SysV service '/etc/init.d/xfs' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. [ 1476.333813] systemd-sysv-generator[21550]: SysV service '/etc/init.d/xfs' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. [ 1477.284747] systemd-sysv-generator[21848]: SysV service '/etc/init.d/xfs' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. [ 1478.394173] systemd-sysv-generator[22248]: SysV service '/etc/init.d/xfs' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. [ 1479.573707] systemd-sysv-generator[22620]: SysV service '/etc/init.d/xfs' lacks a native systemd unit file. Automatically generating a unit file for compatibility. Please update package to include a native systemd unit file, in order to make it more safe and robust. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c7 --- Comment #7 from Michiya Hagimoto <kokeko@mac.com> --- Created attachment 854231 --> http://bugzilla.opensuse.org/attachment.cgi?id=854231&action=edit VM xml file -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 Michiya Hagimoto <kokeko@mac.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(kokeko@mac.com) | -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c8 --- Comment #8 from Michiya Hagimoto <kokeko@mac.com> --- Created attachment 854232 --> http://bugzilla.opensuse.org/attachment.cgi?id=854232&action=edit VM log -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c9 --- Comment #9 from Michiya Hagimoto <kokeko@mac.com> --- I use Arch Linux(XFS) and Clear Linux(ext4) on same physical machine, but no I/O error another machine. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c10 --- Comment #10 from Michiya Hagimoto <kokeko@mac.com> --- Today, try to ZFS setting and Libvirt xml seting change. cache='none'->cache='wirteback' : NG same IO error. io='io_uring'->io='native' : NG same IO error. ZFS ARC(cache) setting change : NG same IO error. I think it is QEMU block device driver(virtio) error? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c11 --- Comment #11 from Dario Faggioli <dfaggioli@suse.com> --- (In reply to Michiya Hagimoto from comment #10)
Today, try to ZFS setting and Libvirt xml seting change.
cache='none'->cache='wirteback' : NG same IO error. io='io_uring'->io='native' : NG same IO error. ZFS ARC(cache) setting change : NG same IO error.
I think it is QEMU block device driver(virtio) error?
Well, yes, it could be. TBH, io_uring on top of ZFS (and on a not super new software stack such as the one in Leap) is not a configuration I'm familiar with, and I'm quite sure it's not used/tested that often. Now, the experiments above show that probably io_uring is not among the causes, as you have the problem even without it. It could be QEMU but I guess it could even be BTRFS (or kernel in general) inside the guest, maybe due to how they interact with ZFS (which I really don't know much about). In fact, you mention that other distros with different filesystems work fine, right? And QEMU is the same among all of them, isn't it? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c12 --- Comment #12 from Michiya Hagimoto <kokeko@mac.com> --- Yes, all guests use the same Qemu. I use multiple distributions in my virtual machine for safety, but I've had the same problem with other distros using BTRFS. Different filesystems such as EXT4 and XFS, F2FS have at least not errored enough to fail. There was an update to Qemu last month, and since then this issue has surfaced, and I suspect that something has changed there. At least it's been stable for two years since I built this environment with ZFS in 2019, but it's been volatile for the past month. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c13 --- Comment #13 from Dario Faggioli <dfaggioli@suse.com> --- (In reply to Michiya Hagimoto from comment #12)
Yes, all guests use the same Qemu. There was an update to Qemu last month, and since then this issue has surfaced, and I suspect that something has changed there.
Ok, do you happen to remember (or have in some logs, like zypper history, etc) from which version to which version the update was? That may help us understand whether the error could be there and, if yes, to poinpoint some specific changes or patches -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c14 --- Comment #14 from Michiya Hagimoto <kokeko@mac.com> --- (In reply to Dario Faggioli from comment #13)
(In reply to Michiya Hagimoto from comment #12)
Yes, all guests use the same Qemu. There was an update to Qemu last month, and since then this issue has surfaced, and I suspect that something has changed there.
Ok, do you happen to remember (or have in some logs, like zypper history, etc) from which version to which version the update was?
That may help us understand whether the error could be there and, if yes, to poinpoint some specific changes or patches
There is no specific log, but I think the update is as follows. From qemu-5.2.0-103.2.x86_64.rpm To qemu-5.2.0-106.4.x86_64.rpm No update from here, still problem happen. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c15 Dario Faggioli <dfaggioli@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |lma@suse.com Flags| |needinfo?(lma@suse.com) --- Comment #15 from Dario Faggioli <dfaggioli@suse.com> --- (In reply to Michiya Hagimoto from comment #14)
(In reply to Dario Faggioli from comment #13)
(In reply to Michiya Hagimoto from comment #12)
Yes, all guests use the same Qemu. There was an update to Qemu last month, and since then this issue has surfaced, and I suspect that something has changed there.
Ok, do you happen to remember (or have in some logs, like zypper history, etc) from which version to which version the update was?
From qemu-5.2.0-103.2.x86_64.rpm To qemu-5.2.0-106.4.x86_64.rpm
Ok, thanks! If that's the case, this is what happened to the package between those two versions: * Thu Sep 30 2021 jose.ziviani@suse.com - Fix out-of-bounds write in UAS (USB Attached SCSI) device emulation (bsc#1189702, CVE-2021-3713) uas-add-stream-number-sanity-checks.patch - Fix heap use-after-free in virtio_net_receive_rcu (bsc#1189938, CVE-2021-3748) virtio-net-fix-use-after-unmap-free-for-.patch * Tue Sep 14 2021 lma@suse.com - Add transfer length item in block limits page of scsi vpd (bsc#1190425) * Patches added: block-add-max_hw_transfer-to-BlockLimits.patch block-backend-align-max_transfer-to-requ.patch file-posix-fix-max_iov-for-dev-sg-device.patch file-posix-try-BLKSECTGET-on-block-devic.patch osdep-provide-ROUND_DOWN-macro.patch scsi-generic-pass-max_segments-via-max_i.patch * Fri Sep 03 2021 lma@suse.com - Fix qemu crash while deleting xen-block (bsc#1189234) * Patches added: xen-remove-BlockBackend-object-reference.patch I can try to provide test packages with some of these patches removed, as soon as I find the time for it. In the meanwhile, Ma Lin, do you think any of those "block-" patches you added (or others) could be responsible for soemthing like this? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c16 --- Comment #16 from Michiya Hagimoto <kokeko@mac.com> --- This is my personal opinion, but when I look at the log on the guest side, I can read that 0 can be read even though I wrote it. [1979.965517] [C1] BTRFS error (device vda2): bdev / dev / vda2 errs: wr 897, rd 0, flush 0, corrupt 0, gen 0 The opposite of Use after free, isn't it read before the writing is reflected? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c17 Lin Ma <lma@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(lma@suse.com) | --- Comment #17 from Lin Ma <lma@suse.com> --- (In reply to Dario Faggioli from comment #15)
(In reply to Michiya Hagimoto from comment #14)
(In reply to Dario Faggioli from comment #13)
(In reply to Michiya Hagimoto from comment #12)
Yes, all guests use the same Qemu. There was an update to Qemu last month, and since then this issue has surfaced, and I suspect that something has changed there.
Ok, do you happen to remember (or have in some logs, like zypper history, etc) from which version to which version the update was?
From qemu-5.2.0-103.2.x86_64.rpm To qemu-5.2.0-106.4.x86_64.rpm
Ok, thanks!
If that's the case, this is what happened to the package between those two versions:
* Thu Sep 30 2021 jose.ziviani@suse.com - Fix out-of-bounds write in UAS (USB Attached SCSI) device emulation (bsc#1189702, CVE-2021-3713) uas-add-stream-number-sanity-checks.patch - Fix heap use-after-free in virtio_net_receive_rcu (bsc#1189938, CVE-2021-3748) virtio-net-fix-use-after-unmap-free-for-.patch
* Tue Sep 14 2021 lma@suse.com - Add transfer length item in block limits page of scsi vpd (bsc#1190425) * Patches added: block-add-max_hw_transfer-to-BlockLimits.patch block-backend-align-max_transfer-to-requ.patch file-posix-fix-max_iov-for-dev-sg-device.patch file-posix-try-BLKSECTGET-on-block-devic.patch osdep-provide-ROUND_DOWN-macro.patch scsi-generic-pass-max_segments-via-max_i.patch
* Fri Sep 03 2021 lma@suse.com - Fix qemu crash while deleting xen-block (bsc#1189234) * Patches added: xen-remove-BlockBackend-object-reference.patch
I can try to provide test packages with some of these patches removed, as soon as I find the time for it.
In the meanwhile, Ma Lin, do you think any of those "block-" patches you added (or others) could be responsible for soemthing like this?
I happened to dig into this issue as well in days, So far I can't reproduce it either. IMO, The above patches(including mine and others) have nothing to do with this issue. The patches which were added at Sep 14 are about scsi vpd emulation, The vm xml in comment#7 shows it uses virtio-blk, no any scsi HBAs. Anyway, I'll keep looking into it. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c18 --- Comment #18 from Dario Faggioli <dfaggioli@suse.com> --- (In reply to Michiya Hagimoto from comment #14)
There is no specific log, but I think the update is as follows.
From qemu-5.2.0-103.2.x86_64.rpm To qemu-5.2.0-106.4.x86_64.rpm
One more thing, though. I guess that in a month, quite a few updates/changes happened inside the guest as well (it being MicroOS --> Tumbleweed), so there is still the chance for the problem to be somewehre else too (or for it to be due to a combination of multiple factors). Are you able to re-install that qemu-5.2.0-103.2.x86_64.rpm package and test it with the current state of the VM? That would tell us with even more confidence whether the problem is really in the new QEMU... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c19 --- Comment #19 from Dario Faggioli <dfaggioli@suse.com> --- (In reply to Lin Ma from comment #17)
(In reply to Dario Faggioli from comment #15)
* Thu Sep 30 2021 jose.ziviani@suse.com - Fix out-of-bounds write in UAS (USB Attached SCSI) device emulation (bsc#1189702, CVE-2021-3713) uas-add-stream-number-sanity-checks.patch - Fix heap use-after-free in virtio_net_receive_rcu (bsc#1189938, CVE-2021-3748) virtio-net-fix-use-after-unmap-free-for-.patch
* Tue Sep 14 2021 lma@suse.com - Add transfer length item in block limits page of scsi vpd (bsc#1190425) * Patches added: block-add-max_hw_transfer-to-BlockLimits.patch block-backend-align-max_transfer-to-requ.patch file-posix-fix-max_iov-for-dev-sg-device.patch file-posix-try-BLKSECTGET-on-block-devic.patch osdep-provide-ROUND_DOWN-macro.patch scsi-generic-pass-max_segments-via-max_i.patch
* Fri Sep 03 2021 lma@suse.com - Fix qemu crash while deleting xen-block (bsc#1189234) * Patches added: xen-remove-BlockBackend-object-reference.patch
IMO, The above patches(including mine and others) have nothing to do with this issue.
The patches which were added at Sep 14 are about scsi vpd emulation, The vm xml in comment#7 shows it uses virtio-blk, no any scsi HBAs.
Yep, I saw "block-" and figured there might be a relationship, but now I checked both the patches themselves and the issue they are fixing, and indeed they seem unrelated. Therefore, a test with the "old" QEMU would IMO be more relevant than one done with a special QEMU where we remove patches which likely don't have anything to do with the problem, I think -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c20 --- Comment #20 from Lin Ma <lma@suse.com> --- (In reply to Dario Faggioli from comment #18)
(In reply to Michiya Hagimoto from comment #14)
There is no specific log, but I think the update is as follows.
From qemu-5.2.0-103.2.x86_64.rpm To qemu-5.2.0-106.4.x86_64.rpm
One more thing, though. I guess that in a month, quite a few updates/changes happened inside the guest as well (it being MicroOS --> Tumbleweed), so there is still the chance for the problem to be somewehre else too (or for it to be due to a combination of multiple factors).
Are you able to re-install that qemu-5.2.0-103.2.x86_64.rpm package and test it with the current state of the VM?
That would tell us with even more confidence whether the problem is really in the new QEMU...
This is what I thought as well :-) -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c21 --- Comment #21 from Michiya Hagimoto <kokeko@mac.com> --- OK, I try it. Install old version Qemu and downgrade package. zypper install --oldpackage qemu-5.2.0-103.2.x86_64 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c22 --- Comment #22 from Michiya Hagimoto <kokeko@mac.com> --- 5.2.0-103.2 is NG, Same error on guest console. I try more rollback qemu package. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c23 --- Comment #23 from Michiya Hagimoto <kokeko@mac.com> --- rollback more old qemu package. zypper install --oldpackage qemu-5.2.0-23.1.x86_64 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c24 --- Comment #24 from Michiya Hagimoto <kokeko@mac.com> --- 5.2.0-23.1.x86_64 is NG. I try more rollback qemu package. However, I feel that the frequency of error occurrence is decreasing. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c25 --- Comment #25 from Michiya Hagimoto <kokeko@mac.com> --- Rollback to 5.2.0-20.1 It is from 15.3-2 main repo. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c26 --- Comment #26 from Michiya Hagimoto <kokeko@mac.com> --- Now rollback to 20.1, testtisg zypper install --oldpackage qemu-5.2.0-20.1.x86_64 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c27 --- Comment #27 from Michiya Hagimoto <kokeko@mac.com> --- Unfortunately, I'm still getting the same error when I revert to this version. However, since the frequency is decreasing, this itself seems to have some effect, but the underlying reason is different. If one transitional update fails, all will be discarded, so let's see if we can partially update it. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c28 --- Comment #28 from Lin Ma <lma@suse.com> --- (In reply to Michiya Hagimoto from comment #27)
Unfortunately, I'm still getting the same error when I revert to this version. However, since the frequency is decreasing, this itself seems to have some effect, but the underlying reason is different.
If one transitional update fails, all will be discarded, so let's see if we can partially update it.
First of all, Thanks for your cooperation to run those tests with various qemu version! Due to the frequency is decreasing with 5.2.0-23.1 and 5.2.0-20.1, Could you please help to run the following tests with qemu 5.2.0-103.2? test 1: * Still use ovmf-x86_64-smm-ms-code.bin as firmware. * Change disk backend: Use file image(e.g. qcow2 or raw) instead of block device. test 2: * Still use zfs block device. * Change firmware: Use bios rather than ovmf test 3: * Still use zfs block device. * Still use ovmf-x86_64-smm-ms-code.bin as firmware. * Change virtual storage controller: Use virtio-scsi instead of virtio-blk. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c29 --- Comment #29 from Lin Ma <lma@suse.com> --- (In reply to Michiya Hagimoto from comment #27)
Unfortunately, I'm still getting the same error when I revert to this version. However, since the frequency is decreasing, this itself seems to have some effect, but the underlying reason is different.
If one transitional update fails, all will be discarded, so let's see if we can partially update it.
I'm afraid that the issue will be harder to reproduce if you only perform partially update. I'd like to add test 4 with qemu 5.2.0-103.2: test 4: * keep everything unchange. * update rpm alone. * update rest of packages with the dup command. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c30 --- Comment #30 from Michiya Hagimoto <kokeko@mac.com> --- OK, I'll try it. Now lock qemu version is 5.2.0-103.2, and test. Test4 -> Test3 -> Test2 -> Test1 Because, my system infra(snapshot system and backup) depend to ZFS function, Using ZFS is a high demand for convenience. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c31 --- Comment #31 from Michiya Hagimoto <kokeko@mac.com> --- Do you think the presence or absence of SMM mode has an effect on OVMF? Switching to OVMF non-SMM has less impact than switching to BIOS mode, so I'd like to try this. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c32 --- Comment #32 from Michiya Hagimoto <kokeko@mac.com> --- Test4 is fail. No rpm update, dup is fail... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c33 --- Comment #33 from Michiya Hagimoto <kokeko@mac.com> --- test3 is fail. I change drive by virtio-scsi, but same i/o error happen... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c34 --- Comment #34 from Michiya Hagimoto <kokeko@mac.com> --- test2 is failed. Change OVMF to BIOS and run transactional-update, same i/o error happen on guest console, abort transactional-update. test1 cannot exec. I try to qcow2 file on my zfs storage, because /var/lib/libvirt/images on ZFS. but cannot boot qemu on my enviroment, same as raw file... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c35 --- Comment #35 from Michiya Hagimoto <kokeko@mac.com> --- I'm sorry, none of the suggested methods worked in my environment. I also tested it in a more compact environment and got the same results. Should I revert to Qemu before 15.3-2? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c36 --- Comment #36 from Dario Faggioli <dfaggioli@suse.com> --- (In reply to Michiya Hagimoto from comment #35)
I'm sorry, none of the suggested methods worked in my environment. I also tested it in a more compact environment and got the same results. Should I revert to Qemu before 15.3-2?
I can't be 100% sure, but I think it's time to start to think that this is due to something that is happening inside the guest. Not necessarily just a BTRFS bug, as I run MicroOS myself on different kind of real hardware and in VMs, and I don't see the problem. But maybe something that manifests when BTRFS (or some other block-related kernel component, or whatever) works in combination with whatever else is involved in your configuration. I think a way to test this would be to rollback the guest, if you still have the snapshots available, to a month ago or in general to a previous state wrt when the problems started to show up, but I'm curious to hear what Ma Lin thinks about this... It may also be useful to start to bring in some block and/or filesystem people, to see if they have ideas about how to proceed further. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c37 --- Comment #37 from Michiya Hagimoto <kokeko@mac.com> --- I try to rollback to 1month ago snapshot (more older snapshot remeved). but same i/o error happen. If it have btrfs structure broken? Do you know something scrub/fsck on guest machine. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c38 --- Comment #38 from Michiya Hagimoto <kokeko@mac.com> --- I try BTRFS scrub on guest on transactional-update shell. repot following message. WARNING: errors detected during scrubbing, corrected btrfs scrub status / UUID: b6484f21-0b8b-4109-8150-117bb040b496 Scrub started: Sat Dec 4 13:54:38 2021 Status: finished Duration: 0:00:06 Total to scrub: 1.59GiB Rate: 270.93MiB/s Error summary: no errors found No error? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c39 --- Comment #39 from Michiya Hagimoto <kokeko@mac.com> --- 3 times scrub test repot i/o error every scrub. I think no guest problem. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c40 --- Comment #40 from Michiya Hagimoto <kokeko@mac.com> --- Use btrfs check find leak btrfs check --force /dev/vda2 Opening filesystem to check... WARNING: filesystem mounted, continuing because of --force Checking filesystem on /dev/vda2 UUID: 6345bd37-2527-41c5-9809-1c6e97b45157 [1/7] checking root items [2/7] checking extents [3/7] checking free space tree [4/7] checking fs roots [5/7] checking only csums items (without verifying data) [6/7] checking root refs [7/7] checking quota groups ERROR: out of memory ERROR: Loading qgroups from disk: -2 ERROR: failed to check quota groups found 2016813056 bytes used, error(s) found total csum bytes: 1863848 total tree bytes: 108232704 total fs tree bytes: 98877440 total extent tree bytes: 6553600 btree space waste bytes: 24137918 file data blocks allocated: 10962644992 referenced 5847863296 extent buffer leak: start 35127296 len 16384 extent buffer leak: start 35192832 len 16384 I try to shutdown VM and repair this volume on host machine. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c41 --- Comment #41 from Michiya Hagimoto <kokeko@mac.com> --- Repair on host. btrfs check --repair /dev/zvol/pool/guest/bastion/microos-part2 enabling repair mode Opening filesystem to check... Checking filesystem on /dev/zvol/pool/guest/bastion/microos-part2 UUID: b6484f21-0b8b-4109-8150-117bb040b496 [1/7] checking root items Fixed 0 roots. [2/7] checking extents No device size related problem found [3/7] checking free space tree cache and super generation don't match, space cache will be invalidated [4/7] checking fs roots [5/7] checking only csums items (without verifying data) [6/7] checking root refs [7/7] checking quota groups ERROR: out of memory ERROR: Loading qgroups from disk: -2 ERROR: failed to check quota groups found 1599578112 bytes used, error(s) found total csum bytes: 1458440 total tree bytes: 106135552 total fs tree bytes: 97959936 total extent tree bytes: 5898240 btree space waste bytes: 25762174 file data blocks allocated: 29529686016 referenced 4831297536 extent buffer leak: start 30818304 len 16384 extent buffer leak: start 58867712 len 16384 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c42 --- Comment #42 from Michiya Hagimoto <kokeko@mac.com> --- I try MicroOS btrfs check by leap btrfs-progs. Is it fail? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c43 --- Comment #43 from Michiya Hagimoto <kokeko@mac.com> --- Today, host kernel update to 5.3.18-59.37-default. Now try to update test. Small update is success, I try to more big update. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c44 --- Comment #44 from Michiya Hagimoto <kokeko@mac.com> --- Oh... Big update is failed. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c45 --- Comment #45 from Michiya Hagimoto <kokeko@mac.com> --- Leap VM is update success(no i/o error on console), Tumbleweed/MicroOS VM update failed. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c46 --- Comment #46 from Lin Ma <lma@suse.com> --- Considering the tumbleweed upgrading's been stable for two years and been volatile in past month, After repairing volumes on guest/host, I'd like to see the result of upgrading about targeting to an older snapshot instead of the latest snapshot. E.g: 1. Install tumbleweed-cli package in the guest. 2. Initialize and get snapshot list. guest:~ # /usr/bin/tumbleweed init guest:~ # /usr/bin/tumbleweed list 3. Switch to an old snapshot. guest:~ # /usr/bin/tumbleweed switch 20211110 4. Upgrade: guest:~ # zypper ref guest:~ # zypper dup BTW, reverting to qemu before 15.3-2 in host is unnecessary, It seems not helpful. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c47 --- Comment #47 from Michiya Hagimoto <kokeko@mac.com> --- Uh, something wrong, Leap VM update to latest version, but Tumbleweed/MicroOS VM cannot update. Rollback Qemu to 5.2.0-17.1, It is minimal version of Leap repo. But same i/o error is happen. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c48 --- Comment #48 from Michiya Hagimoto <kokeko@mac.com> --- try to rollback libvirt. libvirt-daemon-driver-qemu-7.1.0-4.1 -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c49 --- Comment #49 from Lin Ma <lma@suse.com> --- (In reply to Michiya Hagimoto from comment #48)
try to rollback libvirt.
libvirt-daemon-driver-qemu-7.1.0-4.1
Hmm...I don't think rollbacking libvirt is helpful. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c50 --- Comment #50 from Michiya Hagimoto <kokeko@mac.com> --- (In reply to Lin Ma from comment #49)
Hmm...I don't think rollbacking libvirt is helpful.
libvirt is only generate qemu's argument? Not for control binary? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c51 --- Comment #51 from Michiya Hagimoto <kokeko@mac.com> --- Uhhh, same i/o error, happen... -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c52 --- Comment #52 from Dario Faggioli <dfaggioli@suse.com> --- (In reply to Michiya Hagimoto from comment #50)
(In reply to Lin Ma from comment #49)
Hmm...I don't think rollbacking libvirt is helpful.
libvirt is only generate qemu's argument? Not for control binary?
Well, yes, it's more or less like that -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c53 --- Comment #53 from Michiya Hagimoto <kokeko@mac.com> --- (In reply to Dario Faggioli from comment #52)
(In reply to Michiya Hagimoto from comment #50)
(In reply to Lin Ma from comment #49)
Hmm...I don't think rollbacking libvirt is helpful.
libvirt is only generate qemu's argument? Not for control binary?
Well, yes, it's more or less like that
Thanks, I rollback libvirt Version but same i/o error happen, Return to latest version. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c54 --- Comment #54 from Michiya Hagimoto <kokeko@mac.com> --- (In reply to Lin Ma from comment #28)
test 3: * Still use zfs block device. * Still use ovmf-x86_64-smm-ms-code.bin as firmware. * Change virtual storage controller: Use virtio-scsi instead of virtio-blk.
Today, I try again change virtio-blk to virtio-scsi, Success!!!!!! virtio-scsi is more stable and scaleable, I will change all device to virtio-scsi. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c55 --- Comment #55 from Lin Ma <lma@suse.com> --- (In reply to Michiya Hagimoto from comment #54)
(In reply to Lin Ma from comment #28)
test 3: * Still use zfs block device. * Still use ovmf-x86_64-smm-ms-code.bin as firmware. * Change virtual storage controller: Use virtio-scsi instead of virtio-blk.
Today, I try again change virtio-blk to virtio-scsi, Success!!!!!! virtio-scsi is more stable and scaleable, I will change all device to virtio-scsi.
But according to comment#33, The same i/o error did use to happen with virtio-scsi, didn't it? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c56 --- Comment #56 from Michiya Hagimoto <kokeko@mac.com> --- (In reply to Lin Ma from comment #55)
(In reply to Michiya Hagimoto from comment #54)
(In reply to Lin Ma from comment #28)
test 3: * Still use zfs block device. * Still use ovmf-x86_64-smm-ms-code.bin as firmware. * Change virtual storage controller: Use virtio-scsi instead of virtio-blk.
Today, I try again change virtio-blk to virtio-scsi, Success!!!!!! virtio-scsi is more stable and scaleable, I will change all device to virtio-scsi.
But according to comment#33, The same i/o error did use to happen with virtio-scsi, didn't it?
Yes, but some update (mainly kernel) after #33. I think something change? -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c57 --- Comment #57 from Lin Ma <lma@suse.com> --- (In reply to Michiya Hagimoto from comment #56)
(In reply to Lin Ma from comment #55)
(In reply to Michiya Hagimoto from comment #54)
(In reply to Lin Ma from comment #28)
test 3: * Still use zfs block device. * Still use ovmf-x86_64-smm-ms-code.bin as firmware. * Change virtual storage controller: Use virtio-scsi instead of virtio-blk.
Today, I try again change virtio-blk to virtio-scsi, Success!!!!!! virtio-scsi is more stable and scaleable, I will change all device to virtio-scsi.
But according to comment#33, The same i/o error did use to happen with virtio-scsi, didn't it?
Yes, but some update (mainly kernel) after #33. I think something change?
The latest snapshot of tumbleweed is '20211205', Could you please help to copy/paste the following output from that VM? (the one with virtio-scsi and successfully upgraded) 1. Install tumbleweed-cli package in the VM. 2. guest:~ # /usr/bin/tumbleweed init && /usr/bin/tumbleweed installed -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c58 --- Comment #58 from Michiya Hagimoto <kokeko@mac.com> --- (In reply to Lin Ma from comment #57)
(In reply to Michiya Hagimoto from comment #56)
(In reply to Lin Ma from comment #55)
(In reply to Michiya Hagimoto from comment #54)
(In reply to Lin Ma from comment #28)
1. Install tumbleweed-cli package in the VM. 2. guest:~ # /usr/bin/tumbleweed init && /usr/bin/tumbleweed installed
Is this tool use on transactional-server pattern? I try to 1years ago, It couldnot work. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c59 --- Comment #59 from Lin Ma <lma@suse.com> --- It's in the 'repo-oss' E.g: tw-guest:~ # zypper lr repo-oss Alias : repo-oss Name : openSUSE-Tumbleweed-Oss URI : http://download.opensuse.org/tumbleweed/repo/oss/ Enabled : Yes ...... tw-guest:~ # zypper info tumbleweed-cli Loading repository data... Reading installed packages... Information for package tumbleweed-cli: --------------------------------------- Repository : openSUSE-Tumbleweed-Oss Name : tumbleweed-cli Version : 0.3.3-1.6 Arch : noarch Vendor : openSUSE Installed Size : 10.3 KiB Installed : No Status : not installed Source package : tumbleweed-cli-0.3.3-1.6.src Summary : Command line interface for interacting with tumbleweed snapshots Description : tumbleweed-cli provides a command line interface for interacting with tumbleweed snapshots. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c60 --- Comment #60 from Michiya Hagimoto <kokeko@mac.com> --- Thanks for information. update tw 20211104 to 20211205, success. I think BLK doesn't have to be usable if the virtual machine works this way. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c62 --- Comment #62 from Michiya Hagimoto <kokeko@mac.com> --- (In reply to Dario Faggioli from comment #61)
Err... Right, but I don't think tumbluweed-cli and microos, which uses transactional-update, should be mixed (not even if it works, which I don't think)
I agree to you, tumbleweed-cli for zypper update system, not for transactional-update. Now I change virtio-scsi on my VMs. -- You are receiving this mail because: You are on the CC list for the bug.
http://bugzilla.opensuse.org/show_bug.cgi?id=1193019 http://bugzilla.opensuse.org/show_bug.cgi?id=1193019#c64 --- Comment #64 from Michiya Hagimoto <kokeko@mac.com> --- Everything fine to change virtio-scsi. Thanks for support. -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠suse.com