http://bugzilla.suse.com/show_bug.cgi?id=1169774 Adrian Schröter changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dimstar@opensuse.org, | |ro@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 Takashi Iwai changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bpetkov@suse.com, | |jroedel@suse.com, | |jslaby@suse.com, | |tiwai@suse.com -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c2 --- Comment #2 from Adrian Schröter --- the guest logs are the build logs. dmesg on the host just lists all the usual EXT4-fs mount messages. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c4 --- Comment #4 from Borislav Petkov --- Well, you'd have to give me something in order to try to debug this. As in, host kernel is <version>, guest kernel is 5.6 and the workload is <workload>. It worked fine with guest+host kernel <version> and it is broken with 5.6 32-bit guest kernel and host kernel <version>. And anything else that you think is relevant. And of course I need dmesg to know what your host is running: BIOS, microcode, etc. and the guest dmesg is needed so that I know how you start your guest. Also, how do you start your guest exactly? qemu cmdline, etc. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c6 --- Comment #6 from Adrian Schröter --- the host was always openSUSE Leap 15.1 based and had no change. The guest became bad with 5.6, the workload is the usual build process. You can run it local in the same way as usual via: # osc build --vm-type=kvm .... You also see the qemu command line there (and also in the build logs). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c8 --- Comment #8 from Adrian Schröter --- Microcode: patch_level=0x08301025 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c10 --- Comment #10 from Adrian Schröter --- The guest is using the kernel (and initrd) provided by kernel-obs-build package which is maintained as part of the kernel sources. (same vmlinux as kernel-default, just repackaged with own initrd) For running a local build with the mentioned example package you need to do # osc co openSUSE:Factory installation-images # cd openSUSE:Factory/installation-images # osc build --vm-type=kvm -M openSUSE standard i586 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c12 --- Comment #12 from Jiri Slaby --- (In reply to Adrian Schröter from comment #10)

# osc build --vm-type=kvm -M openSUSE standard i586

and add --userootforbuild --vm-disk-size=30000. On epyc machine (remus), I had to add -j 32, as -j 33 (or more) results in:

smpboot: Total of 33 processors activated (148221.48 BogoMIPS) BUG: kernel NULL pointer dereference, address: 00000d24 #PF: supervisor read access in kernel mode #PF: error_code(0x0000) - not-present page *pdpt = 0000000000000000 *pde = f000ff53f000ff53 Oops: 0000 [#1] SMP NOPTI CPU: 0 PID: 1 Comm: swapper/0 Tainted: G S W 5.6.4-1-pae #1 openSUSE Tumbleweed (unreleased) Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.0-59-gc9ba527-rebuilt.suse.com 04/01/2014 EIP: __alloc_pages_nodemask+0xd6/0x2b0

-- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c14 --- Comment #14 from Jiri Slaby --- Grabbing this into rpms/: http://download.opensuse.org/history/20200331/tumbleweed/repo/oss/i586/kerne... and adding -p rpms to osc build indeed makes a difference... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c16 --- Comment #16 from Jiri Slaby --- Created attachment 836393 --> http://bugzilla.suse.com/attachment.cgi?id=836393&action=edit dmesg from guest with 5.6 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c19 --- Comment #19 from Jiri Slaby --- It seems that I managed to reproduce with custom-built kernel, so I am starting bisection. It seems that the problem is I/O.

Device tps kB_read/s kB_wrtn/s kB_dscd/s kB_read kB_wrtn kB_dscd vda 436.66 167.78 3114.85 0.00 935805 17372932 0 vda 270.82 822.20 9404.21 0.00 760877 8702840 0

The first is 5.6 = bad, the second is good = 5.5. 3 times slower writes, 5 times slower reads. But it could be due to accumulation of I/O buffers. Let's see if I can bisect it at last as I failed 2 times already (bisection lead to a merge commit or so...). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c21 --- Comment #21 from Jiri Slaby --- Created attachment 837073 --> http://bugzilla.suse.com/attachment.cgi?id=837073&action=edit remote build log on lamb63 (fast) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c23 Jiri Slaby changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jack@suse.com Flags| |needinfo?(jack@suse.com) --- Comment #23 from Jiri Slaby --- Now perhaps Honza will advise what to do next... -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c25 Jiri Slaby changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |IN_PROGRESS --- Comment #25 from Jiri Slaby --- FYI I reverted the commit in stable and pushed to TW as: https://build.opensuse.org/request/show/799215 -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c28 --- Comment #28 from Jan Kara --- So some more data: One fsmark loop dirties ~812000 unique metadata blocks (this is regardless of which writeback strategy is used). Now 'lock' runs do some 110 *millions* of metadata block dirty events, 'nolock' run does ~125 millions of metadata block dirty events. So we see redirtying really happens a *lot* - one block is redirtied over 100 times on average. Due to transaction batching, we actually journal only ~5.5 millions of metadata blocks for 'lock' runs and ~7.5 millions of metadata blocks for 'nolock' runs - transactions are smaller so batching is less efficient here. These differences match very well the difference in the amount of writes we can see in comment 27 and explain the performance difference. So I have now good understanding why the regression happens. Now I have to figure out if we can somehow improve this so that 'nolock' mode isn't hit that badly. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 http://bugzilla.suse.com/show_bug.cgi?id=1169774#c30 Jan Kara changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(jack@suse.com) | --- Comment #30 from Jan Kara --- Created attachment 837903 --> http://bugzilla.suse.com/attachment.cgi?id=837903&action=edit [PATCH] jbd2: Avoid leaking transaction credits when unreserving This patch fixes the fsmark performance for me. Once upstream accepts it, I'll replace the mount-option switch in master branch with it. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.suse.com/show_bug.cgi?id=1169774 Jan Kara changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P3 - Medium CC| |kernel-performance-bugs@sus | |e.de -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1169774 https://bugzilla.suse.com/show_bug.cgi?id=1169774#c32 Jan Kara changed: What |Removed |Added ---------------------------------------------------------------------------- Status|IN_PROGRESS |RESOLVED Resolution|--- |FIXED --- Comment #32 from Jan Kara --- Patch is upstream and in our master branch as well. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1169774 https://bugzilla.suse.com/show_bug.cgi?id=1169774#c35 Fabian Vogt changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(fvogt@suse.com) | --- Comment #35 from Fabian Vogt --- (In reply to Jan Kara from comment #34)

Fabian, are you sure this is the same problem? I mean was this consistently failing like this for past 6 months when the change was introduced? AFAIU you're speaking about the 'failed' builds on 'sheep' and 'goat' hosts, aren't you? 'lamb', 'cloud', and 'build' hosts seem to work fine.

The osc jobhist only goes back until beginning of october. In that timeframe, it's been consistently slow/failing i586 goat/sheep builds. I did some tests using different filesystems inside the VM using "Buildflags: vmfstype:foo" in the prjconf. Using btrfs made no difference (compared to ext4). XFS appears to work much better. So the issue apparently impacts ext4 and btrfs the most. Kiwi is still using ext4 for the filesystem on the livecd though, so I could only compare the time until the final rsync. XFS on sheep81, 23s for kernel-firmware: [ 94s] [ DEBUG ]: 16:02:24 | system: ( 47/1065) Installing: kbd-legacy-2.3.0-1.1.noarch [............done] [ 94s] [ DEBUG ]: 16:02:24 | system: Additional rpm output: [ 94s] [ DEBUG ]: 16:02:24 | system: warning: /var/cache/kiwi/packages/f796d7d2bc4daf38063ff386ebbc072d/kbd-legacy.rpm: Header V3 RSA/SHA256 Signature, key ID 3dbdc284: NOKEY [ 117s] [ DEBUG ]: 16:02:47 | system: ( 48/1065) Installing: kernel-firmware-20201023-1.1.noarch [.......................done] [ 117s] [ INFO ]: Processing: [# ] 2%[ DEBUG ]: 16:02:47 | system: Additional rpm output: [ 117s] [ DEBUG ]: 16:02:47 | system: warning: /var/cache/kiwi/packages/f796d7d2bc4daf38063ff386ebbc072d/kernel-firmware.rpm: Header V3 RSA/SHA256 Signature, key ID 3dbdc284: NOKEY btrfs on sheep82, 141s for kernel-firmware: [ 108s] [ DEBUG ]: 16:27:22 | system: ( 47/1065) Installing: kbd-legacy-2.3.0-1.1.noarch [............done] [ 108s] [ DEBUG ]: 16:27:22 | system: Additional rpm output: [ 108s] [ DEBUG ]: 16:27:22 | system: warning: /var/cache/kiwi/packages/f796d7d2bc4daf38063ff386ebbc072d/kbd-legacy.rpm: Header V3 RSA/SHA256 Signature, key ID 3dbdc284: NOKEY [ 249s] [ DEBUG ]: 16:29:43 | system: ( 48/1065) Installing: kernel-firmware-20201023-1.1.noarch [...................................................................done] [ 249s] [ INFO ]: Processing: [# ] 2%[ DEBUG ]: 16:29:43 | system: Additional rpm output: [ 249s] [ DEBUG ]: 16:29:43 | system: warning: /var/cache/kiwi/packages/f796d7d2bc4daf38063ff386ebbc072d/kernel-firmware.rpm: Header V3 RSA/SHA256 Signature, key ID 3dbdc284: NOKEY

If the start of the problem indeed dates back 6 months, I'd like to check whether mounting the filesystem with 'dioread_lock' mount option indeed fixes the issue. What would be the easiest way to try that with OBS? The option needs to be passed either to mount, to mke2fs (as part of default mount options), or I can provide patched kernel with modified defaults...

I don't think it's possible for users to influence mount options directly, so if you think that test makes sense, a kernel-obs-build package for i586 would be ideal. FTR, my test projects are at https://build.opensuse.org/project/show/home:favogt:boo1169774 (xfs) https://build.opensuse.org/project/show/home:favogt:boo1169774-btrfs (btrfs) -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1169774 https://bugzilla.suse.com/show_bug.cgi?id=1169774#c37 --- Comment #37 from Fabian Vogt --- (In reply to Jan Kara from comment #36)

If this is happening on both ext4 and btrfs it seems unlikely to be caused by ext4 specific problem (provided btrfs ever worked before). But anyway, let's debug this and we'll see where the problem is. The difference between XFS and ext4 shouldn't indeed be that big.

The werird part is IMO that this only affects i586 builds on goat and sheep workers. x86_64 on goat/sheep and i586 on other workers are just fine.

Now I'm mostly ignorant of OBS (well, I use it to build simple packages but that's all). So if I understand right 'vmfstype' influences what is used as a root filesystem of the VM that is doing the build of the package? I.e. in your case of live CD images?

Yep!

What exactly happens between installation of kbd-legacy-2.3.0-1.1.noarch and kernel-firmware-20201023-1.1.noarch in the build system? This is the difference you care about right?

The main issue is "general slowness" of builds, with the rsync difference being the most obvious. That always uses ext4 (not configurable) and takes too long for quick testing, so I picked some other part of the build for comparison.

...

But even in the "fast" case it takes 23 seconds so I'd like to understand what the system is doing during this time...

Probably just installation of the kernel-firmware rpm. It's quite massive (zypper says "219.3 MiB (563.5 MiB unpacked)") and contains 2554 files. Other package installations are affected as well, but most are just smaller library packages and a sub-second difference isn't that visible in the log.

Also where can I see the full logs you've pasted from?

btrfs: https://build.opensuse.org/package/live_build_log/home:favogt:boo1169774-btr... ext4: https://build.opensuse.org/package/live_build_log/home:favogt:boo1169774-ext... xfs: https://build.opensuse.org/package/live_build_log/home:favogt:boo1169774/liv... -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1169774 https://bugzilla.suse.com/show_bug.cgi?id=1169774#c39 --- Comment #39 from Ruediger Oertel ---

The werird part is IMO that this only affects i586 builds on goat and sheep workers. x86_64 on goat/sheep and i586 on other workers are just fine.

this basically says 98% of all workers, IMHO this only leaves the cloud??? (101-138) and build?? (70-85) machines, all older Intel CPUs and all sheep/lamb machines are Opteron and goat are EPYC. Something known with 32bit on AMD ? -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1169774 https://bugzilla.suse.com/show_bug.cgi?id=1169774#c41 --- Comment #41 from Fabian Vogt --- I did some more tests. Looking just at the time it takes to install kernel-firmware doesn't really work out, sometimes it's fast even on a build which times out later. So instead I'll write down the time until the root system contents are done, i.e. when "mkfs.ext4" gets called. goat03 + ext4 + TW kernel: 1739s goat19 + xfs + TW kernel: 613s lamb72 + btrfs + TW kernel: 384s lamb66 + ext4 + TW kernel: 303s goat19 + ext4 + 15.1 (GA) kernel: 189s sheep81 + ext4 + 15.2 (GA) kernel: 224s So with kernel-obs-build-4.12.14-lp151.27.3 from openSUSE:Leap:15.1 the build on goat was considerably faster than others, which matches the x86_64 build times. Same for kernel-obs-build-5.3.18-lp152.19.2 from openSUSE:Leap:15.2 on sheep81. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1169774 https://bugzilla.suse.com/show_bug.cgi?id=1169774#c43 --- Comment #43 from Jan Kara --- Thanks for all the tests! This looks like a very different from the one originally reported in this bug. The original issue was specific to ext4 filesystem while what you report now seems like a general slow down (just affecting different filesystems to different extent - see how XFS build on goat19 takes over 600s with TW kernel on XFS while it takes just 189s with 15.1 kernel on ext4 - it would be good to get comparison with 15.1 build on XFS but I'd expect it to be close to ext4 numbers on that kernel). So I'll clone this bug so that we don't confuse this bug. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1169774 https://bugzilla.suse.com/show_bug.cgi?id=1169774#c44 --- Comment #44 from Jan Kara --- Let's solve this in bug 1178762 which is cloned from this bug. I'll copy there the relevant comments. -- You are receiving this mail because: You are on the CC list for the bug.