https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c2 Dominique Leuenberger changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |adrian.schroeter@suse.com Flags|needinfo?(dimstar@opensuse. |needinfo?(adrian.schroeter@ |org) |suse.com) --- Comment #2 from Dominique Leuenberger --- (In reply to Jiri Slaby from comment #1)

And yet, it looks like host-specific. lamb and old-cirrus machines seem to be affected only. I don't know what's their configuration though. Maybe Dominique can shed some light?

Not really - except if you can do something out of the information provided by

osc workerinfo x86_64:lamb02:1

I'm sure Adrian is better suited to answer the original question:

And yet, it looks like host-specific. lamb and old-cirrus machines seem to be > affected only. I don't know what's their configuration though. Maybe Dominique > can shed some light?

-- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c4 Marcus R�ckert changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mrueckert@suse.com --- Comment #4 from Marcus R�ckert --- Bad run: ``` [ 11s] D: ========== +++ libdb-4_8-4.8.30-41.81 x86_64-linux 0x0 [ 11s] D: install: libdb-4_8-4.8.30-41.81.x86_64 has 2 files [ 11s] D: Plugin: calling hook psm_pre in selinux plugin [ 11s] libdb-4_8-4.8.30-41.81 D: create 100755 1 ( 0, 0)1595712 /usr/lib64/libdb-4.8.so;63989ef2 [ 11s] D: create 100755 1 ( 0, 0)1743256 /usr/lib64/libdb_cxx-4.8.so;63989ef2 [ 11s] D: ========== Directories not explicitly included in package: [ 11s] D: 0 /usr/lib64/ [ 11s] D: ========== [ 11s] ######################################## [ 11s] D: Plugin: calling hook fsm_file_prepare in fsverity plugin [ 11s] D: fsverity no signature for: path /usr/lib64/libdb-4.8.so;63989ef2 dest /usr/lib64/libdb-4.8.so [ 11s] D: Plugin: calling hook fsm_file_prepare in ima plugin [ 11s] D: Plugin: calling hook fsm_file_prepare in selinux plugin [ 11s] D: Plugin: calling hook fsm_file_prepare in fsverity plugin [ 11s] D: fsverity no signature for: path /usr/lib64/libdb_cxx-4.8.so;63989ef2 dest /usr/lib64/libdb_cxx-4.8.so [ 11s] D: Plugin: calling hook fsm_file_prepare in ima plugin [ 11s] D: Plugin: calling hook fsm_file_prepare in selinux plugin [ 11s] D: adding "libdb-4_8" to Name index. [ 11s] D: adding 2 entries to Basenames index. [ 11s] D: adding "System/Libraries" to Group index. [ 11s] D: adding 25 entries to Requirename index. [ 11s] D: adding 5 entries to Providename index. [ 11s] D: adding 1 entries to Dirnames index. [ 11s] D: adding 1 entries to Installtid index. [ 11s] D: adding 1 entries to Sigmd5 index. [ 11s] D: adding "39322b96c5751e2783f0d4fc0307783ae06e355e" to Sha1header index. [ 11s] D: Plugin: calling hook psm_post in syslog plugin [ 11s] error: libdb-4_8-4.8.30-41.81.x86_64: install failed ``` a good package right before this one ``` [ 11s] D: ========== +++ libasan8-12.2.1+git537-2.20 x86_64-linux 0x0 [ 11s] D: install: libasan8-12.2.1+git537-2.20.x86_64 has 2 files [ 11s] D: Plugin: calling hook psm_pre in selinux plugin [ 11s] libasan8-12.2.1+git537-2.20 D: create 120777 1 ( 0, 0) 16 /usr/lib64/libasan.so.8;63989ef2 [ 11s] D: create 100755 1 ( 0, 0)1307120 /usr/lib64/libasan.so.8.0.0;63989ef2 [ 11s] D: ========== Directories not explicitly included in package: [ 11s] D: 0 /usr/lib64/ [ 11s] D: ========== [ 11s] ######################################## [ 11s] D: Plugin: calling hook fsm_file_prepare in fsverity plugin [ 11s] D: fsverity skipping non regular: path /usr/lib64/libasan.so.8;63989ef2 dest /usr/lib64/libasan.so.8 [ 11s] D: Plugin: calling hook fsm_file_prepare in ima plugin [ 11s] D: Plugin: calling hook fsm_file_prepare in selinux plugin [ 11s] D: Plugin: calling hook fsm_file_prepare in fsverity plugin [ 11s] D: fsverity no signature for: path /usr/lib64/libasan.so.8.0.0;63989ef2 dest /usr/lib64/libasan.so.8.0.0 [ 11s] D: Plugin: calling hook fsm_file_prepare in ima plugin [ 11s] D: Plugin: calling hook fsm_file_prepare in selinux plugin [ 11s] D: adding "libasan8" to Name index. [ 11s] D: adding 2 entries to Basenames index. [ 11s] D: adding "Development/Languages/C and C++" to Group index. [ 11s] D: adding 21 entries to Requirename index. [ 11s] D: adding 3 entries to Providename index. [ 11s] D: adding 1 entries to Conflictname index. [ 11s] D: adding 1 entries to Dirnames index. [ 11s] D: adding 1 entries to Installtid index. [ 11s] D: adding 1 entries to Sigmd5 index. [ 11s] D: adding "2973d96fce735d6078efd3998891e1a7d8e1f98a" to Sha1header index. [ 11s] D: %post(libasan8-12.2.1+git537-2.20.x86_64): scriptlet start [ 11s] D: %post(libasan8-12.2.1+git537-2.20.x86_64): execv(/sbin/ldconfig) pid 957 [ 11s] D: Plugin: calling hook scriptlet_fork_post in prioreset plugin [ 11s] D: Plugin: calling hook scriptlet_fork_post in selinux plugin [ 11s] D: %post(libasan8-12.2.1+git537-2.20.x86_64): waitpid(957) rc 957 status 0 [ 11s] D: Plugin: calling hook scriptlet_post in syslog plugin [ 11s] D: Plugin: calling hook psm_post in syslog plugin ``` -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c6 Jiri Slaby changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mls@suse.com Flags| |needinfo?(mls@suse.com) --- Comment #6 from Jiri Slaby --- (In reply to Marcus R�ckert from comment #5)

so from the logs it looks like on the newer kernel rpm sometimes fails to launch the processes for the post scriptlets. sadly without any useful error.

If post hooks failed, that would dump: Plugin %s: hook psm_post failed But that didn't happen. So post hooks likely finished w/o error. It's something done after post hooks, right? Michael? -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c8 --- Comment #8 from Jiri Slaby --- (In reply to Jan Kara from comment #3)

BTW, can we get kernel logs from the build in OBS if there's anything interesting?

Adding to this, if we run debug runs, can we increase the kernel debug level? Currently, /usr/lib/build/build-vm does: echo 4 > /proc/sysrq-trigger so "echo 8" might show us more. -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c10 --- Comment #10 from Dominique Leuenberger --- (In reply to Marcus R�ckert from comment #5)

so from the logs it looks like on the newer kernel rpm sometimes fails to launch the processes for the post scriptlets. sadly without any useful error.

From a different build: https://build.opensuse.org/package/live_build_log/openSUSE:Factory:Staging:J... [ 18s] python-rpm-macros-20221117.8687578-2.1######################################## [ 18s] error: python-rpm-macros-20221117.8687578-2.10.noarch: install failed python-rpm-macros does not even have scriptlets; so I think we're following a red herring with the scriptlets -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c12 --- Comment #12 from Jiri Slaby --- (In reply to Jiri Slaby from comment #11)

ret += idxdbPut(dbi, rpmtag, hdrNum, h);

One things where the above returns RPMRC_FAIL is when this fails: flock(pkgdb->fd, LOCK_EX); Let me check if there are any changes in file locking in between 6.0 and 6.1. -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c14 --- Comment #14 from Jiri Slaby --- (In reply to Jiri Slaby from comment #13)

Can anyone test with this patched rpm: https://build.opensuse.org/package/show/home:jirislaby:debug61/rpm ? There is a patch with many more rpmlog(RPMLOG_DEBUG, ...) to narrow what and how actually fails.

I hit it once in home:jirislaby:debug61:rebuild and it points to rpmidxPutInternal()'s: addnewkey(idxdb, key, keyl, &keyoff) And that does pwrite(), mmap() and mremap() internally. I respinned the debug patch to narrow it further down. -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c16 --- Comment #16 from Vlastimil Babka --- (In reply to Jiri Slaby from comment #15)

(In reply to Jiri Slaby from comment #14)

...

And that does pwrite(), mmap() and mremap() internally.

And the winner is: [ 30s] error: remapslot (223): mapmem(old=0xffffffffffffffff, osz=1056768, nsz=1060864, fl=3, fd=4, off=868352) failed 12

How exactly is that log related to mremap() syscall and its parameters? Can I assume the address and old/new sizes are 1:1 parameters? if 'fl=3' are flags of mremap() then going by tools/include/uapi/linux/mman.h it would be MREMAP_MAYMOVE | MREMAP_FIXED. With MREMAP_FIXED there should be also fifth parameter saying the new address, which I don't see here. Instead the "fd=4, off=868352" are not mremap() parameters.

i.e. mremap fails with ENOMEM. Vlasta, any ideas?

At least old=0xffffffffffffffff looks quite wrong, but it should result in EINVAL rather than ENOMEM. In general 6.1 includes the maple tree, so that would be my first suspect to look, but we should make sure what the mremap() parameters exactly are and ideally get a /proc/pid/smaps snapshot upon the failure. -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c18 --- Comment #18 from Jiri Slaby --- Created attachment 863524 --> https://bugzilla.suse.com/attachment.cgi?id=863524&action=edit smaps (In reply to Jiri Slaby from comment #17)

...

In general 6.1 includes the maple tree, so that would be my first suspect to look, but we should make sure what the mremap() parameters exactly are and ideally get a /proc/pid/smaps snapshot upon the failure.

Let me catch also the *previous* mmap() giving MAP_FAILED which is put to oldaddr.

smaps dump attached. I don't know where this "-1 as first argument" is from. slot->mapped (the old addr) is never assigned MAP_FAILED or -1. error: remapslot (232): mapped=0xffffffffffffffff shift=0 mapmem(old=0xffffffffffffffff, osz=1056768, nsz=1060864, prot=3, fd=4, off=868352) failed 12 -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c20 --- Comment #20 from Jiri Slaby --- (In reply to Vlastimil Babka from comment #16)

At least old=0xffffffffffffffff looks quite wrong, but it should result in EINVAL rather than ENOMEM. In general 6.1 includes the maple tree, so that would be my first suspect to look

I have to go now, so for now, I came to a suspicion that this is memory corruption. That 0xffffffffffffffff must be written by something else. So could this mmap() kernel rewrite ended up in something like this in rpm: 1) slots = calloc(); (it's 16320B, so likely via sbrk(), I didn't check) 2) do a lot of mremap() and mmap(), some of them simply returns "slots" address from 1) again. 3) slot[X].mapped gets rewritten by a mistake due to 2) by a bunch of 0xff. 4) we see slot->mapped as -1 here. ? -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c22 --- Comment #22 from Jiri Slaby --- (In reply to Michael Schr�der from comment #21)

But wouldn't that lead to EINVAL because offset_in_page(addr) is not zero? That sounds more like your debug statement does not match what happens.

I might do some wrong mistake. Feel free to check the patch ;): https://build.opensuse.org/package/view_file/home:jirislaby:debug61/rpm/debu... -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c24 Fabian Vogt changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fvogt@suse.com --- Comment #24 from Fabian Vogt --- Created attachment 863532 --> https://bugzilla.suse.com/attachment.cgi?id=863532&action=edit straced-rpm.zst I was able to get rpm to fail inside strace -f. Full build log attached. -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c26 Jiri Slaby changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #863524|0 |1 is obsolete| | --- Comment #26 from Jiri Slaby --- Created attachment 863539 --> https://bugzilla.suse.com/attachment.cgi?id=863539&action=edit smaps Smaps for this error (I fixed the "mapped" variable dump): error: remapslot (235): mapped=0x7f72f4e78000 shift=0 mapmem(old=0x7f72f4e78000, osz=1056768, nsz=1060864, prot=3, fd=4, off=868352) failed 12 -- You are receiving this mail because: You are the assignee for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1206359 https://bugzilla.suse.com/show_bug.cgi?id=1206359#c28 Jiri Slaby changed: What |Removed |Added ---------------------------------------------------------------------------- Assignee|kernel-bugs@opensuse.org |vbabka@suse.com --- Comment #28 from Jiri Slaby --- So I reverted: commit ca3d76b0aa808a06997297d123b66d17b81e5285 Author: Jakub Mat��na Date: Fri Jun 3 16:57:19 2022 +0200 mm: add merging after mremap resize in my home:jirislaby:debug61 tesbed. And it seems to fix the problem. I submitted to Staging to verify. The problem is I don't understand why it helps as vma merge in mremap happens only in the "if (vma_expandable(vma, new_len - old_len)) {" case. So the remap we have here should not be affected. Leaving up to Vlasta, I believe he knows far better than me. -- You are receiving this mail because: You are the assignee for the bug.