https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c1 --- Comment #1 from Sarah Kriesch --- Thank you for this bug report, Reinhard! Additional important information from the other bug report: Aaron Puchert: Since it comes from ld (or collect2?) and that's an LTO link, I suppose it might be a race condition. I agree, it should be a separate bug. Maybe add Michael Matz for binutils and Martin Liska for GCC LTO. IBM tends to have weaker memory models, and that race might just not materialize with the stronger model of x86. Michael Matz: I was involved for that already. I never was able to reproduce it with a local build (on s390x of course), with none of the postgres packages. And triggering a rebuild of an affected package, even if it got to the same build machine, sometime also makes it build through (just to fail on the next rebuilds again). So, it's most definitely a race somewhere, but without being at least somewhat locally reproducable it's hopeless. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c3 --- Comment #3 from Sarah Kriesch --- The build is failing with PostgreSQL11 at the moment. Log: https://build.opensuse.org/package/live_build_log/openSUSE:Factory:zSystems/... -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 Ihno Krumreich changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P3 - Medium -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c7 --- Comment #7 from Michael Matz --- (In reply to Richard Biener from comment #5)

There's not really anything "racy" in how GCC LTO is set up - the actual

That's correct. As usual we use the term 'race' for anything unexplainable and unreproducable, where data doesn't match anything reasonable. We could call it faulty memory, or moon phase, or bad luck, or maybe undefined behaviour :-)

error message from ld is also not exactly helpful, referencing libLLVM.so twice:

Actually it's helpful a bit in showing that something strange is going on:

[ 994s] /usr/lib64/gcc/s390x-suse-linux/11/../../../../s390x-suse-linux/bin/ld: @GLIBCXX_3.4.11: TLS reference in /usr/lib64/libLLVM.so mismatches non-TLS reference in /usr/lib64/libLLVM.so

Here, the symbol for which there are TLS and non-TLS references is mentioned, it's '@GLIBCXX_3.4.11'. That is of course not any sensible symbol, it's an empty symbol name with a symbol version, or the symbol version itself interpreted as symbol name. My guess is that the linker plugin does something strange (aka undefined), either caused by a problem in llvms use of it, or in ld. It manipulates the symbol tables and the above definitely is something strange with the symbol table.

[ 994s] /usr/lib64/gcc/s390x-suse-linux/11/../../../../s390x-suse-linux/bin/ld: /usr/lib64/libLLVM.so: error adding symbols: bad value

But given that /usr/lib64/libLLVM.so is not built when postgresql is the error should be visible there. There's exactly two TLS references in libLLVM.so.12

23: 0000000000000000 0 TLS GLOBAL DEFAULT UND _ZSt11__once_call@GLIBCXX_3.4.11 (16) 29: 0000000000000000 0 TLS GLOBAL DEFAULT UND _ZSt15__once_callable@GLIBCXX_3.4.11 (16)

No no, if it were anything you can see directly in the input files, then the error would be reproducable. Of course the inputs are completely fine.

to simplify things I would also try to remove the -Wl,--as-needed and try to filter out -ffat-lto-objects from the parts that don't need it.

OTOH the error is from _bfd_elf_merge_symbol but calling that with two same bfds is a bit odd.

All the more reason to believe in some error of the linker plugin. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c12 --- Comment #12 from openQA Review --- This is an autogenerated message for openQA integration by the openqa_review script: This bug is still referenced in a failing openQA test: textmode-server https://openqa.opensuse.org/tests/1944822 To prevent further reminder comments one of the following options should be followed: 1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted 2. The openQA job group is moved to "Released" or "EOL" (End-of-Life) 3. The bugref in the openQA scenario is removed or replaced, e.g. `label:wontfix:boo1234` -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c14 --- Comment #14 from Reinhard Max --- I accepted that SR and expanded the fix to all versions, but unfortunately the error still happens at least on versions 13 and 14 (11 and 12 are still waiting for a compliant worker). -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c16 --- Comment #16 from Reinhard Max --- I just had a successful manual build on a VM with less than 2GB of RAM, so I don't think it depends on the amount of memory. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c18 --- Comment #18 from Sarah Kriesch --- I have checked with Google today for the TLS error message. There was also a hint to replace the compiler with another version as a possible solution. Additionally, I took a look at Fedora and their patches for PostgreSQL13.4. They had an openssl build issue. TLS is integrated in openssl. I do not know, whether that could be a solution. These patches are from PostgreSQL Maintainers. Therefore, they should be also upstream available: https://src.fedoraproject.org/rpms/postgresql/c/21b8d7a7d28c2765e94ab9c8e4f0... -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c23 --- Comment #23 from openQA Review --- This is an autogenerated message for openQA integration by the openqa_review script: This bug is still referenced in a failing openQA test: textmode-server https://openqa.opensuse.org/tests/2023061 To prevent further reminder comments one of the following options should be followed: 1. The test scenario is fixed by applying the bug fix to the tested product or the test is adjusted 2. The openQA job group is moved to "Released" or "EOL" (End-of-Life) 3. The bugref in the openQA scenario is removed or replaced, e.g. `label:wontfix:boo1234` -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 Sarah Kriesch changed: What |Removed |Added ---------------------------------------------------------------------------- See Also| |https://gcc.gnu.org/bugzill | |a/show_bug.cgi?id=103364 -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c27 --- Comment #27 from Michael Matz --- That's a very generic error message. Nothing like this can be inferred from it. If you believe it's related, please create a new bug report with full information, in particular full command line and error message. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c32 Fabian Vogt changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |fvogt@suse.com --- Comment #32 from Fabian Vogt --- I collected some more info while doing test runs for bug 1188896. I used IBS (OBS is too busy) for test builds of postgresql14, against oS:F:zSystems and my kernel-default branch with the potential fix for bug 1188896 applied. 25/29 succeeded, 3/29 failed with the error reported here and 1/29 got stuck badly even before the build started. On OBS it seems to be worse, maybe just because of additional load or because the workers have less memory and parallel jobs configured. To get some more info, I patched the .spec file to retry make once after it failed. To my surprise, that failed reliably still (2/2 times), so it appears like it's probably reproducible after it happens once. I also made it upload libLLVM.so* and the jit/llvm/ directory after the (successful or failed) build. Also to my surprise, the contents are completely identical, just that the failed builds don't have libllvmjit.so inside. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c42 --- Comment #42 from Swamp Workflow Management --- SUSE-RU-2022:0566-1: An update that has two recommended fixes can now be installed. Category: recommended (moderate) Bug References: 1190740,1195680 CVE References: JIRA References: Sources used: SUSE Linux Enterprise Software Development Kit 12-SP5 (src): postgresql14-14.2-3.6.1 SUSE Linux Enterprise Server 12-SP5 (src): postgresql14-14.2-3.6.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c44 --- Comment #44 from Swamp Workflow Management --- SUSE-RU-2022:0700-1: An update that has two recommended fixes can now be installed. Category: recommended (moderate) Bug References: 1190740,1195680 CVE References: JIRA References: Sources used: SUSE Linux Enterprise Software Development Kit 12-SP5 (src): postgresql13-13.6-3.18.1 SUSE Linux Enterprise Server 12-SP5 (src): postgresql13-13.6-3.18.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c46 --- Comment #46 from Swamp Workflow Management --- openSUSE-RU-2022:0889-1: An update that has two recommended fixes can now be installed. Category: recommended (moderate) Bug References: 1190740,1195680 CVE References: JIRA References: Sources used: openSUSE Leap 15.4 (src): postgresql10-10.20-8.44.1 openSUSE Leap 15.3 (src): postgresql10-10.20-8.44.1 -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c48 --- Comment #48 from Swamp Workflow Management --- openSUSE-RU-2022:0893-1: An update that has two recommended fixes can now be installed. Category: recommended (moderate) Bug References: 1190740,1195680 CVE References: JIRA References: Sources used: openSUSE Leap 15.4 (src): postgresql13-13.6-5.25.1 openSUSE Leap 15.3 (src): postgresql13-13.6-5.25.1 -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c51 --- Comment #51 from Swamp Workflow Management --- SUSE-RU-2022:1457-1: An update that has two recommended fixes can now be installed. Category: recommended (moderate) Bug References: 1190740,1195680 CVE References: JIRA References: Sources used: openSUSE Leap 15.4 (src): postgresql12-12.10-8.29.1 openSUSE Leap 15.3 (src): postgresql12-12.10-8.29.1 SUSE Linux Enterprise Realtime Extension 15-SP2 (src): postgresql12-12.10-8.29.1 SUSE Linux Enterprise Module for Packagehub Subpackages 15-SP3 (src): postgresql12-12.10-8.29.1 SUSE Linux Enterprise Module for Legacy Software 15-SP3 (src): postgresql12-12.10-8.29.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.

https://bugzilla.suse.com/show_bug.cgi?id=1190740 https://bugzilla.suse.com/show_bug.cgi?id=1190740#c59 --- Comment #59 from Swamp Workflow Management --- SUSE-SU-2022:2893-1: An update that solves four vulnerabilities and has 6 fixes is now available. Category: security (important) Bug References: 1179945,1183168,1185952,1187751,1190177,1190740,1192516,1195680,1199475,1202368 CVE References: CVE-2021-23214,CVE-2021-23222,CVE-2022-1552,CVE-2022-2625 JIRA References: Sources used: SUSE Linux Enterprise Server for SAP 15-SP1 (src): postgresql-12.0.1-150000.8.19.1 SUSE Linux Enterprise Server for SAP 15 (src): postgresql-12.0.1-150000.8.19.1, postgresql10-10.22-150000.4.42.1 SUSE Linux Enterprise Server 15-SP1-LTSS (src): postgresql-12.0.1-150000.8.19.1 SUSE Linux Enterprise Server 15-SP1-BCL (src): postgresql-12.0.1-150000.8.19.1 SUSE Linux Enterprise Server 15-LTSS (src): postgresql-12.0.1-150000.8.19.1, postgresql10-10.22-150000.4.42.1 SUSE Linux Enterprise High Performance Computing 15-SP1-LTSS (src): postgresql-12.0.1-150000.8.19.1 SUSE Linux Enterprise High Performance Computing 15-SP1-ESPOS (src): postgresql-12.0.1-150000.8.19.1 SUSE Linux Enterprise High Performance Computing 15-LTSS (src): postgresql-12.0.1-150000.8.19.1, postgresql10-10.22-150000.4.42.1 SUSE Linux Enterprise High Performance Computing 15-ESPOS (src): postgresql-12.0.1-150000.8.19.1, postgresql10-10.22-150000.4.42.1 SUSE Enterprise Storage 6 (src): postgresql-12.0.1-150000.8.19.1 SUSE CaaS Platform 4.0 (src): postgresql-12.0.1-150000.8.19.1 NOTE: This line indicates an update has been released for the listed product(s). At times this might be only a partial fix. If you have questions please reach out to maintenance coordination. -- You are receiving this mail because: You are on the CC list for the bug.