[opensuse-buildservice] Mysterious build failures: i586 chroot on CentOS 7 host
Hi all, I am setting up a private OBS to build packages for Scientific Linux 6 (i586, x86_64) and CentOS 7 (x86_64 only). Going out on a limb, I have installed obsworker on various SL/CentOS boxes, rather than openSUSE VMs. Rationale: a similar setup has been working fine for us for years with OBS 2.3; we are a Red Hat shop, with SL 6 or CentOS 7 installed directly on the hardware, and I figure builds will be faster with less layers of virtualization in the way. But I'm having some bizarre problems building i586 packages on CentOS 7 hosts. They all seem to boil down to one thing: the filesystem does not behave properly. Specifically, simple programs that do simple filesystem operations, like "which" or "alternatives" or "pkg-config", fail in very strange ways. The same packages build just fine for x86_64 SL 6 on CentOS 7. And they build just fine for i586 SL 6 on an SL 6 host. The only time builds fail mysteriously is when the build host is CentOS 7, and the build target is i586 SL 6. And not all packages fail, just some. Example failure: building c3p0, a Java connection pool library because /usr/bin/java does not exist. Here's the chain of failures: * setting up the build chroot installs java-1.6.0-openjdk-1:1.6.0.35-1.13.7.1.el6_6 * postinstall script for that package fails when running "alternatives --install [...]", which is supposed to ensure that /usr/bin/java exists: [ 99s] [229/244] installing java-1.6.0-openjdk-1:1.6.0.35-1.13.7.1.el6_6 [ 105s] altdir /etc/alternatives invalid [ 105s] altdir /etc/alternatives invalid [ 105s] altdir /etc/alternatives invalid * thus, no /etc/alternatives/java * thus, no /usr/bin/java * thus, /usr/bin/ant crashes with "Couldn't find java virtual machine" * thus, build fails I should note that several other RPM operations failed while setting up the build chroot. Lots of packages printed "altdir /etc/alternatives invalid" errors. Others were even worse: [ 75s] [183/244] installing fontconfig-2.8.0-5.el6 [ 77s] "/usr/share/fonts": Value too large for defined data type [ 77s] warning: %post(fontconfig-2.8.0-5.el6.i686) scriptlet failed, exit status 1 [ 77s] [184/244] installing gpm-1.20.6-12.el6 [ 78s] /var/tmp/rpm-tmp.YKOvqv: line 1: 17643 Bus error /sbin/chkconfig --add gpm I can easily reproduce the error if I enter the build chroot myself: $ sudo chroot /data/obs/build/root_3 /bin/bash bash-4.1# alternatives --install /usr/bin/java java /usr/lib/jvm/jre-1.6.0-openjdk/bin/java 16000 altdir /etc/alternatives invalid strace output is ... unhelpful. I see no obviously failing system calls. I see a *successful* "stat64("/etc/alternatives", {st_mode=S_IFDIR|0755, st_size=10, ...}) = 0", FWIW. Any clues? My gut instinct tells me to chase after that "Bus error" from /sbin/chkconfig. Thanks! Greg -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Feb 27 2017, Greg Ward
I can easily reproduce the error if I enter the build chroot myself:
$ sudo chroot /data/obs/build/root_3 /bin/bash bash-4.1# alternatives --install /usr/bin/java java /usr/lib/jvm/jre-1.6.0-openjdk/bin/java 16000 altdir /etc/alternatives invalid
strace output is ... unhelpful. I see no obviously failing system calls. I see a *successful* "stat64("/etc/alternatives", {st_mode=S_IFDIR|0755, st_size=10, ...}) = 0", FWIW.
Any clues?
Most likely the caller was not compiled with _FILE_OFFSET_BITS=64, and some of the values returned by the kernel in struct stat64 are not representable by struct stat (perhaps the inode number). Try running strace with `-e verbose=all' to see the full struct contents. Andreas. -- Andreas Schwab, schwab@linux-m68k.org GPG Key fingerprint = 58CA 54C7 6D53 942B 1756 01D3 44D5 214B 8276 4ED5 "And now for something completely different." -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Feb 28, 2017, at 05:36, Andreas Schwab wrote:
Most likely the caller was not compiled with _FILE_OFFSET_BITS=64, and some of the values returned by the kernel in struct stat64 are not representable by struct stat (perhaps the inode number). Try running strace with `-e verbose=all' to see the full struct contents.
Ooooh, that's a very good hypothesis. The bus errors are also a strong hint: it's been many years and many OS/architecture combinations since I saw a bus error, but as I recall it was caused by alignment problems. So it might not be _FILE_OFFSET_BITS, but it really smells like a binary incompatibility between some SL 6 32-bit binaries and the CentOS 7 64-bit kernel. If correct, this means that something about how SL 6 was compiled precludes using that OS in a chroot environment hosted on CentOS 7. That is unfortunate, but not fatal. I guess I'll just have to restrict my 32-bit builds to running only on SL 6 hosts. Time to go learn about build constraints... Thanks! Greg -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tuesday 2017-02-28 15:47, Greg Ward wrote:
On Tue, Feb 28, 2017, at 05:36, Andreas Schwab wrote:
Most likely the caller was not compiled with _FILE_OFFSET_BITS=64, and some of the values returned by the kernel in struct stat64 are not representable by struct stat (perhaps the inode number). Try running strace with `-e verbose=all' to see the full struct contents.
Ooooh, that's a very good hypothesis. The bus errors are also a strong hint: it's been many years and many OS/architecture combinations since I saw a bus error, but as I recall it was caused by alignment problems.
SIGBUS, or more specifically CPU exceptions, are generated on e.g. SPARC and (I might have heard) x86's SSE, but not the "normal" x86 or ppc instructions. It just handles it behind the scenes [likely at a slower pace - but still]. The lack of _FILE_OFFSET_BITS=64 in 32-bit compilations would also normally manifest in (seemingly-spurious) error returns (EOVERFLOW/"Value too large"), not so much asynchronous signals. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tue, Feb 28, 2017, at 10:07, Jan Engelhardt wrote:
On Tuesday 2017-02-28 15:47, Greg Ward wrote:
The bus errors are also a strong hint: it's been many years and many OS/architecture combinations since I saw a bus error, but as I recall it was caused by alignment problems.
SIGBUS, or more specifically CPU exceptions, are generated on e.g. SPARC and (I might have heard) x86's SSE, but not the "normal" x86 or ppc instructions. It just handles it behind the scenes [likely at a slower pace - but still].
I can confirm from experience that SIGBUS happens with alignment errors on MIPS chips from the 1990s. Doubt anyone cares about that now, though!
The lack of _FILE_OFFSET_BITS=64 in 32-bit compilations would also normally manifest in (seemingly-spurious) error returns (EOVERFLOW/"Value too large"), not so much asynchronous signals.
Indeed, I saw EOVERFLOW from getdents() system calls. So the _FILE_OFFSET_BITS hypothesis looks pretty strong. Anyways, I'm not going to dig any deeper: I'm just going to conclude "32-bit SL 6 in a chroot under 64-bit CentOS 7 does not work" and move on. I'm certainly not going to recompile an entire OS with different preprocessor flags. Thanks for your help! Greg -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Tuesday 2017-02-28 17:20, Greg Ward wrote:
Anyways, I'm not going to dig any deeper: I'm just going to conclude "32-bit SL 6 in a chroot under 64-bit CentOS 7 does not work"
I have seen the same with Debian i386 chroots. Inode numbers above 4.2G are easy to reach - takes just a 2 TB XFS volume. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (3)
-
Andreas Schwab -
Greg Ward -
Jan Engelhardt