OBS: file system performance problem? tests timing out
Hi *, in a couple of coreutils-testsuite builds, I see the tests timing out always in the same test (the one _after_ 'tests/misc/invalid-opt.pl' ... which is 'tests/rm/ext3-perf.sh' [1]. [ 575s] make check-TESTS check-local [ 575s] make[5]: Entering directory '/home/abuild/rpmbuild/BUILD/coreutils-9.0' [ 575s] make[6]: Entering directory '/home/abuild/rpmbuild/BUILD/coreutils-9.0' [ 580s] PASS: tests/misc/help-version.sh [ 581s] PASS: tests/misc/help-version-getopt.sh [ 582s] PASS: tests/tail-2/inotify-race.sh [ 583s] PASS: tests/tail-2/inotify-race2.sh [ 585s] PASS: tests/misc/invalid-opt.pl [ 5994s] qemu-kvm: terminating on signal 15 from pid 26826 (fuser) Job seems to be stuck here, killed. (after 5400 seconds of inactivity) I've seen this on 'i586' and 'x86_64' with both the older v8.32 in B:S/coreutils and the brand-new v9.0 in my home project [3]. Unfortunately, the qemu-vm is directly killed, so I think I can't even see the log output of that test. Maybe it is already killed during the test setup where a lot of files are created, or it's the program under test - 'rm' - that suffers from a regression. Of course, I could simply exempt that particular test case on OBS, but this smells like a problem outside of coreutils which may be worth fixing ... yes, that test has run for a long time without problems and just recently started to fail. New glibc version maybe? Any hints / comments / ideas? [1] https://git.sv.gnu.org/cgit/coreutils.git/tree/tests/rm/ext3-perf.sh?h=v9.0&id=725bb111bd [2] https://build.opensuse.org/package/live_build_log/Base:System/coreutils:test... [3] https://build.opensuse.org/package/live_build_log/home:berny:branches:Base:S... Thanks & have a nice day, Berny
P.S. Just now I've seen this problem also on ppc64le, ppc64 and aarch64. So it's not just limited to i586/x86_64. On 9/29/21 23:51, Bernhard Voelker wrote:
Hi *,
in a couple of coreutils-testsuite builds, I see the tests timing out always in the same test (the one _after_ 'tests/misc/invalid-opt.pl' ... which is 'tests/rm/ext3-perf.sh' [1].
[ 575s] make check-TESTS check-local [ 575s] make[5]: Entering directory '/home/abuild/rpmbuild/BUILD/coreutils-9.0' [ 575s] make[6]: Entering directory '/home/abuild/rpmbuild/BUILD/coreutils-9.0' [ 580s] PASS: tests/misc/help-version.sh [ 581s] PASS: tests/misc/help-version-getopt.sh [ 582s] PASS: tests/tail-2/inotify-race.sh [ 583s] PASS: tests/tail-2/inotify-race2.sh [ 585s] PASS: tests/misc/invalid-opt.pl [ 5994s] qemu-kvm: terminating on signal 15 from pid 26826 (fuser)
Job seems to be stuck here, killed. (after 5400 seconds of inactivity)
I've seen this on 'i586' and 'x86_64' with both the older v8.32 in B:S/coreutils and the brand-new v9.0 in my home project [3]. Unfortunately, the qemu-vm is directly killed, so I think I can't even see the log output of that test. Maybe it is already killed during the test setup where a lot of files are created, or it's the program under test - 'rm' - that suffers from a regression.
Of course, I could simply exempt that particular test case on OBS, but this smells like a problem outside of coreutils which may be worth fixing ... yes, that test has run for a long time without problems and just recently started to fail. New glibc version maybe?
Any hints / comments / ideas?
[1] https://git.sv.gnu.org/cgit/coreutils.git/tree/tests/rm/ext3-perf.sh?h=v9.0&id=725bb111bd [2] https://build.opensuse.org/package/live_build_log/Base:System/coreutils:test... [3] https://build.opensuse.org/package/live_build_log/home:berny:branches:Base:S...
Thanks & have a nice day, Berny
On Mittwoch, 29. September 2021, 23:54:02 CEST Bernhard Voelker wrote:
P.S. Just now I've seen this problem also on ppc64le, ppc64 and aarch64. So it's not just limited to i586/x86_64.
On 9/29/21 23:51, Bernhard Voelker wrote:
Hi *,
in a couple of coreutils-testsuite builds, I see the tests timing out always in the same test (the one _after_ 'tests/misc/invalid-opt.pl' ... which is 'tests/rm/ext3-perf.sh' [1].
[ 575s] make check-TESTS check-local [ 575s] make[5]: Entering directory '/home/abuild/rpmbuild/BUILD/coreutils-9.0' [ 575s] make[6]: Entering directory '/home/abuild/rpmbuild/BUILD/coreutils-9.0' [ 580s] PASS: tests/misc/help-version.sh [ 581s] PASS: tests/misc/help-version-getopt.sh [ 582s] PASS: tests/tail-2/inotify-race.sh [ 583s] PASS: tests/tail-2/inotify-race2.sh [ 585s] PASS: tests/misc/invalid-opt.pl [ 5994s] qemu-kvm: terminating on signal 15 from pid 26826 (fuser)
Job seems to be stuck here, killed. (after 5400 seconds of inactivity)
I've seen this on 'i586' and 'x86_64' with both the older v8.32 in B:S/coreutils and the brand-new v9.0 in my home project [3]. Unfortunately, the qemu-vm is directly killed, so I think I can't even see the log output of that test. Maybe it is already killed during the test setup where a lot of files are created, or it's the program under test - 'rm' - that suffers from a regression.
Of course, I could simply exempt that particular test case on OBS, but this smells like a problem outside of coreutils which may be worth fixing ... yes, that test has run for a long time without problems and just recently started to fail. New glibc version maybe?
try to reproduce it local. If it works in chroot, switch to KVM build to get same kernel and filesystem as on the server. sorry, but this looks like a content problem, so you need to hunt it a bit... bye adrian
Any hints / comments / ideas?
[1] https://git.sv.gnu.org/cgit/coreutils.git/tree/tests/rm/ext3-perf.sh?h=v9.0&id=725bb111bd [2] https://build.opensuse.org/package/live_build_log/Base:System/coreutils:test... [3] https://build.opensuse.org/package/live_build_log/home:berny:branches:Base:S...
Thanks & have a nice day, Berny
-- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
On 9/30/21 07:35, Adrian Schröter wrote:
try to reproduce it local.
If it works in chroot, switch to KVM build to get same kernel and filesystem as on the server.
sorry, but this looks like a content problem, so you need to hunt it a bit...
Thanks. There's no problem locally. The test creates a 400000-entry directory as setup which takes 18s here, and the subsequent rm(1) needs 10s. Switching to KVM doesn't work here: $ env -C /home/voelkerb/osc/home:berny:branches:Base:System/coreutils osc build -M testsuite --vm-type=kvm coreutils.spec Building coreutils.spec for openSUSE_Factory/x86_64 ... Skipping verification of package signatures due to secure VM build Writing build configuration Running build host does not support kvm either the kvm kernel-module is not loaded or kvm is not installed or hardware virtualization is deactivated in the BIOS. I'm having my environment on TW inside of VirtualBox - I assume that's the reason. I'm afraid I don't have much more time to chase this down. I think I'll exempt it from the testsuite. Have a nice day, Berny
participants (2)
-
Adrian Schröter -
Bernhard Voelker