Base:System: Too many open files: coreutils %check failing
Hi *, since a couple days, Base:System/coreutils fails during 'make check'. [ 239s] FAIL: tests/split/r-chunk [ 239s] ========================= [ 239s] [ 239s] ./tests/split/r-chunk.sh: pipe error: Too many open files [ 239s] stat: cannot statx 'x*': No such file or directory [ 239s] FAIL tests/split/r-chunk.sh (exit status: 1) The test looks quite innocent: # Ensure we fall back to appending to a file at a time # if we hit the limit for the number of open files. rm x* (ulimit -n 20 && yes | head -n90 | split -n r/30 ) || fail=1 It's not split(1) which is failing, but the calling shell process is unable to create the pipe for yes|head|split. I added some more test statement (https://build.opensuse.org/build/home:berny:branches:Base:System/openSUSE_Tu...) (ulimit -n 20 && yes | ls -l /proc/self/fd ) || fail=1 and we can see that the calling shell now has a lot of file descriptors open: [ 249s] + ulimit -n 20 [ 249s] + yes [ 249s] + ls -l /proc/self/fd [ 249s] total 0 [ 249s] lr-x------ 1 abuild abuild 64 Sep 26 06:45 0 -> pipe:[63339] [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 1 -> /home/abuild/rpmbuild/BUILD/coreutils-9.1/tests/split/r-chunk.log [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 10 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 11 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 12 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 13 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 14 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 15 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 16 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 17 -> /tmp/GmIHNZVW [ 249s] lr-x------ 1 abuild abuild 64 Sep 26 06:45 18 -> /proc/4390/fd [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 2 -> /home/abuild/rpmbuild/BUILD/coreutils-9.1/tests/split/r-chunk.log [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 20 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 21 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 3 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 4 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 5 -> /tmp/GmIHNZVW [ 249s] lrwx------ 1 abuild abuild 64 Sep 26 06:45 6 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 7 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 8 -> /tmp/GmIHNZVW [ 249s] lrwx------ 1 abuild abuild 64 Sep 26 06:45 9 -> /tmp/Gm5FnqeH (deleted) Was there a recent change in the kernel/glibc/bash which would show such fd leaking? P.S. I see that also the 'dtc' package fails with the same EMFILE error during compilation: https://build.opensuse.org/package/live_build_log/Base:System/dtc/openSUSE_F... [ 4723s] cc -I libfdt -I . -DFDT_ASSUME_MASK=0 -DNO_VALGRIND -g -Os -fPIC -Werror -Wall -Wpointer-arith \ -Wcast-qual -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wredundant-decls -Wshadow -O2 -Wall \ -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables \ -fstack-clash-protection -Werror=return-type -flto=auto -g -pie -o dtc.o -c dtc.c [ 4723s] util.c:20:10: fatal error: libfdt/unistd.h: Too many open files Thanks & have a nice day, Berny
Hi, Am Montag, 26. September 2022, 13:29:48 CEST schrieb Bernhard Voelker:
Hi *,
since a couple days, Base:System/coreutils fails during 'make check'.
[ 239s] FAIL: tests/split/r-chunk [ 239s] ========================= [ 239s] [ 239s] ./tests/split/r-chunk.sh: pipe error: Too many open files [ 239s] stat: cannot statx 'x*': No such file or directory [ 239s] FAIL tests/split/r-chunk.sh (exit status: 1)
The test looks quite innocent:
# Ensure we fall back to appending to a file at a time # if we hit the limit for the number of open files. rm x* (ulimit -n 20 && yes | head -n90 | split -n r/30 ) || fail=1
It's not split(1) which is failing, but the calling shell process is unable to create the pipe for yes|head|split.
I added some more test statement (https://build.opensuse.org/build/home:berny:branches:Base:System/openSUSE_Tu...)
(ulimit -n 20 && yes | ls -l /proc/self/fd ) || fail=1
and we can see that the calling shell now has a lot of file descriptors open:
[ 249s] + ulimit -n 20 [ 249s] + yes [ 249s] + ls -l /proc/self/fd [ 249s] total 0 [ 249s] lr-x------ 1 abuild abuild 64 Sep 26 06:45 0 -> pipe:[63339] [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 1 -> /home/abuild/rpmbuild/BUILD/coreutils-9.1/tests/split/r-chunk.log [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 10 -> /tmp/GmIHNZVW
GNU make recently gained support for using named fifo for jobserver coordination. That filename matches the /tmp/Gm* pattern: https://build.opensuse.org/package/view_file/openSUSE:Factory/make/jobserver... (line 466). So could be a leak (or design issue) in make. Cheers, Fabian
[ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 11 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 12 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 13 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 14 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 15 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 16 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 17 -> /tmp/GmIHNZVW [ 249s] lr-x------ 1 abuild abuild 64 Sep 26 06:45 18 -> /proc/4390/fd [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 2 -> /home/abuild/rpmbuild/BUILD/coreutils-9.1/tests/split/r-chunk.log [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 20 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 21 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 3 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 4 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 5 -> /tmp/GmIHNZVW [ 249s] lrwx------ 1 abuild abuild 64 Sep 26 06:45 6 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 7 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 8 -> /tmp/GmIHNZVW [ 249s] lrwx------ 1 abuild abuild 64 Sep 26 06:45 9 -> /tmp/Gm5FnqeH (deleted)
Was there a recent change in the kernel/glibc/bash which would show such fd leaking?
P.S. I see that also the 'dtc' package fails with the same EMFILE error during compilation:
https://build.opensuse.org/package/live_build_log/Base:System/dtc/openSUSE_F...
[ 4723s] cc -I libfdt -I . -DFDT_ASSUME_MASK=0 -DNO_VALGRIND -g -Os -fPIC -Werror -Wall -Wpointer-arith \ -Wcast-qual -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wredundant-decls -Wshadow -O2 -Wall \ -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables \ -fstack-clash-protection -Werror=return-type -flto=auto -g -pie -o dtc.o -c dtc.c [ 4723s] util.c:20:10: fatal error: libfdt/unistd.h: Too many open files
Thanks & have a nice day, Berny
On 9/26/22 13:42, Fabian Vogt wrote:
Hi,
Am Montag, 26. September 2022, 13:29:48 CEST schrieb Bernhard Voelker:
Hi *,
since a couple days, Base:System/coreutils fails during 'make check'.
[ 239s] FAIL: tests/split/r-chunk [ 239s] ========================= [ 239s] [ 239s] ./tests/split/r-chunk.sh: pipe error: Too many open files [ 239s] stat: cannot statx 'x*': No such file or directory [ 239s] FAIL tests/split/r-chunk.sh (exit status: 1)
The test looks quite innocent:
# Ensure we fall back to appending to a file at a time # if we hit the limit for the number of open files. rm x* (ulimit -n 20 && yes | head -n90 | split -n r/30 ) || fail=1
It's not split(1) which is failing, but the calling shell process is unable to create the pipe for yes|head|split.
I added some more test statement (https://build.opensuse.org/build/home:berny:branches:Base:System/openSUSE_Tu...)
(ulimit -n 20 && yes | ls -l /proc/self/fd ) || fail=1
and we can see that the calling shell now has a lot of file descriptors open:
[ 249s] + ulimit -n 20 [ 249s] + yes [ 249s] + ls -l /proc/self/fd [ 249s] total 0 [ 249s] lr-x------ 1 abuild abuild 64 Sep 26 06:45 0 -> pipe:[63339] [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 1 -> /home/abuild/rpmbuild/BUILD/coreutils-9.1/tests/split/r-chunk.log [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 10 -> /tmp/GmIHNZVW
GNU make recently gained support for using named fifo for jobserver coordination. That filename matches the /tmp/Gm* pattern: https://build.opensuse.org/package/view_file/openSUSE:Factory/make/jobserver... (line 466). So could be a leak (or design issue) in make.
Hmm, both this coreutils and the dtc issue is already discussed upstream by Andreas: https://lists.gnu.org/archive/html/bug-make/2022-09/msg00178.html @Andreas: do you have news on this? Do we need to work around that GNU make bug in B:S/coreutils?
[ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 11 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 12 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 13 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 14 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 15 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 16 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 17 -> /tmp/GmIHNZVW [ 249s] lr-x------ 1 abuild abuild 64 Sep 26 06:45 18 -> /proc/4390/fd [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 2 -> /home/abuild/rpmbuild/BUILD/coreutils-9.1/tests/split/r-chunk.log [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 20 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 21 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 3 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 4 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 5 -> /tmp/GmIHNZVW [ 249s] lrwx------ 1 abuild abuild 64 Sep 26 06:45 6 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 7 -> /tmp/GmIHNZVW [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 8 -> /tmp/GmIHNZVW [ 249s] lrwx------ 1 abuild abuild 64 Sep 26 06:45 9 -> /tmp/Gm5FnqeH (deleted)
Was there a recent change in the kernel/glibc/bash which would show such fd leaking?
P.S. I see that also the 'dtc' package fails with the same EMFILE error during compilation:
https://build.opensuse.org/package/live_build_log/Base:System/dtc/openSUSE_F...
[ 4723s] cc -I libfdt -I . -DFDT_ASSUME_MASK=0 -DNO_VALGRIND -g -Os -fPIC -Werror -Wall -Wpointer-arith \ -Wcast-qual -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wredundant-decls -Wshadow -O2 -Wall \ -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=3 -fstack-protector-strong -funwind-tables -fasynchronous-unwind-tables \ -fstack-clash-protection -Werror=return-type -flto=auto -g -pie -o dtc.o -c dtc.c [ 4723s] util.c:20:10: fatal error: libfdt/unistd.h: Too many open files
Thanks & have a nice day, Berny
On 9/26/22 20:52, Bernhard Voelker wrote:
On 9/26/22 13:42, Fabian Vogt wrote:
and we can see that the calling shell now has a lot of file descriptors open:
[ 249s] + ulimit -n 20 [ 249s] + yes [ 249s] + ls -l /proc/self/fd [ 249s] total 0 [ 249s] lr-x------ 1 abuild abuild 64 Sep 26 06:45 0 -> pipe:[63339] [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 1 -> /home/abuild/rpmbuild/BUILD/coreutils-9.1/tests/split/r-chunk.log [ 249s] l-wx------ 1 abuild abuild 64 Sep 26 06:45 10 -> /tmp/GmIHNZVW
GNU make recently gained support for using named fifo for jobserver coordination. That filename matches the /tmp/Gm* pattern: https://build.opensuse.org/package/view_file/openSUSE:Factory/make/jobserver... (line 466). So could be a leak (or design issue) in make.
Hmm, both this coreutils and the dtc issue is already discussed upstream by Andreas:
https://lists.gnu.org/archive/html/bug-make/2022-09/msg00178.html
@Andreas: do you have news on this? Do we need to work around that GNU make bug in B:S/coreutils?
For my own reference, this is a snippet which would work to close all those leaked file descriptors: # GNU make 4.3.90 leaks a lot of file descriptors when using the jobserver. # Close all of them pointing to files matching '/tmp/Gm*'. p=$$ for n in $(find /proc/$p/fd -lname '/tmp/Gm*' -printf '%f\n'); do exec $n>- done Thanks & have a nice day, Berny
On Sep 26 2022, Bernhard Voelker wrote:
https://lists.gnu.org/archive/html/bug-make/2022-09/msg00178.html
@Andreas: do you have news on this?
If there is any news you will find it in this thread. -- Andreas Schwab, SUSE Labs, schwab@suse.de GPG Key fingerprint = 0196 BAD8 1CE9 1970 F4BE 1748 E4D4 88E3 0EEA B9D7 "And now for something completely different."
On 9/27/22 09:04, Andreas Schwab wrote:
On Sep 26 2022, Bernhard Voelker wrote:
https://lists.gnu.org/archive/html/bug-make/2022-09/msg00178.html
@Andreas: do you have news on this?
If there is any news you will find it in this thread.
How to continue with the new make version (https://build.opensuse.org/request/show/1006617)? I believe that the FD leak in new make should be fixed rather than adding workarounds in failing packages (https://build.opensuse.org/request/show/1030521). Have a nice day, Berny
On Wednesday 2022-10-26 00:39, Bernhard Voelker wrote:
On 9/27/22 09:04, Andreas Schwab wrote:
On Sep 26 2022, Bernhard Voelker wrote:
https://lists.gnu.org/archive/html/bug-make/2022-09/msg00178.html
@Andreas: do you have news on this?
If there is any news you will find it in this thread.
How to continue with the new make version (https://build.opensuse.org/request/show/1006617)?
How about the following patch (for make-4.3.92)? diff --git a/src/posixos.c b/src/posixos.c index 48e8cf5c..4e6cc9ec 100644 --- a/src/posixos.c +++ b/src/posixos.c @@ -139,9 +139,13 @@ unsigned int jobserver_setup (int slots, const char *style) { int r; + if (style == NULL) + { + style = "pipe"; + } #if HAVE_MKFIFO - if (style == NULL || strcmp (style, "fifo") == 0) + if (strcmp (style, "fifo") == 0) { /* Unfortunately glibc warns about uses of mktemp even though we aren't using it in dangerous way here. So avoid this by generating our own
On 10/26/22 07:54, Jan Engelhardt wrote:
On Wednesday 2022-10-26 00:39, Bernhard Voelker wrote:
On 9/27/22 09:04, Andreas Schwab wrote:
On Sep 26 2022, Bernhard Voelker wrote:
https://lists.gnu.org/archive/html/bug-make/2022-09/msg00178.html
@Andreas: do you have news on this?
If there is any news you will find it in this thread.
How to continue with the new make version (https://build.opensuse.org/request/show/1006617)?
How about the following patch (for make-4.3.92)?
yes, why not? But the issue should be solved upstream (as well). I'm wondering why no other users of the upstream version have that problem ... Have a nice day, Berny
diff --git a/src/posixos.c b/src/posixos.c index 48e8cf5c..4e6cc9ec 100644 --- a/src/posixos.c +++ b/src/posixos.c @@ -139,9 +139,13 @@ unsigned int jobserver_setup (int slots, const char *style) { int r; + if (style == NULL) + { + style = "pipe"; + }
#if HAVE_MKFIFO - if (style == NULL || strcmp (style, "fifo") == 0) + if (strcmp (style, "fifo") == 0) { /* Unfortunately glibc warns about uses of mktemp even though we aren't using it in dangerous way here. So avoid this by generating our own
On Wed, Oct 26, 2022 at 2:54 AM Jan Engelhardt <jengelh@inai.de> wrote:
How about the following patch (for make-4.3.92)?
diff --git a/src/posixos.c b/src/posixos.c index 48e8cf5c..4e6cc9ec 100644 --- a/src/posixos.c +++ b/src/posixos.c @@ -139,9 +139,13 @@ unsigned int jobserver_setup (int slots, const char *style) { int r; + if (style == NULL) + { + style = "pipe"; + }
#if HAVE_MKFIFO - if (style == NULL || strcmp (style, "fifo") == 0) + if (strcmp (style, "fifo") == 0) { /* Unfortunately glibc warns about uses of mktemp even though we aren't using it in dangerous way here. So avoid this by generating our own
Yeah, +1 . or whatever make it default to "pipe" and we are done.
participants (5)
-
Andreas Schwab -
Bernhard Voelker -
Cristian Rodríguez -
Fabian Vogt -
Jan Engelhardt