Am Samstag, 12. November 2022, 16:56:35 CET schrieb Jan Engelhardt:
On Friday 2022-11-11 20:56, Hans-Peter Jansen wrote:
some late changes in the build chain degrades build performance to a crawl: e.g. a package typically builds in under two minutes:
https://build.opensuse.org/package/live_build_log/vdr:plugins/vdr-plugin-li ve/ openSUSE_Tumbleweed/x86_64
shows this now:
[ 2624s] CC live.o [ 3161s] CC thread.o
1. `osc build ...` in one terminal
2. Second terminal,
strace -fe execve -p 24582
(pid of make)
and I observed that make re-runs pkg-config a damn lot.
What recently changed in openSUSE is the make program (4.3 -> 4.4).
From make's changelog, I identify two changes that seem like they
could be relevant:
https://lists.gnu.org/archive/html/info-gnu/2022-10/msg00008.html
Previously each target in a explicit grouped target rule was considered individually: if the targets needed by the build were not out of date the recipe was not run even if other targets in the group were out of date
or
Previously makefile variables marked as export were not exported to commands started by the $(shell ...) function. Now, all exported variables are exported to $(shell ...).
Of course, these alone do not constitute problem - we have over 1000 packages that exercise `make` one way or the other, but they do so through a proper means of Makefile generators like configure or cmake or something of the sort.
3. So we're looking at a shitty Makefile.
vdr$ make -ddd clean Reading makefile 'Makefile'... grep: warning: stray \ before # Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Reading makefile 'global.mk' (search path) (no ~ expansion)... Makefile:27: not recursively expanding CFLAGS to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function ...
Just look at the mess:
PKGCFG = $(if $(VDRDIR),$(shell pkg-config --variable=$(1) $(VDRDIR)/vdr.pc),$(shell PKG_CONFIG_PATH="$$PKG_CONFIG_PATH:../../.." pkg-config --variable=$(1) vd... LIBDIR = $(call PKGCFG,libdir) LOCDIR = $(call PKGCFG,locdir) PLGCFG = $(call PKGCFG,plgcfg) RESDIR = $(call PKGCFG,resdir)
### The compiler options: export CFLAGS = $(call PKGCFG,cflags) export CXXFLAGS = $(call PKGCFG,cxxflags)
...
ifeq ($(TNTNET-CONFIG),) TNTVERSION = $(shell pkg-config --modversion tntnet | sed -e's/\.// g' | sed -e's/pre.*//g' | awk '/^..$$/ { print $$1."000"} /^...$$/ { print $$1."00"} /^....$ CXXFLAGS += $(shell pkg-config --cflags tntnet) LIBS += $(shell pkg-config --libs tntnet) else TNTVERSION = $(shell tntnet-config --version | sed -e's/\.//g' | sed -e's/pre.*//g' | awk '/^..$$/ { print $$1."000"} /^...$$/ { print $$1."00"} /^....$$/ { pr CXXFLAGS += $(shell tntnet-config --cxxflags) LIBS += $(shell tntnet-config --libs)
So remember what the changelog said: export all variables whenever $(shell ..) is run. The irony is that, to set up the environment for that shell run, it has to expand PKGCFG, which involves calling at least another $(shell ...).
And that multiplies fast.
*yuck*
First at all, thanks Jan! Your analysis is much appreciated.. Oh my, wth. This reveals a bunch of questions: * How to address this mess? Probably preparing the environment before in order to avoid those shell expansions within the Makefile will cut it * The new make behaviour is kind of questionable, * since it can be easily tricked into a DoS, since it might behave like a fork bomb now * feels like a regression nevertheless Reminds me of a saying: You don't understand recursion unless you understand recursion! Thanks again, Pete