some late build chain changes in factory degrades build performance of some packages
Hi, some late changes in the build chain degrades build performance to a crawl: e.g. a package typically builds in under two minutes: https://build.opensuse.org/package/live_build_log/vdr:plugins/vdr-plugin-liv... openSUSE_Tumbleweed/x86_64 shows this now: [ 2624s] CC live.o [ 3161s] CC thread.o [ 3828s] CC tntconfig.o [ 4459s] CC setup.o [ 4933s] CC i18n.o [ 5492s] CC timers.o [ 6065s] CC tools.o [ 6730s] CC recman.o [ 7457s] CC tasks.o [ 8034s] CC status.o [ 8608s] CC epg_events.o [ 9116s] CC epgsearch.o Any idea, what could have caused this? Cheers, Pete -- Life without chameleons is possible, but pointless.
On Friday 2022-11-11 20:56, Hans-Peter Jansen wrote:
some late changes in the build chain degrades build performance to a crawl: e.g. a package typically builds in under two minutes:
https://build.opensuse.org/package/live_build_log/vdr:plugins/vdr-plugin-liv... openSUSE_Tumbleweed/x86_64
shows this now:
[ 2624s] CC live.o [ 3161s] CC thread.o
1. `osc build ...` in one terminal 2. Second terminal, strace -fe execve -p 24582 (pid of make) and I observed that make re-runs pkg-config a damn lot. What recently changed in openSUSE is the make program (4.3 -> 4.4). From make's changelog, I identify two changes that seem like they could be relevant: https://lists.gnu.org/archive/html/info-gnu/2022-10/msg00008.html >Previously each target in a explicit grouped target rule was >considered individually: if the targets needed by the build >were not out of date the recipe was not run even if other >targets in the group were out of date or >Previously makefile variables marked as export were not >exported to commands started by the $(shell ...) function. >Now, all exported variables are exported to $(shell ...). Of course, these alone do not constitute problem - we have over 1000 packages that exercise `make` one way or the other, but they do so through a proper means of Makefile generators like configure or cmake or something of the sort. 3. So we're looking at a shitty Makefile. vdr$ make -ddd clean Reading makefile 'Makefile'... grep: warning: stray \ before # Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Reading makefile 'global.mk' (search path) (no ~ expansion)... Makefile:27: not recursively expanding CFLAGS to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function ... Just look at the mess: PKGCFG = $(if $(VDRDIR),$(shell pkg-config --variable=$(1) $(VDRDIR)/vdr.pc),$(shell PKG_CONFIG_PATH="$$PKG_CONFIG_PATH:../../.." pkg-config --variable=$(1) vd... LIBDIR = $(call PKGCFG,libdir) LOCDIR = $(call PKGCFG,locdir) PLGCFG = $(call PKGCFG,plgcfg) RESDIR = $(call PKGCFG,resdir) ### The compiler options: export CFLAGS = $(call PKGCFG,cflags) export CXXFLAGS = $(call PKGCFG,cxxflags) ... ifeq ($(TNTNET-CONFIG),) TNTVERSION = $(shell pkg-config --modversion tntnet | sed -e's/\.//g' | sed -e's/pre.*//g' | awk '/^..$$/ { print $$1."000"} /^...$$/ { print $$1."00"} /^....$ CXXFLAGS += $(shell pkg-config --cflags tntnet) LIBS += $(shell pkg-config --libs tntnet) else TNTVERSION = $(shell tntnet-config --version | sed -e's/\.//g' | sed -e's/pre.*//g' | awk '/^..$$/ { print $$1."000"} /^...$$/ { print $$1."00"} /^....$$/ { pr CXXFLAGS += $(shell tntnet-config --cxxflags) LIBS += $(shell tntnet-config --libs) So remember what the changelog said: export all variables whenever $(shell ..) is run. The irony is that, to set up the environment for that shell run, it has to expand PKGCFG, which involves calling at least another $(shell ...). And that multiplies fast. *yuck*
Am Samstag, 12. November 2022, 16:56:35 CET schrieb Jan Engelhardt:
On Friday 2022-11-11 20:56, Hans-Peter Jansen wrote:
some late changes in the build chain degrades build performance to a crawl: e.g. a package typically builds in under two minutes:
https://build.opensuse.org/package/live_build_log/vdr:plugins/vdr-plugin-li ve/ openSUSE_Tumbleweed/x86_64
shows this now:
[ 2624s] CC live.o [ 3161s] CC thread.o
1. `osc build ...` in one terminal
2. Second terminal,
strace -fe execve -p 24582
(pid of make)
and I observed that make re-runs pkg-config a damn lot.
What recently changed in openSUSE is the make program (4.3 -> 4.4).
From make's changelog, I identify two changes that seem like they
could be relevant:
https://lists.gnu.org/archive/html/info-gnu/2022-10/msg00008.html
Previously each target in a explicit grouped target rule was considered individually: if the targets needed by the build were not out of date the recipe was not run even if other targets in the group were out of date
or
Previously makefile variables marked as export were not exported to commands started by the $(shell ...) function. Now, all exported variables are exported to $(shell ...).
Of course, these alone do not constitute problem - we have over 1000 packages that exercise `make` one way or the other, but they do so through a proper means of Makefile generators like configure or cmake or something of the sort.
3. So we're looking at a shitty Makefile.
vdr$ make -ddd clean Reading makefile 'Makefile'... grep: warning: stray \ before # Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Reading makefile 'global.mk' (search path) (no ~ expansion)... Makefile:27: not recursively expanding CFLAGS to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function ...
Just look at the mess:
PKGCFG = $(if $(VDRDIR),$(shell pkg-config --variable=$(1) $(VDRDIR)/vdr.pc),$(shell PKG_CONFIG_PATH="$$PKG_CONFIG_PATH:../../.." pkg-config --variable=$(1) vd... LIBDIR = $(call PKGCFG,libdir) LOCDIR = $(call PKGCFG,locdir) PLGCFG = $(call PKGCFG,plgcfg) RESDIR = $(call PKGCFG,resdir)
### The compiler options: export CFLAGS = $(call PKGCFG,cflags) export CXXFLAGS = $(call PKGCFG,cxxflags)
...
ifeq ($(TNTNET-CONFIG),) TNTVERSION = $(shell pkg-config --modversion tntnet | sed -e's/\.// g' | sed -e's/pre.*//g' | awk '/^..$$/ { print $$1."000"} /^...$$/ { print $$1."00"} /^....$ CXXFLAGS += $(shell pkg-config --cflags tntnet) LIBS += $(shell pkg-config --libs tntnet) else TNTVERSION = $(shell tntnet-config --version | sed -e's/\.//g' | sed -e's/pre.*//g' | awk '/^..$$/ { print $$1."000"} /^...$$/ { print $$1."00"} /^....$$/ { pr CXXFLAGS += $(shell tntnet-config --cxxflags) LIBS += $(shell tntnet-config --libs)
So remember what the changelog said: export all variables whenever $(shell ..) is run. The irony is that, to set up the environment for that shell run, it has to expand PKGCFG, which involves calling at least another $(shell ...).
And that multiplies fast.
*yuck*
First at all, thanks Jan! Your analysis is much appreciated.. Oh my, wth. This reveals a bunch of questions: * How to address this mess? Probably preparing the environment before in order to avoid those shell expansions within the Makefile will cut it * The new make behaviour is kind of questionable, * since it can be easily tricked into a DoS, since it might behave like a fork bomb now * feels like a regression nevertheless Reminds me of a saying: You don't understand recursion unless you understand recursion! Thanks again, Pete
| On Sat, Nov 12, 2022 at 2:37 PM Hans-Peter Jansen <hpj@urpla.net> wrote:
* How to address this mess?
It shouldn't be that difficult to write a meson.build and ninja build it.
Probably preparing the environment before in order to avoid those shell expansions within the Makefile will cut it * The new make behaviour is kind of questionable, * since it can be easily tricked into a DoS, since it might behave like a fork bomb now * feels like a regression nevertheless
Reminds me of a saying:
Just don't. Normal human beings and even skilled developers suck at writing makefiles. if it is not this it will be another problem. On Sat, Nov 12, 2022 at 2:37 PM Hans-Peter Jansen <hpj@urpla.net> wrote:
Am Samstag, 12. November 2022, 16:56:35 CET schrieb Jan Engelhardt:
On Friday 2022-11-11 20:56, Hans-Peter Jansen wrote:
some late changes in the build chain degrades build performance to a crawl: e.g. a package typically builds in under two minutes:
https://build.opensuse.org/package/live_build_log/vdr:plugins/vdr-plugin-li ve/ openSUSE_Tumbleweed/x86_64
shows this now:
[ 2624s] CC live.o [ 3161s] CC thread.o
1. `osc build ...` in one terminal
2. Second terminal,
strace -fe execve -p 24582
(pid of make)
and I observed that make re-runs pkg-config a damn lot.
What recently changed in openSUSE is the make program (4.3 -> 4.4).
From make's changelog, I identify two changes that seem like they
could be relevant:
https://lists.gnu.org/archive/html/info-gnu/2022-10/msg00008.html
>Previously each target in a explicit grouped target rule was >considered individually: if the targets needed by the build >were not out of date the recipe was not run even if other >targets in the group were out of date
or
>Previously makefile variables marked as export were not >exported to commands started by the $(shell ...) function. >Now, all exported variables are exported to $(shell ...).
Of course, these alone do not constitute problem - we have over 1000 packages that exercise `make` one way or the other, but they do so through a proper means of Makefile generators like configure or cmake or something of the sort.
3. So we're looking at a shitty Makefile.
vdr$ make -ddd clean Reading makefile 'Makefile'... grep: warning: stray \ before # Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Reading makefile 'global.mk' (search path) (no ~ expansion)... Makefile:27: not recursively expanding CFLAGS to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function Makefile:18: not recursively expanding PKGCFG to export to shell function ...
Just look at the mess:
PKGCFG = $(if $(VDRDIR),$(shell pkg-config --variable=$(1) $(VDRDIR)/vdr.pc),$(shell PKG_CONFIG_PATH="$$PKG_CONFIG_PATH:../../.." pkg-config --variable=$(1) vd... LIBDIR = $(call PKGCFG,libdir) LOCDIR = $(call PKGCFG,locdir) PLGCFG = $(call PKGCFG,plgcfg) RESDIR = $(call PKGCFG,resdir)
### The compiler options: export CFLAGS = $(call PKGCFG,cflags) export CXXFLAGS = $(call PKGCFG,cxxflags)
...
ifeq ($(TNTNET-CONFIG),) TNTVERSION = $(shell pkg-config --modversion tntnet | sed -e's/\.// g' | sed -e's/pre.*//g' | awk '/^..$$/ { print $$1."000"} /^...$$/ { print $$1."00"} /^....$ CXXFLAGS += $(shell pkg-config --cflags tntnet) LIBS += $(shell pkg-config --libs tntnet) else TNTVERSION = $(shell tntnet-config --version | sed -e's/\.//g' | sed -e's/pre.*//g' | awk '/^..$$/ { print $$1."000"} /^...$$/ { print $$1."00"} /^....$$/ { pr CXXFLAGS += $(shell tntnet-config --cxxflags) LIBS += $(shell tntnet-config --libs)
So remember what the changelog said: export all variables whenever $(shell ..) is run. The irony is that, to set up the environment for that shell run, it has to expand PKGCFG, which involves calling at least another $(shell ...).
And that multiplies fast.
*yuck*
First at all, thanks Jan! Your analysis is much appreciated..
Oh my, wth.
This reveals a bunch of questions: * How to address this mess? Probably preparing the environment before in order to avoid those shell expansions within the Makefile will cut it * The new make behaviour is kind of questionable, * since it can be easily tricked into a DoS, since it might behave like a fork bomb now * feels like a regression nevertheless
Reminds me of a saying:
You don't understand recursion unless you understand recursion!
Thanks again, Pete
On 13.11.22 12:59, Cristian Rodríguez wrote:
|
On Sat, Nov 12, 2022 at 2:37 PM Hans-Peter Jansen <hpj@urpla.net> wrote:
* How to address this mess?
It shouldn't be that difficult to write a meson.build and ninja build it.
You are free to try convincing upstream to switch to that stuff :-) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
On Sunday 2022-11-13 15:06, Stefan Seyfried wrote:
On 13.11.22 12:59, Cristian Rodríguez wrote:
On Sat, Nov 12, 2022 at 2:37 PM Hans-Peter Jansen <hpj@urpla.net> wrote:
* How to address this mess?
It shouldn't be that difficult to write a meson.build and ninja build it.
You are free to try convincing upstream to switch to that stuff :-)
Well if they don't, upstream is just going to play themselves. Because make 4.4 will make an entrance into practically all distros eventually.
On 13.11.22 16:44, Jan Engelhardt wrote:
On Sunday 2022-11-13 15:06, Stefan Seyfried wrote:
On 13.11.22 12:59, Cristian Rodríguez wrote:
It shouldn't be that difficult to write a meson.build and ninja build it.
You are free to try convincing upstream to switch to that stuff :-)
Well if they don't, upstream is just going to play themselves. Because make 4.4 will make an entrance into practically all distros eventually.
The Makefile was trivial to patch so that it works reasonably well again, so there really was no need to introduce this newfangled meson stuff at all ;-) -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman
participants (4)
-
Cristian Rodríguez -
Hans-Peter Jansen -
Jan Engelhardt -
Stefan Seyfried