[opensuse-packaging] Non-blocking %check
There has been some work and advice recently about improving the parallelization of project builds. However, one issue that occurred to me is that a lot of time can be spent running the %check scriptlet that shouldn't have any impact on the final RPMs. My idea was that perhaps the RPM could be used for building packages that depend on it while the %check scriptlet is running. If the %check succeeds, then everything continues as normal. If not, then builds that depend on it are canceled and RPMs that depend on it are invalidated before being published. I understand that this is a big change. In addition to needed big changes in how OSC works, this would also require a fundamental change in the %check macro. It would mean that things that happen during the %check scriptlet would no longer have any impact on the final RPMs. Personally I think this is a feature rather than a bug, I always thought it was strange that tests are allowed to change the package, but it is a big behavior change. I am not sure this is worth the effort to implement or worth breaking existing RPM semantics. But if we are having a problem with the builds not being as parallel as they could be, this might be something at least to consider. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 04/20/2017 05:08 AM, Todd Rme wrote:
There has been some work and advice recently about improving the parallelization of project builds. However, one issue that occurred to me is that a lot of time can be spent running the %check scriptlet that shouldn't have any impact on the final RPMs.
My idea was that perhaps the RPM could be used for building packages that depend on it while the %check scriptlet is running. If the %check succeeds, then everything continues as normal. If not, then builds that depend on it are canceled and RPMs that depend on it are invalidated before being published.
I understand that this is a big change. In addition to needed big changes in how OSC works, this would also require a fundamental change in the %check macro. It would mean that things that happen during the %check scriptlet would no longer have any impact on the final RPMs. Personally I think this is a feature rather than a bug, I always thought it was strange that tests are allowed to change the package, but it is a big behavior change.
I am not sure this is worth the effort to implement or worth breaking existing RPM semantics. But if we are having a problem with the builds not being as parallel as they could be, this might be something at least to consider.
I'm not sure to which discussion you refer to (as you didn't point to it). I can only tell how it is done for the coreutils package [1] where the full check (`make check-very-expensive`) may run between ~15min and ~1.5h depending on the architecture. Without the checks, the package is built in ~3-15min. There's a 2nd spec file 'coreutils-testsuite.spec' which is generated from the regular 'coreutils.spec', and the %check section is only used by that secondary package: %check %if "%{name}" == "coreutils-testsuite" ... # Avoid parallel make, because otherwise some timeout based tests like # rm/ext3-perf may fail due to high CPU or IO load. make check-very-expensive \ && ... %endif Does this idea go into the direction you're thinking? [1] https://build.opensuse.org/package/view_file/Base:System/coreutils/coreutils... Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thursday 20 April 2017, Bernhard Voelker wrote:
On 04/20/2017 05:08 AM, Todd Rme wrote:
There has been some work and advice recently about improving the parallelization of project builds. However, one issue that occurred to me is that a lot of time can be spent running the %check scriptlet that shouldn't have any impact on the final RPMs.
My idea was that perhaps the RPM could be used for building packages that depend on it while the %check scriptlet is running. If the %check succeeds, then everything continues as normal. If not, then builds that depend on it are canceled and RPMs that depend on it are invalidated before being published.
I understand that this is a big change. In addition to needed big changes in how OSC works, this would also require a fundamental change in the %check macro. It would mean that things that happen during the %check scriptlet would no longer have any impact on the final RPMs. Personally I think this is a feature rather than a bug, I always thought it was strange that tests are allowed to change the package, but it is a big behavior change.
I am not sure this is worth the effort to implement or worth breaking existing RPM semantics. But if we are having a problem with the builds not being as parallel as they could be, this might be something at least to consider.
I'm not sure to which discussion you refer to (as you didn't point to it). I can only tell how it is done for the coreutils package [1] where the full check (`make check-very-expensive`) may run between ~15min and ~1.5h depending on the architecture. Without the checks, the package is built in ~3-15min. There's a 2nd spec file 'coreutils-testsuite.spec' which is generated from the regular 'coreutils.spec', and the %check section is only used by that secondary package:
%check %if "%{name}" == "coreutils-testsuite" ... # Avoid parallel make, because otherwise some timeout based tests like # rm/ext3-perf may fail due to high CPU or IO load. make check-very-expensive \ && ... %endif
Does this idea go into the direction you're thinking?
This is a nice idea but it has a few drawbacks: 1.A bad coreutils-testsuite does not seem to prevent automatically that the coreutils package would be released to the users. 2.coreutils-testsuite is not testing the *same* binaries which we install finally in the distro. Bugs or inconsistencies in OSB or rpm could cause that "coreutils" builds broken binaries but "coreutils-testsuite" looks ok. What I would really like is that project's like coreutils would have a "make check*" target which runs tests against the installed coreutils. Similar to some of our python packages, users could even test their final installation, like python -c "import numpy; numpy.test()" cu, Rudi -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 04/20/2017 01:30 PM, Ruediger Meier wrote:
This is a nice idea but it has a few drawbacks: 1.A bad coreutils-testsuite does not seem to prevent automatically that the coreutils package would be released to the users.
there is still the package maintainer who is repsonsible for forwarding packages ... so I don't see this danger.
2.coreutils-testsuite is not testing the *same* binaries which we install finally in the distro. Bugs or inconsistencies in OSB or rpm could cause that "coreutils" builds broken binaries but "coreutils-testsuite" looks ok.
That's a valid point though ...
What I would really like is that project's like coreutils would have a "make check*" target which runs tests against the installed coreutils.
... it has been like that for quite a while now. I don't know what was the motivation behind the split, but I can imagine that this speeded up the build cycles on OBS dramatically. Thus said, I'm not against removing 'coreutils-testsuite'. Maybe it'd be an improvement if only the smaller (faster) 'make check' is run for the coreutils package while it would run the slower 'make check-very-expensive' tests in the coreutils-testsuite case. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thursday 20 April 2017, Bernhard Voelker wrote:
On 04/20/2017 01:30 PM, Ruediger Meier wrote:
This is a nice idea but it has a few drawbacks: 1.A bad coreutils-testsuite does not seem to prevent automatically that the coreutils package would be released to the users.
there is still the package maintainer who is repsonsible for forwarding packages ... so I don't see this danger.
2.coreutils-testsuite is not testing the *same* binaries which we install finally in the distro. Bugs or inconsistencies in OSB or rpm could cause that "coreutils" builds broken binaries but "coreutils-testsuite" looks ok.
That's a valid point though ...
What I would really like is that project's like coreutils would have a "make check*" target which runs tests against the installed coreutils.
... it has been like that for quite a while now. I don't know what was the motivation behind the split, but I can imagine that this speeded up the build cycles on OBS dramatically.
Yes, the non-testsuite package has less "BuildRequires" to prevent "unnecessary" rebuilds. But different "BuildRequires" even increases the probability that the installed and tested binaries could differ. IHMO our OBS bootstrapping way to rebuild again and again until everything converges to reproducable packages is bad especially when we add such "build cycle optimizations". It makes spec files and tests very ugly, complicated or even unsafe (e.g. util-linux). It's also a pain to review submit requests for such packages having duplicated spec files and changelogs.
Thus said, I'm not against removing 'coreutils-testsuite'. Maybe it'd be an improvement if only the smaller (faster) 'make check' is run for the coreutils package while it would run the slower 'make check-very-expensive' tests in the coreutils-testsuite case.
Have a nice day, Berny
cu, Rudi -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
JFYI, the idea to handle build results and QA results seperate exists since quite some time. Means being able to switch in repository or request configurations to accept or not accept QA failures independend of source changes. However, it is unlikely that we work on that soon, sorry. Nevertheless, I think the seperate -testsuite packages are a proper workaround for many cases. You can still see the results, and still get build results soon. On Donnerstag, 20. April 2017, 15:40:29 CEST wrote Ruediger Meier:
On Thursday 20 April 2017, Bernhard Voelker wrote:
On 04/20/2017 01:30 PM, Ruediger Meier wrote:
This is a nice idea but it has a few drawbacks: 1.A bad coreutils-testsuite does not seem to prevent automatically that the coreutils package would be released to the users.
there is still the package maintainer who is repsonsible for forwarding packages ... so I don't see this danger.
2.coreutils-testsuite is not testing the *same* binaries which we install finally in the distro. Bugs or inconsistencies in OSB or rpm could cause that "coreutils" builds broken binaries but "coreutils-testsuite" looks ok.
That's a valid point though ...
What I would really like is that project's like coreutils would have a "make check*" target which runs tests against the installed coreutils.
... it has been like that for quite a while now. I don't know what was the motivation behind the split, but I can imagine that this speeded up the build cycles on OBS dramatically.
Yes, the non-testsuite package has less "BuildRequires" to prevent "unnecessary" rebuilds. But different "BuildRequires" even increases the probability that the installed and tested binaries could differ.
IHMO our OBS bootstrapping way to rebuild again and again until everything converges to reproducable packages is bad especially when we add such "build cycle optimizations". It makes spec files and tests very ugly, complicated or even unsafe (e.g. util-linux). It's also a pain to review submit requests for such packages having duplicated spec files and changelogs.
Thus said, I'm not against removing 'coreutils-testsuite'. Maybe it'd be an improvement if only the smaller (faster) 'make check' is run for the coreutils package while it would run the slower 'make check-very-expensive' tests in the coreutils-testsuite case.
Have a nice day, Berny
cu, Rudi
-- Adrian Schroeter email: adrian@suse.de SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thursday 20 April 2017, Adrian Schröter wrote:
JFYI, the idea to handle build results and QA results seperate exists since quite some time. Means being able to switch in repository or request configurations to accept or not accept QA failures independend of source changes.
However, it is unlikely that we work on that soon, sorry.
Nevertheless, I think the seperate -testsuite packages are a proper workaround for many cases. You can still see the results, and still get build results soon.
No, you don't see *the* results. You can see *some* results of different binaries which were built using a different build script (spec file), different BuildRequires and probably on a different build host. To fix that we could add a subpackage providing the whole original build directory and install this later (at QA time) to run the missing check only. cu, Rudi -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 04/20/2017 04:18 PM, Ruediger Meier wrote:
No, you don't see *the* results. You can see *some* results of different binaries which were built using a different build script (spec file), different BuildRequires and probably on a different build host.
The additional BuildRequires for the coreutils-testsuite are only used for %check, but don't lead to a different build result: %if "%{name}" == "coreutils-testsuite" BuildRequires: acl BuildRequires: gdb BuildRequires: perl-Expect BuildRequires: python-pyinotify BuildRequires: strace BuildRequires: timezone %ifarch %ix86 x86_64 ppc ppc64 s390x armv7l armv7hl BuildRequires: valgrind %endif %endif All the other packages - which are used for building - are identical. There might be still some risk, but I've never seen any fallout from this (which should be theoretically be 50% of such cases, i.e., where the binaries in coreutils are good but those in the testsuite are not). I think it's more risky that sometimes you have to mask a certain test which does simply not work on OBS; and even others are just SKIPped because the prerequisites on OBS are not met. And finally, I'd rather like to be able to run some tests as root on OBS to get more test coverage (yes, there are some special tests written to be run as root), but this is a different story. After all, I'm fine with the current splitting. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thursday 20 April 2017, Bernhard Voelker wrote:
On 04/20/2017 04:18 PM, Ruediger Meier wrote:
No, you don't see *the* results. You can see *some* results of different binaries which were built using a different build script (spec file), different BuildRequires and probably on a different build host.
The additional BuildRequires for the coreutils-testsuite are only used for %check, but don't lead to a different build result:
I believe you that this is true. But OBS is designed to "not believe" us human maintainers. Normally it rebuilds indomitably whenever a dependency changes. The OBS design is even somehow obsessed about reproducible builds. But why not in this important case where we want to check *the* *built* binaries?
%if "%{name}" == "coreutils-testsuite" BuildRequires: acl BuildRequires: gdb
Just an example. What if one day gdb would bring us another whatever-devel package and ./configure would be happy to use it?
BuildRequires: perl-Expect BuildRequires: python-pyinotify BuildRequires: strace BuildRequires: timezone %ifarch %ix86 x86_64 ppc ppc64 s390x armv7l armv7hl BuildRequires: valgrind %endif %endif
Couldn't we just remove that ifdef above and tell OBS to not trigger coreutils rebuild if these packages changes? I mean I find it really too strange nowadays. For util-linux I was not even allowed to install "procps" anymore. We optimzed away the simple "ps" command which is an absolute basic POSIX requirement!
All the other packages - which are used for building - are identical.
There might be still some risk, but I've never seen any fallout from this (which should be theoretically be 50% of such cases, i.e., where the binaries in coreutils are good but those in the testsuite are not).
I have also never seen any fallout for many other cases. I would say the most rebuilds are not nessesary. Nethertheless we do all theoretically needed rebuilds to be on the safe side. Just the "unimportant" testing has been disabled to make things faster ... cu, Rudi -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On Thu, 20 Apr 2017, Ruediger Meier wrote:
No, you don't see *the* results. You can see *some* results of different binaries which were built using a different build script (spec file), different BuildRequires and probably on a different build host.
While your arguments are valid and true in the fullest generic sense, in the particular case we're talking about here (and in fact for most such cases) it's moot. The binaries _are_ the same, so the tests are valid. And while it is also true that it would be even nicer if more packages would support installed testing, that's something for upstream to solve usually. (Sometimes you can get away with running make check with the appropriate setting of various makefile variables). But it would also be nicer if I had a pink zombie pony; alas it is not so. So, for the time being, and with the realities we're working in, and considering that some packages are in fact in the critical path and hence create a _real_ problem when they build for a long time, the %-testsuite extra .spec files are the sensible work around. The other realistic work-around would be to not run the testsuite at all, and that would be worse.
To fix that we could add a subpackage providing the whole original build directory and install this later (at QA time) to run the missing check only.
True, that would work as well. It would also create many new extremely large binary packages nobody except automatic testers would use. If it actually solves any problem we in reality have is doubtful. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On Thursday 20 April 2017, Michael Matz wrote:
Hi,
On Thu, 20 Apr 2017, Ruediger Meier wrote:
No, you don't see *the* results. You can see *some* results of different binaries which were built using a different build script (spec file), different BuildRequires and probably on a different build host.
While your arguments are valid and true in the fullest generic sense, in the particular case we're talking about here (and in fact for most such cases) it's moot. The binaries _are_ the same, so the tests are valid.
And while it is also true that it would be even nicer if more packages would support installed testing, that's something for upstream to solve usually. (Sometimes you can get away with running make check with the appropriate setting of various makefile variables). But it would also be nicer if I had a pink zombie pony; alas it is not so.
So, for the time being, and with the realities we're working in, and considering that some packages are in fact in the critical path and hence create a _real_ problem when they build for a long time, the %-testsuite extra .spec files are the sensible work around. The other realistic work-around would be to not run the testsuite at all, and that would be worse.
I agree that splitting off *testsuite is not the most critical thing. However my concerns are not only about the *testsuite packages but mainly about our general way how to optimize the build cycles. Look at our util-linux package. It's almost impossible to understand these spec files and AFAIK no other distro is building ul in a similar manner, completely different than supposed by upstream and autotools inventors.
To fix that we could add a subpackage providing the whole original build directory and install this later (at QA time) to run the missing check only.
True, that would work as well. It would also create many new extremely large binary packages nobody except automatic testers would use. If it actually solves any problem we in reality have is doubtful.
Regarding reality problems. Here you see an accepted and forwarded submit request where all the duplicated spec files are not in sync, applying different patch-sets: https://build.opensuse.org/request/show/487529 cu, Rudi -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Hi, On Thu, 20 Apr 2017, Ruediger Meier wrote:
I agree that splitting off *testsuite is not the most critical thing.
However my concerns are not only about the *testsuite packages but mainly about our general way how to optimize the build cycles. Look at our util-linux package.
So, you have a problem with one specific package. I don't see how that's related to build cycle length, the optimization of it or the %check section.
It's almost impossible to understand these spec files and AFAIK no other distro is building ul in a similar manner, completely different than supposed by upstream and autotools inventors.
Hmm, don't really see what you mean, some things therein could also be done by patch instead of in place editing, but otherwise it looks fairly standard if a bit verbose.
Regarding reality problems. Here you see an accepted and forwarded submit request where all the duplicated spec files are not in sync, applying different patch-sets: https://build.opensuse.org/request/show/487529
That's a bug, and it shouldn't have been accepted. Alex forgot to call pre_checkin.sh which must be done for every package that contains one (and is even hinted at in the very .spec file itself). That should probably be automatically checked somehow. Ciao, Michael. -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
Am 20.04.2017 um 16:18 schrieb Ruediger Meier:
On Thursday 20 April 2017, Adrian Schröter wrote:
Nevertheless, I think the seperate -testsuite packages are a proper workaround for many cases. You can still see the results, and still get build results soon.
No, you don't see *the* results. You can see *some* results of different binaries which were built using a different build script (spec file), different BuildRequires and probably on a different build host.
You're making wrong assumptions, those are individual packagers' faults.
To fix that we could add a subpackage providing the whole original build directory and install this later (at QA time) to run the missing check only.
Take a look at qemu-testsuite. I tackled that very issue by overriding variables and/or symlinking the installed binaries into the build tree, avoiding lengthy rebuilds and testing the very binaries the user gets. Ugly, but it worked sufficiently. Regards, Andreas -- SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
* Bernhard Voelker (mail@bernhard-voelker.de) [20170420 14:28]:
... it has been like that for quite a while now. I don't know what was the motivation behind the split, but I can imagine that this speeded up the build cycles on OBS dramatically.
With coreutils being a very basic core package, the split had two main reasons. The first one was reducing the build time and the second one that running the testsuite adds dependencies on packages that the build doesn't have. Being that at the time the split was requested it was hinted that it was planned to add to obs so that you could fail the build of another package I grudgingly did the split. Of course that feature never materialised... Philipp -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (7)
-
Adrian Schröter -
Andreas Färber -
Bernhard Voelker -
Michael Matz -
Philipp Thomas -
Ruediger Meier -
Todd Rme