[opensuse-packaging] Something like BuildRecommends to increase test coverage?
The coreutils-testsuite runs best with a list of tools like valgrind, gdb, python-inotify etc. but some of the tools are not available on some platforms. The problems is: a) if I add a BuildRequires on platforms that don't have that package, then the whole testsuite build on that platform fails. b) Now I can add a guard around the BuildRequires per platform/OS version manually, but: b1) manually maintaining such a list is tedious, b2) if the guards take out to many of BuildRequires, then the coverage is not as good as it could be. This effect is transient if a package gets available on a new platform in future. As a packager, I'd love to just add the list of additional tools the test may or may not use, and the build not failing if the package is not available on that platform like: BuildRecommends: valgrind or BuildSuggests: valgrind Is there something automatic like this? Thanks & have a nice day, Berny -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 3/11/19 9:44 AM, Bernhard Voelker wrote:
As a packager, I'd love to just add the list of additional tools the test may or may not use, and the build not failing if the package is not available on that platform like:
BuildRecommends: valgrind or BuildSuggests: valgrind
Is there something automatic like this?
The argument against having something like this is build reproducability and predictability. Sometimes a package can be available and sometimes it may not. And this can trigger build problems in your package under some situations. The best way to make sure that the package at least builds and tests properly is to have one predictable build state. Many, many years ago I would love to have this feature already for some software. This is a nice feature for development, but not a nice feature for deployment. In deployment, less variability is your friend. - Adam -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 3/11/19 2:51 PM, Adam Majer wrote:
On 3/11/19 9:44 AM, Bernhard Voelker wrote:
As a packager, I'd love to just add the list of additional tools the test may or may not use, and the build not failing if the package is not available on that platform like:
BuildRecommends: valgrind or BuildSuggests: valgrind
Is there something automatic like this?
The argument against having something like this is build reproducability and predictability. Sometimes a package can be available and sometimes it may not. And this can trigger build problems in your package under some situations. The best way to make sure that the package at least builds and tests properly is to have one predictable build state.
Many, many years ago I would love to have this feature already for some software. This is a nice feature for development, but not a nice feature for deployment. In deployment, less variability is your friend.
That's why the *-testsuite is a subpackage: the main package builds fine with a certain set of tests/integrity, and the testsuite could pick whatever is available: e.g. currently there's no valgrind for armv7l available because "nothing provides glibc < 2.28.0". I guess this one will go away in a couple of weeks. Meanwhile, I'd have to take out armv7l from the guards around the BuildRequires:valgrind to get something tested at all, and I won't be given a notice when it would be available again, so that I could take it back in. Well, I understand your point as well, but then: is there a way to automate the maintenance of the guards for each utility package needed? I mean currently it's just repetitive try&error to find the right combination. Thanks & have a nice day, Berny -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 3/11/19 11:47 PM, Bernhard Voelker wrote:
That's why the *-testsuite is a subpackage: the main package builds fine with a certain set of tests/integrity, and the testsuite could pick whatever is available: e.g. currently there's no valgrind for armv7l available because "nothing provides glibc < 2.28.0". I guess this one will go away in a couple of weeks. Meanwhile, I'd have to take out armv7l from the guards around the BuildRequires:valgrind to get something tested at all, and I won't be given a notice when it would be available again, so that I could take it back in.
2 scenarios: Run the valgrind tests only on x86 ? Iff the package doesn't have some arch specific code and is all generic compiled for all arches, then running valgrind on all of them doesn't give you anything except slow tests. You are essentially testing the compiler at this point, not the application. If you do have platform specific code, then it's a different argument of course. Then you could have *-testsuite and *-valgrind-testsuite and run the valgrind-only stuff in the valgrind scenario. And if the valgrind is missing, then that should be OK too as you don't ship that binary. I would also run these tests in the devel project or your home only, not in actual Factory. That way you can keep track of them without having these binaries in factory. You can use %bcond_with guards for the tests and use project config to enable them. - Adam -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 2019-03-12 11:19:52 +0100, Adam Majer wrote:
On 3/11/19 11:47 PM, Bernhard Voelker wrote:
That's why the *-testsuite is a subpackage: the main package builds fine with a certain set of tests/integrity, and the testsuite could pick whatever is available: e.g. currently there's no valgrind for armv7l available because "nothing provides glibc < 2.28.0". I guess this one will go away in a couple of weeks. Meanwhile, I'd have to take out armv7l from the guards around the BuildRequires:valgrind to get something tested at all, and I won't be given a notice when it would be available again, so that I could take it back in.
2 scenarios:
Run the valgrind tests only on x86 ? Iff the package doesn't have some arch specific code and is all generic compiled for all arches, then running valgrind on all of them doesn't give you anything except slow tests. You are essentially testing the compiler at this point, not the application.
If you do have platform specific code, then it's a different argument of course. Then you could have *-testsuite and *-valgrind-testsuite and run the valgrind-only stuff in the valgrind scenario. And if the valgrind is missing, then that should be OK too as you don't ship that binary.
I would also run these tests in the devel project or your home only, not in actual Factory. That way you can keep track of them without having these binaries in factory. You can use %bcond_with guards for the tests and use project config to enable them.
If you just want to run the testsuite in your home or devel project (or a specific testing project), you could do the following: - create a new "coreutils-testsuite-deps" package - add a "Recommends: valgrind", "Recommends: xyz" etc. to the package - add a "BuildRequires: coreutils-testsuite-deps" line to the coreutils-testsuite spec - add a "Expandflags: dorecommends" to your project config Now, if you build the coreutils-testsuite package, the valgrind package will be installed if it is present and all its deps etc. can be satisfied. But if you do it this way, you will _NEVER_ run the full testsuite in the openSUSE:Factory project (since recommends are ignored during expansion). Hence, I'm not sure if this "hack" works for you... Marcus -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 3/12/19 5:29 PM, Marcus Hüwe wrote:
If you just want to run the testsuite in your home or devel project (or a specific testing project), you could do the following:
- create a new "coreutils-testsuite-deps" package - add a "Recommends: valgrind", "Recommends: xyz" etc. to the package - add a "BuildRequires: coreutils-testsuite-deps" line to the coreutils-testsuite spec - add a "Expandflags: dorecommends" to your project config
Now, if you build the coreutils-testsuite package, the valgrind package will be installed if it is present and all its deps etc. can be satisfied.
But if you do it this way, you will _NEVER_ run the full testsuite in the openSUSE:Factory project (since recommends are ignored during expansion). Hence, I'm not sure if this "hack" works for you...
Thanks, that started to sound like a good recipe ... but you are right: I think I don't want to rely on running the *-testsuite in the home project only. I don't know much about it: could we have *-testsuite in a different project with "Expandflags: dorecommends" set while the main package stays in Base:System? Do/don't others have similar issues? Have a nice day, Berny -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
On 3/14/19 8:32 AM, Bernhard Voelker wrote:
Thanks, that started to sound like a good recipe ... but you are right: I think I don't want to rely on running the *-testsuite in the home project only.
I don't know much about it: could we have *-testsuite in a different project with "Expandflags: dorecommends" set while the main package stays in Base:System? Do/don't others have similar issues?
Of course you can create a subproject that does that. Then just use the build repositories from Base:System and you have a repo from Build:System + your subproject. And I say subproject, but it could be anywhere, in Build:System:Test or home:...:Build_system_test. You can create as many subprojects as required to run different combinations of tests, if you like to. I assume the purpose of this is to let *maintainer* know when the tests are failing. You can run the tests *vs.* Tumbleweed without actually shipping anything *in* Tumbleweed. - Adam -- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-packaging+owner@opensuse.org
participants (3)
-
Adam Majer -
Bernhard Voelker -
Marcus Hüwe