On 04/19/2018 04:33 PM, Adrian Schröter wrote:

On Donnerstag, 19. April 2018, 14:49:40 CEST wrote Christian Bruckmayer:

...

On 04/19/2018 02:05 PM, Michael Schroeder wrote:

...

...

since last week our travis tests are running in docker containers based on SLE12 SP3, including API tests. That means that we currently run our tests against SLE12 SP3 - x86_64 twice. With that in mind I would like to update our project conf (in OBS:Server:Unstable) to disable the test suite there for build target SLE_12_SP3 - x86_64.

People that branch from that project and want it to be explicitly enabled would still be able to do that by switching disable_obs_test_suite in their branched project conf.

Since this can not be done via submit request I am writing this mail;-)

If you have any concerns, please raise them now. Hmm, I don't really like it. Who makes sure the containers include

On Thu, Apr 19, 2018 at 10:59:43AM +0200, Bjoern Geuken wrote: the same packages as our Build Service project? If both travis and OBS run the tests, I have more trust in the OBS run as it has the same build environment as the rpms that are created. https://build.opensuse.org/project/show/OBS:Server:Unstable:container:SLE12:...

The containers are built here. Sure there will be some small difference between the container and the actual obs-server package but we're close (close enough that I would say we can neglect the difference for O:S:U).

...

Why is this a problem at all? The test suite doesn't run that long, and I'm pretty sure we could make it faster. And if some tests are flaky, then it's much better to invest time to fix them. The test suite usually runs ~25 minutes in Travis, in OBS it takes > 1 hour (sometimes up to 2 hours). In this timeframe a rebuild usually get's triggered by merging a pull request. The problem is that we do not get many packages during the day so deployment is only possible in the morning or late afternoon.

https://travis-ci.org/openSUSE/open-build-service/builds https://build.opensuse.org/packages/obs-server/job_history/OBS:Server:Unstab...

Sure there is always room for improvement and make it faster but I doubt that we can make it twice as fast (or even more).

In Travis the problem is usually not that the tests are flaky, see travis build log (the one broken today was introduced by me by a rubocop issue): https://travis-ci.org/openSUSE/open-build-service/builds

...

(And all tests must work on non-SLE_12_SP3 nevertheless, so the change will not reduce work.) How do you mean? We test on SLE12 SP3 now in Travis and in the package, the problem is that we usually only get one package during the day. I think the point here is that you actually not to ask to *replace* the code running it inside of the package build, but to opt-out for SLE 12 SP3 repo build of OBS:Server:Unstable on build.opensuse.org only, right?

The containers on travis are a good addition, but they don't cover running the tests for all repos and test builds we do, not for our test instance and not for patched forks like IBS.

Assuming that the containers are always built constant based on the rpms from OBS:Server:Unstable this opt-out may be okay, but it is no replacement.

Is this a correct view?

Just asking, because I sometimes feel a bit alone fixing the problems in the package tests or the setup on build-test.o.o, but we would still maintain these together, right? Sure we do. And we deploy (almost) every day, so there aren't that many errors in the package tests (for SLE12 SP3). And as said, in Travis the tests are succeeding almost always! But we're moving away from the initial discussion :)

On the long run I would go actually the opposite way, avoid the need of external servers and seperate QA and build results in OBS...

For the distribution this may make sense. For our development workflow and for maintaining build.o.o this is not going to work for us. We NEED to get the test result on the pull request before we even merge to master. We want to get feedback as early as possible and deploy as often as possible. -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org