Mailinglist Archive: opensuse-buildservice (43 mails)

< Previous Next >
[opensuse-buildservice] Re: [bs-team] [buildops] Replace rpm test suite for SLE12 with our travis test suite (based on SLE12 now)


On 04/19/2018 04:33 PM, Adrian Schröter wrote:
On Donnerstag, 19. April 2018, 14:49:40 CEST wrote Christian Bruckmayer:
On 04/19/2018 02:05 PM, Michael Schroeder wrote:
On Thu, Apr 19, 2018 at 10:59:43AM +0200, Bjoern Geuken wrote:
since last week our travis tests are running in docker containers based on
SLE12 SP3, including API tests. That means that we currently run our tests
against SLE12 SP3 - x86_64 twice.
With that in mind I would like to update our project conf (in
OBS:Server:Unstable) to disable the test suite there for build target
SLE_12_SP3 - x86_64.

People that branch from that project and want it to be explicitly enabled
would still be able to do that by switching disable_obs_test_suite in their
branched project conf.

Since this can not be done via submit request I am writing this mail;-)

If you have any concerns, please raise them now.
Hmm, I don't really like it. Who makes sure the containers include
the same packages as our Build Service project? If both travis and
OBS run the tests, I have more trust in the OBS run as it has the
same build environment as the rpms that are created.
https://build.opensuse.org/project/show/OBS:Server:Unstable:container:SLE12:SP3

The containers are built here. Sure there will be some small difference
between the container and the actual obs-server package but we're close
(close enough that I would say we can neglect the difference for O:S:U).
Why is this a problem at all? The test suite doesn't run that long,
and I'm pretty sure we could make it faster. And if some tests are
flaky, then it's much better to invest time to fix them.
The test suite usually runs ~25 minutes in Travis, in OBS it takes > 1
hour (sometimes up to 2 hours). In this timeframe a rebuild usually
get's triggered by merging a pull request. The problem is that we do not
get many packages during the day so deployment is only possible in the
morning or late afternoon.

https://travis-ci.org/openSUSE/open-build-service/builds
https://build.opensuse.org/packages/obs-server/job_history/OBS:Server:Unstable/SLE_12_SP3/x86_64

Sure there is always room for improvement and make it faster but I doubt
that we can make it twice as fast (or even more).

In Travis the problem is usually not that the tests are flaky, see
travis build log (the one broken today was introduced by me by a rubocop
issue):
https://travis-ci.org/openSUSE/open-build-service/builds

(And all tests must work on non-SLE_12_SP3 nevertheless, so the
change will not reduce work.)
How do you mean? We test on SLE12 SP3 now in Travis and in the package,
the problem is that we usually only get one package during the day.
I think the point here is that you actually not to ask to *replace* the code
running it inside of the package build, but to opt-out for SLE 12 SP3 repo
build of OBS:Server:Unstable on build.opensuse.org only, right?

The containers on travis are a good addition, but they don't cover running
the tests for all repos and test builds we do, not for our test instance and
not for patched forks like IBS.

Assuming that the containers are always built constant based on the rpms from
OBS:Server:Unstable this opt-out may be okay, but it is no replacement.

Is this a correct view?

Just asking, because I sometimes feel a bit alone fixing the problems in the
package tests or the setup on build-test.o.o, but we would still maintain
these together, right?
Sure we do. And we deploy (almost) every day, so there aren't that many
errors in the package tests (for SLE12 SP3). And as said, in Travis the
tests are succeeding almost always!
But we're moving away from the initial discussion :)

On the long run I would go actually the opposite way, avoid the need
of external servers and seperate QA and build results in OBS...

For the distribution this may make sense. For our development workflow
and for maintaining build.o.o this is not going to work for us. We NEED
to get the test result on the pull request before we even merge to
master. We want to get feedback as early as possible and deploy as often
as possible.


--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-buildservice+owner@xxxxxxxxxxxx

< Previous Next >
This Thread
  • No further messages