as outlined in the mail sent by coolo last Thursday, the current
approach of having openQA only run after Factory emits an ISO image
does not prevent Factory from getting broken. Problems can only be
detected when it's basically already too late. Therefore coolo's mail
proposed this workflow that strengthens openQA's importance by
including pre-integration tests in the Factory submission process.
That mode of operation adds new requirements on openQA. In no
particular order of importance:
In order to allow Factory and staging project maintainers to
evaluate and fix test runs openQA's web interface needs to be
enhanced to work with multiple users. That means introducing an
actual web framework instead of using CGI.pm, introducing a proper
database backend, coming up with a permissions model, etc. Also,
there needs to be a way to allow users to edit test cases. Right
now one has to do that outside of the tool via direct git access.
Fulfilling this requirement is basically precondition for hosting
openQA in public, ie needed for full upstream acceptance.
In it's current state openQA needs constant care taking by an
operator with root access. Way too often jobs get stuck or fail in
an unpredictable way. Many times the reasons for that are deadlocks
in the threading model, issues with the monitor interface of QEMU,
race conditions that depend on the load and disk IO or side effects
of tests. Effort needs to be put into eliminating random failures
by e.g. avoiding Perl threads, enhancing QEMU's QMP interface,
avoiding sleep() or leveraging VM snapshots in a smarter way.
openQA already is able to run several QEMU instances on one
machine. There are natural limits on the amount of RAM and CPU of
affordable machines though. So even the fastest machine seen so far
can only run eight installations in parallel. To be able to run
both Factory tests as well as pre-integration tests projects in a
reasonable amount of time openQA needs to be parallelized to run on
several machines. The way OBS uses workers can serve as inspiration
here. This feature requires implementing smart ways to distribute
input files like test scripts, reference images, RPMs or ISO images
among workers. It requires a database model that allows live
updating of test results and a new way to provide the back channel
for the live view.
Sometimes tests just fail in a way that is not debuggable after the
fact. Therefore there needs to be a mode that allows developers to
interactively take control of the virtual machine that runs the
test. If that doesn't help either, there should be a way to clone
jobs to a local instance to reproduce it there.
So far openQA test results are meant to be interpreted by humans.
To be able to serve as good input for automatically judging whether
a staging project is good enough, the results must be available in
a way for the machine to understand. E.g. for displaying test
results directly in OBS projects.
To implement the listed features, several man months of concentrated
development effort would be needed.
p.s The original author of this email is Ludwig Nussel, but I edited
it a bit before sending. That means that the good parts are from him,
and the mistakes are mine.
To unsubscribe, e-mail: opensuse-factory+unsubscribe(a)opensuse.org
To contact the owner, e-mail: opensuse-factory+owner(a)opensuse.org