Hello list, as outlined in the mail sent by coolo last Thursday, the current approach of having openQA only run after Factory emits an ISO image does not prevent Factory from getting broken. Problems can only be detected when it's basically already too late. Therefore coolo's mail proposed this workflow that strengthens openQA's importance by including pre-integration tests in the Factory submission process. https://progress.opensuse.org/workflow/factory-proposal.html That mode of operation adds new requirements on openQA. In no particular order of importance: 1. Accessible In order to allow Factory and staging project maintainers to evaluate and fix test runs openQA's web interface needs to be enhanced to work with multiple users. That means introducing an actual web framework instead of using CGI.pm, introducing a proper database backend, coming up with a permissions model, etc. Also, there needs to be a way to allow users to edit test cases. Right now one has to do that outside of the tool via direct git access. Fulfilling this requirement is basically precondition for hosting openQA in public, ie needed for full upstream acceptance. 2. Reliable In it's current state openQA needs constant care taking by an operator with root access. Way too often jobs get stuck or fail in an unpredictable way. Many times the reasons for that are deadlocks in the threading model, issues with the monitor interface of QEMU, race conditions that depend on the load and disk IO or side effects of tests. Effort needs to be put into eliminating random failures by e.g. avoiding Perl threads, enhancing QEMU's QMP interface, avoiding sleep() or leveraging VM snapshots in a smarter way. 3. Scalable openQA already is able to run several QEMU instances on one machine. There are natural limits on the amount of RAM and CPU of affordable machines though. So even the fastest machine seen so far can only run eight installations in parallel. To be able to run both Factory tests as well as pre-integration tests projects in a reasonable amount of time openQA needs to be parallelized to run on several machines. The way OBS uses workers can serve as inspiration here. This feature requires implementing smart ways to distribute input files like test scripts, reference images, RPMs or ISO images among workers. It requires a database model that allows live updating of test results and a new way to provide the back channel for the live view. 4. Debuggable Sometimes tests just fail in a way that is not debuggable after the fact. Therefore there needs to be a mode that allows developers to interactively take control of the virtual machine that runs the test. If that doesn't help either, there should be a way to clone jobs to a local instance to reproduce it there. 5. Interfacable So far openQA test results are meant to be interpreted by humans. To be able to serve as good input for automatically judging whether a staging project is good enough, the results must be available in a way for the machine to understand. E.g. for displaying test results directly in OBS projects. To implement the listed features, several man months of concentrated development effort would be needed. p.s The original author of this email is Ludwig Nussel, but I edited it a bit before sending. That means that the good parts are from him, and the mistakes are mine. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org