Richard Brown wrote:
On Wed, 2013-12-04 at 11:57 +0100, Alberto Planas Dominguez wrote:
3. Scalable openQA already is able to run several QEMU instances on one machine. There are natural limits on the amount of RAM and CPU of affordable machines though. So even the fastest machine seen so far can only run eight installations in parallel. To be able to run both Factory tests as well as pre-integration tests projects in a reasonable amount of time openQA needs to be parallelized to run on several machines. The way OBS uses workers can serve as inspiration here. This feature requires implementing smart ways to distribute input files like test scripts, reference images, RPMs or ISO images among workers. It requires a database model that allows live updating of test results and a new way to provide the back channel for the live view.
If you go this route, will it be possible for contributors to donate/add 'testers' (my name for the openQA equivalent of OBS 'builders') to the pool? We can't offer that for OBS for lots of very sensible security reasons, but if it is an option for openQA we'd hopefully have contributors willing to volunteer hardware, bandwidth, etc for testing Factory, which could really help with that scalability (and at least mean the responsibility and cost for all hardware isn't solely SUSEs)
That idea has been floating around at least. Even if the architecture allows it in theory there may be very practical limitations to solve too, like e.g. the huge bandwidth requirements for syncing new builds. Delta-ISOs might become helpful there. This needs to be evaluated.
4. Debuggable Sometimes tests just fail in a way that is not debuggable after the fact. Therefore there needs to be a mode that allows developers to interactively take control of the virtual machine that runs the test. If that doesn't help either, there should be a way to clone jobs to a local instance to reproduce it there.
I've had some thoughts about that. I'm not convinced you need to make that 'openQA's problem'. If we have an issue which openQA is detecting but unable to debug, I think all openQA 'needs' to do is flag the issue and provide all the information it can - this then becomes an issue for a real human tester to test
In my opinion, the best way openQA can help there perhaps some kind of dashboard to show open/undebugged test cases
But I certainly agree that having openQA offer downloadable VM/disk images so the tester can download and run those tests locally, yeah, that would be really nice, and dramatically cut down the work required for those cases which cant be debugged after the fact.
Cutting down the work ie time required to debug failures is exactly what this is about. The less effort required to debug things the more likely people will actually do it.
5. Interfacable So far openQA test results are meant to be interpreted by humans. To be able to serve as good input for automatically judging whether a staging project is good enough, the results must be available in a way for the machine to understand. E.g. for displaying test results directly in OBS projects.
Without knowing the current system, again, hard for me to comment, but perhaps the answer is to flip the problem on its head? have the initial results be machine-parsable, and then have a parser transform those results to generate the human readable ones?
Obviously yes. What I meant was to offer not only an html view but also e.g. a json rpc interface.
To implement the listed features, several man months of concentrated development effort would be needed.
Great, what opportunities will there be for non-openSUSE Team contributors to help out? I assume you'll be accepting pull requests to https://github.com/openSUSE-Team/openQA ?
Yes. That one and os-autoinst in the same directory. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org