On Tue, 2016-11-01 at 16:04 +0000, James Mason wrote:
On Tue, 2016-11-01 at 16:56 +0100, Dominique Leuenberger wrote:
Please check the known defects of this snapshot before upgrading: https://openqa.opensuse.org/tests/overview?distri=opensuse&groupid= 1&; version=Tumbleweed&build=20161031
Hi Dim*,
I understand the "soft failure" vs. "failure" nature of OpenQA, but can you (or can you ask someone to ;-) explain the policy on allowing Tumbleweed builds with failed tests?
I'm just looking for some clarity on why we allow some failures, and how decisions are made to ship with known failures.
Hi James, Let me try to explain, with pointing to some sample test from the last run: * Soft fails are never a reason for a blocked snapshot * 'unit fails' inside a test, unless it is marked important, results in the test suite to 'succeed' (status passed) https://openqa.opensuse.org/tests/295125#step/multi_users_dm/11 => multi_users_dm test is 'failed' - overall test suite is 'softfailed' (multi_users_dm is not marked important) Taking only this into account, snapshot 1031 passed with: Passed: 62 / Soft Failre: 33 / Failed: 11 As explained above, Soft Failures are considered 'passed' - but need some investigation (some are long standing bugs in the tests, quite a few just 'random' failures which are being worked out) Which leaves us with the more interesting case of the '11 failed' (after all, the snapshot had been released) For this we maintain a list of 'known failures' - so tests that are currently broken, due to various issues. If you check the overview page for the snapshot you will see that we try to tag known failures with bug references - making it much easier to track them. Finally, it is my responsibility to investigate on new 'failed' and decide if this is still sensible to let the snapshot pass or if it's required to block it now. The basic rule there is: unless the reason is understood, I'm not adding a failure to the list of known_failures. The overview of current failures can be found at: https://openqa.opensuse.org/tests/overview?distri=opensuse&version=Tumb leweed&build=20161031&groupid=1 Simply filter on 'failed' tests and see the 11 failed; 9 pf them have a bug reference (or progress item) assigned. 2 of the tests (42.1 Updates) 'changed' the reason they failed between the last run and this run and those are the ones without bug assigned, and the thing i am currently looking in to. Does that help? Cheers, Dominique