[opensuse-factory] OpenQA tests too much out of date?
Hi all, I appreciate the work done for testing TW and have to say that it rarely breaks, but if I look at the snapshot announcements, I can only conclude that the OpenQA report is not that useful. E.g. last report: Passed: 54 Incomplete: 3 Soft Failure: 40 Failed: 6 -> Slightly more than half of the tests passed and still this is ok? If you start looking, that might be correct, because for some failed needles it is just the font that has changed or other minor cosmetic changes. However, this makes it hard to find real failures even with the softfailed/failed differentiation. To have a better feedback and useful tests, up-to-date needles are really required in my opinion. How can we get there? Kind regards, Erwin -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Sun, 2016-09-04 at 18:26 +0200, Erwin Van de Velde wrote:
Hi all,
I appreciate the work done for testing TW and have to say that it rarely breaks, but if I look at the snapshot announcements, I can only conclude that the OpenQA report is not that useful.
E.g. last report: Passed: 54 Incomplete: 3 Soft Failure: 40 Failed: 6 -> Slightly more than half of the tests passed and still this is ok? If you start looking, that might be correct, because for some failed needles it is just the font that has changed or other minor cosmetic changes. However, this makes it hard to find real failures even with the softfailed/failed differentiation.
To have a better feedback and useful tests, up-to-date needles are really required in my opinion. How can we get there?
Kind regards, Erwin
Softfailed could as well be called 'limited pass' - it's minor application tests not passing for different reasons (needles is a cat and mouse game, that are being updated). But even with all needles updated, currently there are two apps causing most of those softfails: * chrome -> new versions introduce keyring integration. The test needs to be adapted to this * vlc on KDE: there is a keyboard shortcut double-occupied (alt-p); most users probably won't realize it as they click there with the mouse; we could change the test to do the same, or keep on working with upstream to solve the underlying issue Any single app in a complete test run causing a fail like this brings the overall test to 'softfail' (or 'limited pass' as a softfailed test is not considered for blocking releases of snapshots). so in short, yes, we do try to minimize the softfail count as well, but without actually looking at the reasons of their occurence, it's not always black and white to claim what is wrong with them. As to what can be done about it: work work work Cheers, Dominique
Am Montag, 5. September 2016, 10:35:34 schrieb Dominique Leuenberger / DimStar:
* vlc on KDE: there is a keyboard shortcut double-occupied (alt-p); most users probably won't realize it as they click there with the mouse; we could change the test to do the same, or keep on working with upstream to solve the underlying issue
This patch should fix the problem: https://build.opensuse.org/package/view_file/home:wolfi323:branches:multimed... Shall I submit it? Though I'm not completely sure that the chosen shortcuts are ideal. And I yet have to adjust the translations too of course, otherwise the button and popup menu texts are untranslated. Or is there an upstream fix in sight already anyway? Kind Regards, Wolfgang -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Wed, 2016-09-14 at 22:54 +0200, Wolfgang Bauer wrote:
Am Montag, 5. September 2016, 10:35:34 schrieb Dominique Leuenberger / DimStar:
* vlc on KDE: there is a keyboard shortcut double-occupied (alt-p); most users probably won't realize it as they click there with the mouse; we could change the test to do the same, or keep on working with upstream to solve the underlying issue
This patch should fix the problem: https://build.opensuse.org/package/view_file/home:wolfi323:branches:m ultimedia:libs/vlc/vlc-2.2.4-unambig-shortcuts.patch?expand=1
I you can try to submit it to upstream - but there it would be interesting for the master branch only. for openQA, this has since been 'worked around' in a way a user would do it: it clicks play using the mouse.
Shall I submit it?
Though I'm not completely sure that the chosen shortcuts are ideal. And I yet have to adjust the translations too of course, otherwise the button and popup menu texts are untranslated.
The fact that we break translations is a big no-go for such a patch. It's not worth it.
Or is there an upstream fix in sight already anyway?
No - not unless somebody really works it all out - which means also including the translations and ensuring no duplicates would happen there. but, as said: for openQA this has been 'solved' by using the mouse (same a user will do on his system) Cheers, Dominique
Am Mittwoch, 14. September 2016, 22:59:14 schrieb Dominique Leuenberger / DimStar:
I you can try to submit it to upstream - but there it would be interesting for the master branch only.
I'll have a look. I never worked on VLC upstream yet though...
for openQA, this has since been 'worked around' in a way a user would do it: it clicks play using the mouse.
Ok, fine. So there's no apparent need for that patch in openSUSE any more.
The fact that we break translations is a big no-go for such a patch. It's not worth it.
Of course. I would have updated the patch to adjust the translations too, but I wanted to ask if it's still necessary first before I do *that* work. ;-) Kind Regards, Wolfgang -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (3)
-
Dominique Leuenberger / DimStar
-
Erwin Van de Velde
-
Wolfgang Bauer