On Tuesday, 11 May 2021 15.13.56 CEST Joe Salmeri wrote:
Thanks Oliver, I appreciate it.
I am most interested in how Fabian was able to determine that "Pretty much all of those softfails are because the system doesn't go fully idle in the time openQA expects"
I'm looking for a way to know when the softfails can safely be ignored.
Is there some place that summarizes the process which makes it easy to come to the conclusion that Fabian did?
I don't know for sure. openSUSE release managers would need to tell how they do that exactly right now. Likely the looked at the output of "totest-manager" from https://github.com/openSUSE/opensuse-release-tools to find out which tests referenced which bug. totest-manager looks at openQA test results equivalent to a view https://openqa.opensuse.org/tests/overview? result=softfailed&result=failed&result=incomplete&result=timeout_exceeded&arch=&machine=&modules=&distri=opensuse&groupid=1&version=Tumbleweed# and releases the product, i.e. a new Tumbleweed build, as soon as there are no more unreviewed, unignorable test failures left. A not-ok test is reviewed if there is a comment in openQA with a ticket reference. And a reviewed test will be seen as "ignorable" for totest-manager (ttm) when the release managers inform ttm by an according openQA comment with "@ttm ignore" next to the ticket to which this ignore-comment should be linked. Have fun, Oliver