[opensuse-releaseteam] New Repo Checker Status
opensuse-releaseteam, In order to finish things up in regards to the repo checker (#964) [1] I will summarize the situation in the hopes of agreeing on a path forward. As noted [2] on the pull request I had frozen the existing osc-check_repo.py on packagelists in preparation for merging with delayed deployment for Factory. The changes conflicted with the AppArmor profile and were rolled back by Dominique. The new repo-checker user on OBS and _opensuse.org-repo-checker along with new services have remained on packagelists. The new repo checker has been running against Leap:42.3 for the last two weeks and has managed to discover a few legitimate problems. One of the notifications generated a discussion that resulted in a correction without interaction by the release team! The remaining areas for discussion have corresponding issues: - #973: repo_checker: speed up reviews by comparing has of staging project psuedometa - #974: repo_checker: invoke mirror, cycle, and install checks in parallel - #978: repo-checker.pl: detect/handle corrupt package snippet and general error handling - #979: repo_checker: cycles discussion - #980: repo_checker: include non-targeted package messages from installcheck The last two are the primary issues of interest. The cycles difference should not be too common and has a set of pros and cons associated with the changes. The primary blocker is the the inclusion of all installcheck/fileconflict problems from letter stagings. This change is again beneficial in detecting problems encountered by more complex packaging, but does not pass for both Leap:42.3 nor Factory. I have sent a number of SRs to resolve some of the issues, but the rest need to be resolved before the new repo checker will accept letter stagings. My suggested path forward would be to continue making the fixes rather than relaxing the check and resume the freezing of old repo checker. Max had mentioned the AppArmor profile was relaxed/disabled which would be good to confirm. I will likely look around on packagelists and make the change again. Once all the issues have been resolved in Factory I suggest we add both repo checker users for review and see that the results make sense before disabling the old repo checker. Given that I encountered the package snippet corruption bug twice more during local testing it definitely occurs. For those not familiar I also found a case in the deployed cache on packagelists. Anytime the corruption is encountered the old repo checker will accept all reviews regardless of issues. As such it makes sense to target switching to the new repo checker as soon as possible. The last minor discussion point is that the ReviewBot currently operates either on a project + request type or reviews targeting a user/group. The current repo checker uses the user/group target, but that would mix reviews for Factory and Leap in one cycle. The ReviewBot could be changed (or extended) to allow filtering by project in addition to review target or all reviews could be run in the same cycle (same service on packagelists). Using the project method would need to be tweaked to allow for multiple request types (as currently it has to be run twice, once for submit and once for delete), but that method currently includes already reviewed requests which adds to clutter the repo checker has to dig through. Adding the project filter to review target method likely makes sense, but perhaps there are some unintended side-effects to other bots? Additionally, it will mean that two repo checkers may run at the same time which locks up 100% cpu across the two cores on packagelists which makes the box a tad unresponsive. Either packagelists needs increased resources, the new repo checker should find a new home, or we accept the situation. Given all the other tools are split out per project I am leaning towards adding the project filter. [1] https://github.com/openSUSE/osc-plugin-factory [2] https://github.com/openSUSE/osc-plugin-factory/pull/964#issuecomment-3114991... I look forward to your thoughts, -- Jimmy -- To unsubscribe, e-mail: opensuse-releaseteam+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-releaseteam+owner@opensuse.org
A recent list of issues (ignore the qt one as the staging just needs to be re- frozen) can be found here: https://gist.github.com/jberry-suse/99329e902ecae6a03a903fa8623c0ce3 Basically just a subset of the installcheck/fileconflict errors for the whole project limited to ring packages. I already submitted the fixes related to clang/llvm (see llvm4 508876 and llvm3_8 508877). As noted by Dominique the *-release, installation-images, and skelcd issues are not real problems, but the packaging should be fixed up to avoid this output. Aside from those, just ruby and openssl conflicts to resolve. Once those 4 problem groups are resolved the new repo checker should be happy accepting Factory letter stagings. Leap:42.3 problems are present as a comment on any non-failing, built letter staging project. -- Jimmy -- To unsubscribe, e-mail: opensuse-releaseteam+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-releaseteam+owner@opensuse.org
Jimmy Berry wrote:
In order to finish things up in regards to the repo checker (#964) [1] I will summarize the situation in the hopes of agreeing on a path forward.
Thanks for the work and summary! It indeed finds issues we didn't see before. Unfortunately the file conflicts it finds are too late to fix for 42.3. We certainly need that bot for all future products. Also, maintenance should use it.
[...] - #973: repo_checker: speed up reviews by comparing has of staging project psuedometa - #974: repo_checker: invoke mirror, cycle, and install checks in parallel - #978: repo-checker.pl: detect/handle corrupt package snippet and general error handling - #979: repo_checker: cycles discussion - #980: repo_checker: include non-targeted package messages from installcheck
The last two are the primary issues of interest. The cycles difference should not be too common and has a set of pros and cons associated with the changes. The primary blocker is the the inclusion of all installcheck/fileconflict problems from letter stagings. This change is again beneficial in detecting problems encountered by more complex packaging, but does not pass for both Leap:42.3 nor Factory. I have sent a number of SRs to resolve some of the issues, but the rest need to be resolved before the new repo checker will accept letter stagings.
My suggested path forward would be to continue making the fixes rather than relaxing the check and resume the freezing of old repo checker. Max had mentioned the AppArmor profile was relaxed/disabled which would be good to confirm. I will likely look around on packagelists and make the change again.
I agree that we need to fix the issues it finds, they are valid after all.
The last minor discussion point is that the ReviewBot currently operates either on a project + request type or reviews targeting a user/group. The current repo checker uses the user/group target, but that would mix reviews for Factory and Leap in one cycle. The ReviewBot could be changed (or extended) to allow
Is there a problem with processing all requests in the same bot instance? If we wanted to use different instances we could also use different users instead. Is the new bot faster then the old one?
filtering by project in addition to review target or all reviews could be run in the same cycle (same service on packagelists).
Using the project method would need to be tweaked to allow for multiple request types (as currently it has to be run twice, once for submit and once for delete), but that method currently includes already reviewed requests which adds to clutter the repo checker has to dig through.
Adding the project filter to review target method likely makes sense, but perhaps there are some unintended side-effects to other bots?
I don't mind adding more complex modes of operation, the existing ones should continue to work though to avoid breaking other bots. cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Jane Smithard, Graham Norton, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-releaseteam+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-releaseteam+owner@opensuse.org
On Wednesday, July 12, 2017 7:24:21 AM CDT Ludwig Nussel wrote:
Jimmy Berry wrote: [...]
The last minor discussion point is that the ReviewBot currently operates either on a project + request type or reviews targeting a user/group. The current repo checker uses the user/group target, but that would mix reviews for Factory and Leap in one cycle. The ReviewBot could be changed (or extended) to allow Is there a problem with processing all requests in the same bot instance? If we wanted to use different instances we could also use different users instead. Is the new bot faster then the old one?
Yes, different users could be used instead, but it is not clear if that is best approach since it may add needless obfuscation for contributors. The new bot is 1/4th the size in code (85% less loops, 118 in main code before!?), runs a significantly lower number of queries, and reduces a lot of duplication so I would expect it to run faster. That said I have not run a formal comparison and the new bot also check multiple archs so it now includes i586. For a comparison I may run with multiple archs disabled, pre-download all packages, and limit to same list of requests.
filtering by project in addition to review target or all reviews could be run in the same cycle (same service on packagelists).
Using the project method would need to be tweaked to allow for multiple request types (as currently it has to be run twice, once for submit and once for delete), but that method currently includes already reviewed requests which adds to clutter the repo checker has to dig through.
Adding the project filter to review target method likely makes sense, but perhaps there are some unintended side-effects to other bots?
I don't mind adding more complex modes of operation, the existing ones should continue to work though to avoid breaking other bots.
Adding a new request query type is probably the way to go.
cu Ludwig
I went ahead and re-froze packagelists old repo checker. Assuming everything works out this time I will be looking to merge the new repo checker so if anyone has time to skim/review now would be the time. -- Jimmy -- To unsubscribe, e-mail: opensuse-releaseteam+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-releaseteam+owner@opensuse.org
On Wednesday, July 12, 2017 9:18:43 AM CDT Jimmy Berry wrote:
On Wednesday, July 12, 2017 7:24:21 AM CDT Ludwig Nussel wrote:
Jimmy Berry wrote: [...]
The last minor discussion point is that the ReviewBot currently operates either on a project + request type or reviews targeting a user/group. The current repo checker uses the user/group target, but that would mix reviews for Factory and Leap in one cycle. The ReviewBot could be changed (or extended) to allow
Is there a problem with processing all requests in the same bot instance? If we wanted to use different instances we could also use different users instead. Is the new bot faster then the old one?
I performed a basic performance analysis (73% faster in basic test): see https://github.com/openSUSE/osc-plugin-factory/pull/964#issuecomment-3149246... for details.
Yes, different users could be used instead, but it is not clear if that is best approach since it may add needless obfuscation for contributors.
The new bot is 1/4th the size in code (85% less loops, 118 in main code before!?), runs a significantly lower number of queries, and reduces a lot of duplication so I would expect it to run faster. That said I have not run a formal comparison and the new bot also check multiple archs so it now includes i586. For a comparison I may run with multiple archs disabled, pre-download all packages, and limit to same list of requests.
filtering by project in addition to review target or all reviews could be run in the same cycle (same service on packagelists).
Using the project method would need to be tweaked to allow for multiple request types (as currently it has to be run twice, once for submit and once for delete), but that method currently includes already reviewed requests which adds to clutter the repo checker has to dig through.
Adding the project filter to review target method likely makes sense, but perhaps there are some unintended side-effects to other bots?
I don't mind adding more complex modes of operation, the existing ones should continue to work though to avoid breaking other bots.
Adding a new request query type is probably the way to go.
cu Ludwig
I went ahead and re-froze packagelists old repo checker. Assuming everything works out this time I will be looking to merge the new repo checker so if anyone has time to skim/review now would be the time.
Merged the pull request and close 6 issues. Now those few file conflicts need to be resolved. -- Jimmy -- To unsubscribe, e-mail: opensuse-releaseteam+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-releaseteam+owner@opensuse.org
Hi Jimmy, Could you point me where is the code in the new repochecker processing multibuild package request? Regards, Max On 07/13/2017 07:22 AM, Jimmy Berry wrote:
On Wednesday, July 12, 2017 9:18:43 AM CDT Jimmy Berry wrote:
On Wednesday, July 12, 2017 7:24:21 AM CDT Ludwig Nussel wrote:
Jimmy Berry wrote: [...]
The last minor discussion point is that the ReviewBot currently operates either on a project + request type or reviews targeting a user/group. The current repo checker uses the user/group target, but that would mix reviews for Factory and Leap in one cycle. The ReviewBot could be changed (or extended) to allow
Is there a problem with processing all requests in the same bot instance? If we wanted to use different instances we could also use different users instead. Is the new bot faster then the old one?
I performed a basic performance analysis (73% faster in basic test): see https://github.com/openSUSE/osc-plugin-factory/pull/964#issuecomment-3149246... for details.
Yes, different users could be used instead, but it is not clear if that is best approach since it may add needless obfuscation for contributors.
The new bot is 1/4th the size in code (85% less loops, 118 in main code before!?), runs a significantly lower number of queries, and reduces a lot of duplication so I would expect it to run faster. That said I have not run a formal comparison and the new bot also check multiple archs so it now includes i586. For a comparison I may run with multiple archs disabled, pre-download all packages, and limit to same list of requests.
filtering by project in addition to review target or all reviews could be run in the same cycle (same service on packagelists).
Using the project method would need to be tweaked to allow for multiple request types (as currently it has to be run twice, once for submit and once for delete), but that method currently includes already reviewed requests which adds to clutter the repo checker has to dig through.
Adding the project filter to review target method likely makes sense, but perhaps there are some unintended side-effects to other bots?
I don't mind adding more complex modes of operation, the existing ones should continue to work though to avoid breaking other bots.
Adding a new request query type is probably the way to go.
cu Ludwig
I went ahead and re-froze packagelists old repo checker. Assuming everything works out this time I will be looking to merge the new repo checker so if anyone has time to skim/review now would be the time.
Merged the pull request and close 6 issues.
Now those few file conflicts need to be resolved.
-- To unsubscribe, e-mail: opensuse-releaseteam+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-releaseteam+owner@opensuse.org
On 07/13/2017 04:56 PM, Max Lin wrote:
Hi Jimmy,
Could you point me where is the code in the new repochecker processing multibuild package request?
Nevermind, I got it already. It's synced the binaries by bs_mirrorfull and does not check multibuild result as we can see that on dashboard. No need multibuild api call involved here indeed. Thanks, Max -- To unsubscribe, e-mail: opensuse-releaseteam+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-releaseteam+owner@opensuse.org
participants (3)
-
Jimmy Berry -
Ludwig Nussel -
Max Lin