openSUSE Release Engineering meeting 09.02.2022
All meeting minutes can be found here: https://etherpad.opensuse.org/p/ReleaseEngineering-meeting Meeting is hosted here https://meet.opensuse.org/ReleaseEngineeringMeeting ## Attendees bittin, Sarah, DimStar, Lubos, DocB, Richard, Douglas, Marina, Adrian, Dirk, guillaumeg, Maxlin, Gerald, Maurizio ## Leap * Leap successful build blocked on rpmlint and polkit-default-privs acceptance in SLE - discussion in progress Related discussion with autobuild about making our builds faster. So far the only suggestion/quick solution was to increase priority of job- picking by OBS workers. Bugzilla policy Leap related update prior Beta https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/6... and related QA test manual plan refresh So far the feedback was positive (Increased utilization of public SLE bugs etc). Proposal to drop voting from the policy which doesn't seem to be used much. Overlap in between SLE and Leap (forking WE packages) https://github.com/openSUSE/openSUSE-release-process/issues/71 Talk with maintenance team tomorrow at 3:30 - 4:00pm CEST (does anybody else wants to join?) Leap Micro 5.2 - No update yet, aside from setup being done. https://build.opensuse.org/project/show/openSUSE:Leap:Micro:5.2 Few build failures I'd like to focus in rest of the week on solving them. Then we can start working on the code-o-o Cardano node plan. SLE Micro Beta 1 is expected this week, perhaps a good time to start focusing on it anyway. Interlock is not yet signed (project signoff). Cardano node do we have some existing crypto community that? https://code.opensuse.org/leap/features/issue/59 Discussion with Alex Osthof regarding volunteer interested in Cloud. We came up with idea that he could start a Public Cloud testsuite for Leap, there seems to a blank space where we could benefit from additional testing. Dirk: wouldn't that be redundant after CtLG? Pehaps then test it only in Leap Gerald: What about some targetted smoke test, to ensure it works, branding,...? Dirk agrees. Doug: https://github.com/cisco/openh264/issues/3480 Just needed clarification on the PR as needing to be hosted by cisco. Looks like 3rd party is what is needed, correct? It will be streamlined in 15.4, just not sure if it will be a cisco repository or another, correct? ## openSUSE Tumbleweed * Kernel 5.16.4 in Tumbleweed, 5.16.5 will be in 0207 (QA running), 5.16.7 in staging (with simple drm enabled) * Python 3.6 being EOL, removal of python36-FOO modules ongoing. Last week: 96 left, now 80 * systemd dropped suse specific sysv script handling as announced in https://lists.opensuse.org/archives/list/factory@lists.opensuse.org/thread/3... (snapshot 0206 contained this) * RPM: the buildroot switched from 'glibc-locale' to the much leaner glibc-locale-base (change happenedon 0205) * meson 0.61 - syntax again getting stricted, not silently ignoring errors * Plasma 5.24 will be in snapshot 0207+ (snapshot built, currently in QA) * glibc 2.35 is being staged * GCC 12 pre-integration tests started (Staging:Gcc7); Scheduled for actual merge in April/May DimStar: Big thanks to Dirk regarding firefox submission! ## Richard (Kubic/TW MicroOS) Working on MicroOS images for WSL, using a unique approach of bundling the image inside the TW WSL image, which ends up acting like our usual patched bootloader for MicroOS. ongoing experiments with with kiwi-less, container-first VM image building/adding nerdctl support to Toolbox/buildkit and its limits as a container builder/TW-to-MicroOS converter ## Max llvm13 will be the new default in Leap https://code.opensuse.org/leap/features/issue/55 Just go back from vacation ## Guillaume - Arm Tumbleweed: * No blocker * Upcoming GCC 12 tested for aarch64: https://build.opensuse.org/project/show/home:dirkmueller:branches:openSUSE:F... * Tests of openSUSE Tumbleweed on The Pinebook Pro showed some issues. Most of them are now fixed. Tracked on https://bugzilla.opensuse.org/show_bug.cgi?id=1194491 Leap 15.3: * Issue with installation media (net installation) for devices w/o Real-Time-Clock. Workaround: Trigger rebuild of installation media. https://bugzilla.opensuse.org/show_bug.cgi?id=1194660 lkocman: to update the bug with installmedia refresh. lkocman: Slipped from my radar. Will look at it this week. Leap 15.4: * opencl-headers updated and armnn:opencl is built properly now. * aarch64: * Go compiler from SLE still lacks ld.gold which is still required. This make some packages to fail to build (e.g. openSUSE:Backports:SLE-15-SP4/rclone). Bug opened since Leap 15.3: https://bugzilla.opensuse.org/show_bug.cgi?id=1183043 * armv7: * armv7 images are not tested in openQA yet, because JeOS-efi image does not build properly due to an armv7 kernel bug with btrfs and lzo compression: https://bugzilla.opensuse.org/show_bug.cgi?id=1193852 => kernel overlay with a possible fix added. Need to wait for a new build to check if the fix is ok. ## Sarah - s390x Tumbleweed: * Reboot does not work: https://bugzilla.opensuse.org/show_bug.cgi?id=1193889 (forwarded to IBM for all Linux distributions on s390x) Bugfix from IBM is breaking SLES, Leap and Tumbleweed builds: https://bugzilla.suse.com/show_bug.cgi?id=1188896 Rudi has taken a dump yesterday. The question is how to forward (suggestion SUSE FTP) * lkocman is proactively killing/rescheduling "stuck" Leap builds (so that part is covered) to my knowledge Ihno monitors IBS builds and asks autobuild team to "Free them" ## Doug * openSUSE Conference * keynotes (two possible speakers) * Will meet on Friday with Jitsi founder Emil Ivov about giving keynote * TensorFlow/OSS DevInfra's Mihai Maruseac working on approval from Google Brain to provide keynote * Venueless sent offer (already covered by open PO) * Google Summer of Code * Application 50% complete * PR for projects are coming in. Current listing * Feb. 7 to 21 (deadline) * Leap 15.4 * Progressing with release annoucement * Sponsorship * Open Source Community Africa ## Dirk Reduced rebuild time for firefox packages from hours to ~ 45 minutes (should help with stagings) Implemented more zstd-gzip compatibility, can now build almost all of :Rings with zstd-gzip Fixed /bin/sh == dash breakages. Can build ~ 70% of Tumbleweed. Most problems remain are the kubernetes and python packages. Resurrected nanoOS idea for virtual machine usecases some backports updating (openstack clients untangling) Helped a bit on python 3.10 fallout in tumbleweed ## Gerald / Board [All good] ## Wolfgang (Package Hub), Scott Bahling Not available Need a confirmation from Wolfgang/Scott whether proposal in https://github.com/openSUSE/openSUSE-release-process/issues/71 will fix the issue. ## Maintenance team (Marina or Marcus, Maurizio (m4u)) rpmlint - fixing the plasma5-disks Marina: It's currently in mainteance queue. For any extra details feel free to contact Fabian or Antonio who can provide more information if needed. Dirk: no QE needed, it's just adding polkit policy. - Marcus: 15.3 had a radeon regression in the kernel update, followup update soon - we released big samba update to fix security issues, one regression in samba spotted - Marcus: 15.4 is set up channel wise, some updates will go in testing occasionaly, if you need more ping me, no openqa setup yet ## Adrian - OBS Rudi is fighting with host kernel issues in IBS. There are quite some issues with the SP4 kernel. x86_64 - seems like the issue not caused by SP4/numa. Discussion about the Leap build pace. Seems like switching the rebuild strategy would only result into more issues. Adrian: could be caused by numa. Let's focus on this once we'll have stable workers again.
Hi Lubos, Am Mittwoch, 9. Februar 2022, 11:09:00 CET schrieb Lubos Kocman: ....
## openSUSE Tumbleweed
* Kernel 5.16.4 in Tumbleweed, 5.16.5 will be in 0207 (QA running), 5.16.7 in staging (with simple drm enabled)
While there was a discussion about performance improvments in OBS in todays RelEng-Meeting, I seem to remember that we talked about this for openQA some time ago as well. Looking at the current 20220207 snapshot, the only pending QA jobs are for flavours that are out of maintenance since...years partly: https://paste.opensuse.org/31618724 In order speed testing up I would like to start the discussion about outdated test cases...where should this be discussed? Cheers Axel
On 2/9/22 21:39, Axel Braun wrote:
Hi Lubos,
Am Mittwoch, 9. Februar 2022, 11:09:00 CET schrieb Lubos Kocman: ....
## openSUSE Tumbleweed
* Kernel 5.16.4 in Tumbleweed, 5.16.5 will be in 0207 (QA running), 5.16.7 in staging (with simple drm enabled)
While there was a discussion about performance improvments in OBS in todays RelEng-Meeting, I seem to remember that we talked about this for openQA some time ago as well.
Looking at the current 20220207 snapshot, the only pending QA jobs are for flavours that are out of maintenance since...years partly: https://paste.opensuse.org/31618724
In order speed testing up I would like to start the discussion about outdated test cases...where should this be discussed?
Unfortunately these probably aren't out of date yet traditionally for SLE atleast upgrading by skipping a major version has been supported. Which means in practice its likely that SUSE will want to support upgrades from SLE-12-SP5 (or whatever we get to) to SLE-16 (or whatever its called). In practice at the moment upgrading a system from 42.X to Tumbleweed is the closest approximation we can do in openQA for that use case. So I guess a counter question is, what is the need for speeding up testing? are we at the point where sometimes openQA tests run for greater then 24hrs so new snapshots are released invalidating the testing on the previous one, or are we running low on resources for other tests to run? -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
Am 2022-02-09 um 13:02 schrieb Simon Lees:
On 2/9/22 21:39, Axel Braun wrote:
Hi Lubos,
Am Mittwoch, 9. Februar 2022, 11:09:00 CET schrieb Lubos Kocman: ....
## openSUSE Tumbleweed
* Kernel 5.16.4 in Tumbleweed, 5.16.5 will be in 0207 (QA running), 5.16.7 in staging (with simple drm enabled)
While there was a discussion about performance improvments in OBS in todays RelEng-Meeting, I seem to remember that we talked about this for openQA some time ago as well.
Looking at the current 20220207 snapshot, the only pending QA jobs are for flavours that are out of maintenance since...years partly: https://paste.opensuse.org/31618724
In order speed testing up I would like to start the discussion about outdated test cases...where should this be discussed?
Unfortunately these probably aren't out of date yet traditionally for SLE atleast upgrading by skipping a major version has been supported. Which means in practice its likely that SUSE will want to support upgrades from SLE-12-SP5 (or whatever we get to) to SLE-16 (or whatever its called).
In practice at the moment upgrading a system from 42.X to Tumbleweed is the closest approximation we can do in openQA for that use case.
There are multiple benefits for these upgrade jobs, among others 1. Simply show that it actually still works to upgrade from old Leap to Tumbleweed :) Maybe some users can still use that as a reference to upgrade their completely outdated and unsupported older openSUSE to a current openSUSE Tumbleweed ;) 2. As a "stress-test" doing a lot of package replacement and shuffling around of files
So I guess a counter question is, what is the need for speeding up testing? are we at the point where sometimes openQA tests run for greater then 24hrs so new snapshots are released invalidating the testing on the previous one, or are we running low on resources for other tests to run?
At least for x86_64 and aarch64 I don't think we commonly have that problem on openqa.opensuse.org. Also https://metrics.opensuse.org/d/osrt_openqa/osrt-openqa?orgId=1&from=now-30d&to=now shows that the queue of pending jobs often enough reaches zero meaning there is enough capacity. We even use the free capacity to run automatic investigation jobs and provide the instance for developing tests by individual developers so again, there is enough capacity :)
Hello Axel, since I went through a lot of them today/yesterday again, let me add my two cents. I did file request of removal these old migration suites about a year ago :-) Yet there are colleagues who don't mind ensuring these tests can be still executed as if they're running they provide value :-) It's a double edged sword. However, long as there are people willing to maintain these test suites, there is a reason to keep them. I recall 15.2 retro where people were happy about migration from 42.X. Cheers Lubos Axel Braun píše v St 09. 02. 2022 v 12:09 +0100:
Hi Lubos,
Am Mittwoch, 9. Februar 2022, 11:09:00 CET schrieb Lubos Kocman: ....
## openSUSE Tumbleweed
* Kernel 5.16.4 in Tumbleweed, 5.16.5 will be in 0207 (QA running), 5.16.7 in staging (with simple drm enabled)
While there was a discussion about performance improvments in OBS in todays RelEng-Meeting, I seem to remember that we talked about this for openQA some time ago as well.
Looking at the current 20220207 snapshot, the only pending QA jobs are for flavours that are out of maintenance since...years partly: https://paste.opensuse.org/31618724
In order speed testing up I would like to start the discussion about outdated test cases...where should this be discussed?
Cheers Axel
-- Best regards Lubos Kocman openSUSE Leap Release Manager
24.02.2022 18:03:29 Lubos Kocman <lubos.kocman@suse.com>:
Hello Axel,
since I went through a lot of them today/yesterday again, let me add my two cents.
I did file request of removal these old migration suites about a year ago :-)
Yet there are colleagues who don't mind ensuring these tests can be still executed as if they're running they provide value :-)
It's a double edged sword. However, long as there are people willing to maintain these test suites, there is a reason to keep them.
I recall 15.2 retro where people were happy about migration from 42.X.
Just yesterday I found a machine still running openSUSE Leap 42.3 . The upgrade to Leap 15.3 was actually flawless. So maybe the openQA tests for that actually helped :D
Axel Braun píše v St 09. 02. 2022 v 12:09 +0100:
Hi Lubos,
Am Mittwoch, 9. Februar 2022, 11:09:00 CET schrieb Lubos Kocman: ....
## openSUSE Tumbleweed
* Kernel 5.16.4 in Tumbleweed, 5.16.5 will be in 0207 (QA running), 5.16.7 in staging (with simple drm enabled)
While there was a discussion about performance improvments in OBS in todays RelEng-Meeting, I seem to remember that we talked about this for openQA some time ago as well.
Looking at the current 20220207 snapshot, the only pending QA jobs are for flavours that are out of maintenance since...years partly: https://paste.opensuse.org/31618724
In order speed testing up I would like to start the discussion about outdated test cases...where should this be discussed?
Cheers Axel
-- Best regards
Lubos Kocman openSUSE Leap Release Manager
participants (4)
-
Axel Braun
-
Lubos Kocman
-
Oliver Kurz
-
Simon Lees