On Tue, 2021-05-04 at 15:22 -0400, Neal Gompa wrote:

On Tue, May 4, 2021 at 1:31 PM Sarah Julia Kriesch < ada.lovelace@gmx.de> wrote:

...

Hello together,

we think about all the requirements for openSUSE Kubic for IBM Z at the moment. I have identified the issue, that there are missing container images for s390x in public container registries (I don't want to speak/ am not allowed to speak about the reasons). That does not make sense to provide a Kubernetes platform without container images in the s390x case.

Neal and I are interested to introduce multi-arch images built with OBS for registry.opensuse.org for all required applications based on openSUSE Leap. We have got interest for aarch64 and s390x at the moment. That can be interesting for all the other architectures, too. Therefore, multi- arch images would match all these requirements.

Finally, you should be able to use our container images on all Kubernetes platforms. What is your opinion about this idea?

It looks like the Leap base containers are already built for aarch64 and s390x, including my DNF-based ones. So now it's largely about folks wanting to build layered images on Leap and then building those for alternative architectures.

I would prefer to see more people building more Tumbleweed containers. There we also have support for every architecture, and we have an ecosystem of dozens of existing containers already providing a large amuont of services including: - Kubernetes* - bind* - ceph - dhcp-server* - haproxy* - httpd - nfs-server - nginx* - openldap - postfix - spamassassin* And more, see https://registry.opensuse.org I believe all Tumbleweed-based official openSUSE containers already build for multiple architectures and the ones marked above with a * are already building for s390x. This is compared to Leap where we have 0 official containers derived from the (multi-arch) Leap base container. Personally I see no reason to start from scratch there when we have a working pile of Tumbleweed containers people can already add to. Regards, -- Richard Brown Linux Distribution Engineer - Future Technology Team    Phone +4991174053-361 SUSE Software Solutions Germany GmbH,  Maxfeldstr. 5,  D-90409 Nuernberg (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer

On Wed, May 05, Jimmy Berry wrote:

On Wednesday, May 5, 2021 2:07:23 AM CDT Richard Brown wrote:

...

I would prefer to see more people building more Tumbleweed containers.

Meanwhile we are how many months into "is executable" checking not working in a TW container even on a Leap host? Realities like this are the reason that isn't practical unless we breakages to TW containers are taken seriously and blocked.

I tried to find out why your bug report that TW containers are not working on Leap didn't got fixed. Surprisingly, I couldn't find a single bug report about this for Leap or SLE... Don't complain that things don't get fixed if you don't report the bugs against this products. I found exactly one bug report for older code streams that glibc 2.33 based containers are not working, and that was created by me over two weeks ago. 22 hours later a first fix was submitted to maintainence. 10 days later a maintenance update was released for docker/runc for SLE, so it should be available for Leap, too. So don't complain if you do not even take the time to report it as bug! Now I only need to find out what's the status with podman... Thorsten -- Thorsten Kukuk, Distinguished Engineer, Senior Architect SLES & MicroOS SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany Managing Director: Felix Imendoerffer (HRB 36809, AG Nürnberg)

On Wednesday, May 5, 2021 10:58:25 AM CDT Fabian Vogt wrote:

...

I tried to find out why your bug report that TW containers are not working on Leap didn't got fixed. Surprisingly, I couldn't find a single bug report about this for Leap or SLE...

It was reported to the maintainer on the original bug report already: https://bugzilla.opensuse.org/show_bug.cgi?id=1182451#c32 It was stuck in NEEDINFO for a month until I sent some reminders.

Thanks, this what I was referencing. I updated my Leap host and tried it again today just to make sure it still wasn't fixed. I've already had to move all my TW based CI workfloads to Leap containers. -- Jimmy

On Wednesday, May 5, 2021 1:11:58 PM CDT Thorsten Kukuk wrote:

That's TW, neither Leap nor SLE, so maintenance and others are not getting informed and nobody has on the radar that other products are affected, too.

If there is a bug on a product, report that bug for that product, so that everybody can track it correct, and don't hope that by accident the right people stumble about such bug reports.

At this point it is a matter of semantics. - Is the bug against TW since it updated a package such that the container would be incompatible with other hosts such as Leap and thus break the fundamental feature of a TW container. Or - Is the bug against Leap since it has not made updates to be compatible with the TW container? If the first the bug was filed correctly. If the second either a) circle back to my comment about blocking the update, or b) at the very least the maintainer who knowingly made the incompatible change could file a maintenance bug. The product against which the bug should be filed is not quite straight forward. At this point since the TW change is not going to be rolled back a Leap bug certainly makes sense, but originally a rollback of TW package was the most responsible. If I need to file a separate bug let me know, just hoping this is fixed and per my other email where TW stands on breaking container changes being blocked. -- Jimmy

On Wednesday, 5 May 2021 22.00.06 CEST Thorsten Kukuk wrote:

On Wed, May 05, Jimmy Berry wrote:

...

At this point it is a matter of semantics.

No, there is one clear bug in the container runtime.

...

- Is the bug against TW since it updated a package such that the container would be incompatible with other hosts such as Leap and thus break the fundamental feature of a TW container.

No, the change in TW was correct.

...

Or

- Is the bug against Leap since it has not made updates to be compatible with the TW container?

It's a bug in the container runtime/seccomp handling of Leap.

...

If the first the bug was filed correctly. If the second either a) circle back to my comment about blocking the update, or b) at the very least the maintainer who knowingly made the incompatible change could file a maintenance bug.

Neither one. The bug is in the container runtime, and that is different on every product. So you need single bugs for every product which contains this bug to track that a fix get's released there.

Right, and it should be obvious that users need help by experts in the corresponding field creating these bug reports because – call me stupid, if you like – but even being a SUSE QA engineer I would not be able to achieve that by myself for every issue that I hit. Regards, Oliver

Hi, Am Donnerstag, 6. Mai 2021, 16:57:17 CEST schrieb Jimmy Berry:

On Wednesday, May 5, 2021 3:00:06 PM CDT Thorsten Kukuk wrote:

...

...

- Is the bug against TW since it updated a package such that the container would be incompatible with other hosts such as Leap and thus break the fundamental feature of a TW container.

No, the change in TW was correct.

Alright, so TW containers are not for serious workloads since incredibly common functionality can be "correctly" broken for months. Understood, thanks.

The "correctness" here is not about nitpicking interpretations of some standard, it's literally this: faccessat2(AT_FDCWD, "/", R_OK, 0) returns -EPERM for everything! That applies to *all* syscalls introduced after a certain point in the past and thus only gets worse in the future. For non-coders: It's like running on a CPU with some pins cut off.

TW blocks for all sorts of ring failures caused by upgrades, many times "correct" to one package that exposes a bug in another. The whole point of QA and our promise to users is that openSUSE does not release a borked TW intentionally. Apparently this does not apply to the containers and should be stated somewhere.

It does. openQA currently fully tests Tumbleweed containers only on Tumbleweed, and that always worked. Leap and SLE also perform some tests with the Tumbleweed container, but those didn't actually hit this particular bug as they don't go deep enough. I filed a ticket about testing TW containers on other distros a while ago: https://progress.opensuse.org/issues/88822 However, this was caught pretty soon after the release, so while we would've been aware of the breakage before publishing the affected snapshot, it would still require a fix: a) If caught before publishing, revert the breaking change? This was caused by a glibc version update, which is rather painful to revert. b) Apply a workaround to glibc? The maintainer was *vehemently* opposed to that option, so that was the end of that. AFAICT a temporary workaround at least for x86_64 would've been feasible, especially considering the severity and affected platforms. c) Block Tumbleweed publishing until the runtime is fixed? While this is potentially an option for Leap and SLE because we can handle it ourselves (or at least influence it in a meaningful way), this is not really the case for third-party platforms like GitHub. I would also blame the container runtime and providers there, because the TW container did everything correctly. The nature of the buggy behaviour in the runtime also made it rather tricky to address, because it was actually a whole set of syscalls which were broken in totally unexpected ways. Unfortunately it took quite some time to get this fixed upstream (but before this landed in TW) and even longer to actually have it deployed in third-party environments. Other distributions using glibc 2.33 versions were affected in the exact same way. Cheers, Fabian

Moin, Am Mittwoch, 5. Mai 2021, 21:18:55 CEST schrieb Sarah Julia Kriesch:

Thank you! Then we will continue with Tumbleweed images in the next steps. I have got another question for contributions to https://build.opensuse.org/project/show/openSUSE:Containers:Tumbleweed. All images are packages based on naming as we are using for Tumbleweed during releases. In which version should we create SR for improvements (and adding other architectures)? Or which name should be used for new container images then?

The container images are all in openSUSE:Factory, openSUSE:Containers is the project where built images are stored for publishing.

Is there any howto by us?

See https://en.opensuse.org/Building_derived_containers Cheers, Fabian

Best regards, Sarah