On Thursday, March 15, 2018 12:51:07 PM CDT Fabian Vogt wrote:

:tumbleweed. I think a transition period of a week is enough. The change is trivial enough to be applied everywhere within a week, the

Generally, I imagine people do not sit and wait for changes like this, but rather have work to do. Migrations are planned in advance rather than done adhoc when forced. I understand this is extremely minor to migrate to, but can be just as disruptive given the circumstances. Is this mentioned or explained on a wiki somewhere? We should have a simple page that has the right phrases people will search for so they can find the answer quickly.

bigger issue is to get the message out. My expectation is that the majority of users won't have received the mail, so they'll only notice the change after the image got removed.

I would agree, but a week for spreading isn't all that much.

It might be good idea to wait until Doug is back and can write an article on news.opensuse.org though and start the week for :tumbleweed and :42.2 deletion then.

What do you think about that?

Seems like no harm in that. Overall, great to see attention given to the images and be updated regularly as that was something I requests years ago. Thanks! -- Jimmy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hi, Am Sonntag, 18. März 2018, 20:41:12 CET schrieb Aleksa Sarai:

On 2018-03-15, Fabian Vogt wrote:

...

- You'll have to use opensuse/tumbleweed instead of opensuse:tumbleweed

This should be advertised more widely. I found that most people use the official library images even though they're out-of-date more often, and we don't have direct control over the building/publishing of those images.

Which is exactly the reason why tumbleweed will be gone from those soon. I'll have to see how images can be removed from the official library.

...

- In a week, opensuse/amd64, opensuse/arm64v8, opensuse/s390x and opensuse/ppc64le will be removed. Due to multi-arch images, they are no longer necessary.

How are we consolidating the multi-arch images from the single-arch images built in OBS? umoci doesn't have explicit support for creating multi-arch images (though it can interact with them), and skopeo doesn't really like creating OCI multi-arch images either.

I was quite disappointed about the state of multi-arch support in the tooling as well. It's practically completely missing, which meant I had to write a custom script for that. It's further complicated by the setup of openSUSE:Factory, with openQA in between OBS and releasing and the Ports project being split across multiple projects. It basically takes the images from OBS (built by kiwi, using umoci and skopeo), uploads all blobs (layers, config (why is that a blob...)) and the manifest to the registry. It notes down the digest of the image manifests and uploads a manifestlist combining all those images and assigns it the "latest" tag. The version number of the images is written into the mainfestlist (per-arch) as well, which makes lookup quick and easy. Code is on https://github.com/openSUSE/osc-plugin-factory/pull/1438 I would expect that even if there were multi-arch compatible tooling, it wouldn't allow the advantages this currently has, especially embedding the version into the manifestlist itself, but also not requiring to download all layers for all archs before uploading. Currently it only downloads and uploads the new/updated images. There is some work on the OBS side going on, which should eventually result in OBS being able to push multi-arch images to registries directly. Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hi, Am Montag, 19. März 2018, 16:25:23 CET schrieb Aleksa Sarai:

On 2018-03-19, Fabian Vogt wrote:

...

...

...

- In a week, opensuse/amd64, opensuse/arm64v8, opensuse/s390x and opensuse/ppc64le will be removed. Due to multi-arch images, they are no longer necessary.

How are we consolidating the multi-arch images from the single-arch images built in OBS? umoci doesn't have explicit support for creating multi-arch images (though it can interact with them), and skopeo doesn't really like creating OCI multi-arch images either.

I was quite disappointed about the state of multi-arch support in the tooling as well. It's practically completely missing, which meant I had to write a custom script for that.

This is because multi-arch is mostly just useful for making life easier, with making one name refer to multiple images (though users don't actually interact with each of those images). Multi-arch outside of distribution isn't super useful IMHO.

Isn't distribution literally the only purpose of OCI images and registries?

It's my impression that having skopeo be able to push an OCI image to specific Docker entries in the multi-arch manifest would resolve most of these issues -- because then you can just use the single-arch images built locally and then push them all to the same distribution reference.

Yes, that sounds like the best option.

...

It basically takes the images from OBS (built by kiwi, using umoci and skopeo), uploads all blobs (layers, config (why is that a blob...)) and the manifest to the registry. It notes down the digest of the image manifests and uploads a manifestlist combining all those images and assigns it the "latest" tag. The version number of the images is written into the mainfestlist (per-arch) as well, which makes lookup quick and easy.

Creation of multi-arch images is something that is on umoci's roadmap. The main hang-up is that nobody has a good idea for what the UX should be (and other multi-arch image builders have the same problem), and that it becomes harder to reason about operating on an image once there is more than one rootfs associated with it. Not to mention that it's not clear whether there is a usecase that isn't handled by the above outlined skopeo UX.

AFAIK OCI images can be multi-arch as well, so umoci should support that in some way at least. For our purpose at least it would be enough to have per-arch OCI images which we can independently push with skopeo.

If you have any ideas here, I'd appreciate it if you can add it in an issue on https://github.com/openSUSE/umoci.

...

I would expect that even if there were multi-arch compatible tooling, it wouldn't allow the advantages this currently has, especially embedding the version into the manifestlist itself, but also not requiring to download all layers for all archs before uploading.

Not necessarily, but I agree it would probably would be harder to implement to have the same advantages as it would have to touch both umoci and skopeo.

Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hi, Am Sonntag, 18. März 2018, 21:03:59 CET schrieb Aleksa Sarai:

On 2018-03-18, Peter Simons wrote:

...

Hi Fabian,

...

What does this mean to you? - In a week, opensuse:tumbleweed and opensuse:42.2 (EOL) will be removed - You'll have to use opensuse/tumbleweed instead of opensuse:tumbleweed

I've been using opensuse:leap successfully for a while. What about that name? Is that going to removed, too? opensuse/leap does not seem to exist.

opensuse:leap is an alias for 42.3 currently. IMO the tag is quite bad, as switching it to 15.0 after release might be surprising for some. opensuse:latest is currently eqivalent, pointing to 42.3 as well.

opensuse/amd64:42.3 (or opensuse/amd64:leap) is what we would've suggested using earlier, but with this new setup I would hope that means we're going to push everything to opensuse/<flavor>:<version>.

My original plan was to have a single source for official images. Leap will be available only as part of the official library (so opensuse:15.0) whereas Tumbleweed only as opensuse/tumbleweed. For Leap there's IMO no reason to move it from the official images, except the convoluted submission process, sending a pull request with a text file containing git repos and hashes...

I'm also wondering if there will be snapshot tags for <version> in opensuse/tumbleweed so people can pin what tumbleweed snapshot they want to start from (though pushing every snapshot might cause us some issues with storing too many images in Docker Hub).

I thought about that as well, but it's not terribly useful. There's only a single Tumbleweed snapshot's repo available on download.opensuse.org, so even if you run an older snapshot container, "zypper install" would get you the latest published package. Also, the Tumbleweed Ports architecture means the snapshots aren't in sync. In fact, not even close. This means you'll have (ex.) 20180314 for x86_64, 20180309 for ppc64le and aarch64 and no snapshot for s390x at all. The images are openQA tested, which (hopefully ;-) ) means that there won't be any broken images pushed to the Docker Hub. Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hi, Am Montag, 19. März 2018, 16:32:17 CET schrieb Aleksa Sarai:

On 2018-03-19, Fabian Vogt wrote:

...

...

opensuse/amd64:42.3 (or opensuse/amd64:leap) is what we would've suggested using earlier, but with this new setup I would hope that means we're going to push everything to opensuse/<flavor>:<version>.

My original plan was to have a single source for official images. Leap will be available only as part of the official library (so opensuse:15.0) whereas Tumbleweed only as opensuse/tumbleweed.

For Leap there's IMO no reason to move it from the official images, except the convoluted submission process, sending a pull request with a text file containing git repos and hashes...

The issue is actually larger than this. The official-library doesn't use the image we create in OBS directly. Rather it takes a tar of the rootfs and then applies them *inside* a Docker build.

Yes, that's part of the "convoluted submission process"...

Because they (still!) use AUFS, this causes the rootfs to not be identical to the one we built (in particular this strips certain xattrs like "security.capability").

Ouch, I did not know that. Does https://github.com/docker-library/official-images/blob/master/library/opensu... ("Constraints: !aufs") not have an effect? I assume it would apply not only to the users but also to the builder itself. If it doesn't, it's actually a reason to pull all of the images from the library and not just tumbleweed.

We discovered this when the official-library build broke when we fixed "security.capability" support in KIWI (they didn't notice the breakage during the PR process because they test PRs on non-AUFS Docker, but build it on AUFS Docker), and I had to write some Docker patches to "fix" it so that our images would be in official-library again (the build had been failing since).

I would always recommend that people use the images we build in OBS, for the same reason we recommend people use packages built in OBS rather than packages built manually via rpmbuild(1) on a random machine in some stranger's basement.

We'll likely have an openSUSE registry and notary, which means this won't be an issue in the long term anymore. I'll have to read up on the notary subject though. I don't think it's an issue for the opensuse/ NS on docker hub either, as there we control every bit of the blobs landing on the client and they can verify that by comparing the digest. We'll just have to make the list easy to access and provide a guide for verification. It's basically identical to the downloading of .iso images from download.opensuse.org - the download itself and the file can be modified, but we provide a checksum for (sadly voluntary) verification. There's a few comments about that in the osc-plugin-factory PR.

The obvious downside with only having "opensuse/..." images is that people won't know why "docker pull opensuse" doesn't work anymore.

Yes, but that just needs to be communicated. Who knows, if done properly it might serve as motivation for the official library to get some restructuring. Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hi, Am Dienstag, 20. März 2018, 00:03:20 CET schrieb Aleksa Sarai:

On 2018-03-19, Fabian Vogt wrote:

...

...

The issue is actually larger than this. The official-library doesn't use the image we create in OBS directly. Rather it takes a tar of the rootfs and then applies them *inside* a Docker build.

Yes, that's part of the "convoluted submission process"...

My point was that the *build and publishing* process was also worryingly convoluted. Submission processes only really affect us, but if the build is convoluted and may break things then it affects users.

The build is part of the submission from my viewpoint. The submission is every step until the image appears at its destination. Especially as the image remains even if the source git repo got deleted.

...

...

Because they (still!) use AUFS, this causes the rootfs to not be identical to the one we built (in particular this strips certain xattrs like "security.capability").

Ouch, I did not know that. Does https://github.com/docker-library/official-images/blob/master/library/opensu... ("Constraints: !aufs") not have an effect? I assume it would apply not only to the users but also to the builder itself.

I fear I may be getting a bit old, I completely forgot about this change. :facepalm: On paper that should have fixed the AUFS issue, yes.

However the image builds still do a Docker build which extracts a tar archive. They are somewhat careful about reproducibility, but my experience with it has left me a little jaded (not to mention that it all goes through an entire round-trip inside Docker's tar handling -- which is not using tar-split to ensure layer IDs are consistent).

Yes, I guess to verify the integrity we'd need to download the published image and compare it with the files we pushed. That's not good (i.e. unacceptable).

...

If it doesn't, it's actually a reason to pull all of the images from the library and not just tumbleweed.

I agree. If we're going to move tumbleweed, I don't see why we wouldn't move all of them.

Tumbleweed as rolling release is an exception - but if you think that leap should be moved as well (it also makes publishing easier), it's probably the best option. Currently there's no built+tested+published Leap 15.0 image, once there is I'll set up a repository. I'll try to add updated images for 42.3 as well.

...

I don't think it's an issue for the opensuse/ NS on docker hub either, as there we control every bit of the blobs landing on the client and they can verify that by comparing the digest. We'll just have to make the list easy to access and provide a guide for verification.

It's basically identical to the downloading of .iso images from download.opensuse.org - the download itself and the file can be modified, but we provide a checksum for (sadly voluntary) verification. There's a few comments about that in the osc-plugin-factory PR.

The "opensuse/..." namespace is what I meant by "images we built in OBS", because those bits are exact copies of the files we build inside OBS. The official-library makes the files go through a round-trip build before publishing them (breaking the reproducibility).

Yes, they should be - yet can't be 100% certain that they are as we don't have control over the docker hub either. That's why signing or some other way of verification is important to offer.

This is one of the reasons why I set up "opensuse/amd64" once we discovered the above Docker breakage. The main downside with not using official-library is that we cannot sign images in "opensuse/...".

We can sign images in the official-library? Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Hi Fabian,

...

I've been using opensuse:leap successfully for a while. What about that name? Is that going to removed, too? opensuse/leap does not seem to exist.

opensuse:leap is an alias for 42.3 currently. IMO the tag is quite bad, as switching it to 15.0 after release might be surprising for some.

actually, I am using that tag precisely because I want it to switch to 15.0 when the new version comes out. If I didn't want that, than I'd be using a tag that contains an explicit version number. Anyhow, what tag am I supposed to use to get Leap 15.0 (once it's released)? Best regards, Peter -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org