[opensuse-factory] Changes to the openSUSE images on Docker Hub
Hi everyone, (tl;dr? Skip to the final paragraph) Currently there's a major overhaul of the release process for Docker images ongoing. What's done so far: - The Tumbleweed docker image for x86_64 is part of the distribution and tested by openQA now: https://openqa.opensuse.org/tests/latest?flavor=DVD&test=docker_image&arch=x86_64&machine=64bit&version=Tumbleweed&distri=opensuse - There is an automated release bot publishing the tested image to Docker Hub at opensuse/tumbleweed. - For Leap 15.0 there's a Docker image being built next to the Live images What's missing: - Clean up the opensuse/ namespace on Docker Hub - Clean up the opensuse images in the official library - Get the Leap 15.0 image tested and released - Get the Tumbleweed container built as part of the distribution and tested by openQA for other architectures (Ports) as well. Currently they are built manually to avoid regressions and should be considered experimental. What does this mean to you? - In a week, opensuse:tumbleweed and opensuse:42.2 (EOL) will be removed - You'll have to use opensuse/tumbleweed instead of opensuse:tumbleweed - In a week, opensuse/amd64, opensuse/arm64v8, opensuse/s390x and opensuse/ppc64le will be removed. Due to multi-arch images, they are no longer necessary. - Leap 15.0 will be part of the official library soon (can't give an ETA yet) Have a lot of fun, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday, March 15, 2018 3:18:51 AM CDT Fabian Vogt wrote:
What does this mean to you? - In a week, opensuse:tumbleweed and opensuse:42.2 (EOL) will be removed - You'll have to use opensuse/tumbleweed instead of opensuse:tumbleweed
Unless there is a very good reason it would seem far better to provide a transition period of a few months at least. I can imagine people with various Dockerfiles and chains on dockerhub and similar that would rather everything not break all of the sudden without a good reason. Even if :tumbleweed is not updated that is still better as it went months without update previously. Anyone using it properly should be running `zypper dup` right away and will see no adverse effects. -- Jimmy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Hi, Am Donnerstag, 15. März 2018, 17:22:12 CET schrieb Jimmy Berry:
On Thursday, March 15, 2018 3:18:51 AM CDT Fabian Vogt wrote:
What does this mean to you? - In a week, opensuse:tumbleweed and opensuse:42.2 (EOL) will be removed - You'll have to use opensuse/tumbleweed instead of opensuse:tumbleweed
Unless there is a very good reason it would seem far better to provide a transition period of a few months at least. I can imagine people with various Dockerfiles and chains on dockerhub and similar that would rather everything not break all of the sudden without a good reason.
Even if :tumbleweed is not updated that is still better as it went months without update previously. Anyone using it properly should be running `zypper dup` right away and will see no adverse effects.
I understand your point - but IMO it fits opensuse:42.2 more than :tumbleweed. I think a transition period of a week is enough. The change is trivial enough to be applied everywhere within a week, the bigger issue is to get the message out. My expectation is that the majority of users won't have received the mail, so they'll only notice the change after the image got removed. For opensuse:{amd64,arm64v8,...} a transition period of a week is almost too much, those images aren't supposed to be used at all. Their only purpose was once to act as a fallback if pushing to the official library did not work. It might be good idea to wait until Doug is back and can write an article on news.opensuse.org though and start the week for :tumbleweed and :42.2 deletion then. What do you think about that? Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday, March 15, 2018 12:51:07 PM CDT Fabian Vogt wrote:
:tumbleweed. I think a transition period of a week is enough. The change is trivial enough to be applied everywhere within a week, the
Generally, I imagine people do not sit and wait for changes like this, but rather have work to do. Migrations are planned in advance rather than done adhoc when forced. I understand this is extremely minor to migrate to, but can be just as disruptive given the circumstances. Is this mentioned or explained on a wiki somewhere? We should have a simple page that has the right phrases people will search for so they can find the answer quickly.
bigger issue is to get the message out. My expectation is that the majority of users won't have received the mail, so they'll only notice the change after the image got removed.
I would agree, but a week for spreading isn't all that much.
It might be good idea to wait until Doug is back and can write an article on news.opensuse.org though and start the week for :tumbleweed and :42.2 deletion then.
What do you think about that?
Seems like no harm in that. Overall, great to see attention given to the images and be updated regularly as that was something I requests years ago. Thanks! -- Jimmy -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2018-03-15, Fabian Vogt <fvogt@suse.com> wrote:
- You'll have to use opensuse/tumbleweed instead of opensuse:tumbleweed
This should be advertised more widely. I found that most people use the official library images even though they're out-of-date more often, and we don't have direct control over the building/publishing of those images.
- In a week, opensuse/amd64, opensuse/arm64v8, opensuse/s390x and opensuse/ppc64le will be removed. Due to multi-arch images, they are no longer necessary.
How are we consolidating the multi-arch images from the single-arch images built in OBS? umoci doesn't have explicit support for creating multi-arch images (though it can interact with them), and skopeo doesn't really like creating OCI multi-arch images either. -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH <https://www.cyphar.com/>
Hi, Am Sonntag, 18. März 2018, 20:41:12 CET schrieb Aleksa Sarai:
On 2018-03-15, Fabian Vogt <fvogt@suse.com> wrote:
- You'll have to use opensuse/tumbleweed instead of opensuse:tumbleweed
This should be advertised more widely. I found that most people use the official library images even though they're out-of-date more often, and we don't have direct control over the building/publishing of those images.
Which is exactly the reason why tumbleweed will be gone from those soon. I'll have to see how images can be removed from the official library.
- In a week, opensuse/amd64, opensuse/arm64v8, opensuse/s390x and opensuse/ppc64le will be removed. Due to multi-arch images, they are no longer necessary.
How are we consolidating the multi-arch images from the single-arch images built in OBS? umoci doesn't have explicit support for creating multi-arch images (though it can interact with them), and skopeo doesn't really like creating OCI multi-arch images either.
I was quite disappointed about the state of multi-arch support in the tooling as well. It's practically completely missing, which meant I had to write a custom script for that. It's further complicated by the setup of openSUSE:Factory, with openQA in between OBS and releasing and the Ports project being split across multiple projects. It basically takes the images from OBS (built by kiwi, using umoci and skopeo), uploads all blobs (layers, config (why is that a blob...)) and the manifest to the registry. It notes down the digest of the image manifests and uploads a manifestlist combining all those images and assigns it the "latest" tag. The version number of the images is written into the mainfestlist (per-arch) as well, which makes lookup quick and easy. Code is on https://github.com/openSUSE/osc-plugin-factory/pull/1438 I would expect that even if there were multi-arch compatible tooling, it wouldn't allow the advantages this currently has, especially embedding the version into the manifestlist itself, but also not requiring to download all layers for all archs before uploading. Currently it only downloads and uploads the new/updated images. There is some work on the OBS side going on, which should eventually result in OBS being able to push multi-arch images to registries directly. Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2018-03-19, Fabian Vogt <fvogt@suse.com> wrote:
- In a week, opensuse/amd64, opensuse/arm64v8, opensuse/s390x and opensuse/ppc64le will be removed. Due to multi-arch images, they are no longer necessary.
How are we consolidating the multi-arch images from the single-arch images built in OBS? umoci doesn't have explicit support for creating multi-arch images (though it can interact with them), and skopeo doesn't really like creating OCI multi-arch images either.
I was quite disappointed about the state of multi-arch support in the tooling as well. It's practically completely missing, which meant I had to write a custom script for that.
This is because multi-arch is mostly just useful for making life easier, with making one name refer to multiple images (though users don't actually interact with each of those images). Multi-arch outside of distribution isn't super useful IMHO. It's my impression that having skopeo be able to push an OCI image to specific Docker entries in the multi-arch manifest would resolve most of these issues -- because then you can just use the single-arch images built locally and then push them all to the same distribution reference.
It basically takes the images from OBS (built by kiwi, using umoci and skopeo), uploads all blobs (layers, config (why is that a blob...)) and the manifest to the registry. It notes down the digest of the image manifests and uploads a manifestlist combining all those images and assigns it the "latest" tag. The version number of the images is written into the mainfestlist (per-arch) as well, which makes lookup quick and easy.
Creation of multi-arch images is something that is on umoci's roadmap. The main hang-up is that nobody has a good idea for what the UX should be (and other multi-arch image builders have the same problem), and that it becomes harder to reason about operating on an image once there is more than one rootfs associated with it. Not to mention that it's not clear whether there is a usecase that isn't handled by the above outlined skopeo UX. If you have any ideas here, I'd appreciate it if you can add it in an issue on <https://github.com/openSUSE/umoci>.
I would expect that even if there were multi-arch compatible tooling, it wouldn't allow the advantages this currently has, especially embedding the version into the manifestlist itself, but also not requiring to download all layers for all archs before uploading.
Not necessarily, but I agree it would probably would be harder to implement to have the same advantages as it would have to touch both umoci and skopeo. -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH <https://www.cyphar.com/>
Hi, Am Montag, 19. März 2018, 16:25:23 CET schrieb Aleksa Sarai:
On 2018-03-19, Fabian Vogt <fvogt@suse.com> wrote:
- In a week, opensuse/amd64, opensuse/arm64v8, opensuse/s390x and opensuse/ppc64le will be removed. Due to multi-arch images, they are no longer necessary.
How are we consolidating the multi-arch images from the single-arch images built in OBS? umoci doesn't have explicit support for creating multi-arch images (though it can interact with them), and skopeo doesn't really like creating OCI multi-arch images either.
I was quite disappointed about the state of multi-arch support in the tooling as well. It's practically completely missing, which meant I had to write a custom script for that.
This is because multi-arch is mostly just useful for making life easier, with making one name refer to multiple images (though users don't actually interact with each of those images). Multi-arch outside of distribution isn't super useful IMHO.
Isn't distribution literally the only purpose of OCI images and registries?
It's my impression that having skopeo be able to push an OCI image to specific Docker entries in the multi-arch manifest would resolve most of these issues -- because then you can just use the single-arch images built locally and then push them all to the same distribution reference.
Yes, that sounds like the best option.
It basically takes the images from OBS (built by kiwi, using umoci and skopeo), uploads all blobs (layers, config (why is that a blob...)) and the manifest to the registry. It notes down the digest of the image manifests and uploads a manifestlist combining all those images and assigns it the "latest" tag. The version number of the images is written into the mainfestlist (per-arch) as well, which makes lookup quick and easy.
Creation of multi-arch images is something that is on umoci's roadmap. The main hang-up is that nobody has a good idea for what the UX should be (and other multi-arch image builders have the same problem), and that it becomes harder to reason about operating on an image once there is more than one rootfs associated with it. Not to mention that it's not clear whether there is a usecase that isn't handled by the above outlined skopeo UX.
AFAIK OCI images can be multi-arch as well, so umoci should support that in some way at least. For our purpose at least it would be enough to have per-arch OCI images which we can independently push with skopeo.
If you have any ideas here, I'd appreciate it if you can add it in an issue on <https://github.com/openSUSE/umoci>.
I would expect that even if there were multi-arch compatible tooling, it wouldn't allow the advantages this currently has, especially embedding the version into the manifestlist itself, but also not requiring to download all layers for all archs before uploading.
Not necessarily, but I agree it would probably would be harder to implement to have the same advantages as it would have to touch both umoci and skopeo.
Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2018-03-19, Fabian Vogt <fvogt@suse.com> wrote:
I was quite disappointed about the state of multi-arch support in the tooling as well. It's practically completely missing, which meant I had to write a custom script for that.
This is because multi-arch is mostly just useful for making life easier, with making one name refer to multiple images (though users don't actually interact with each of those images). Multi-arch outside of distribution isn't super useful IMHO.
Isn't distribution literally the only purpose of OCI images and registries?
Distribution and interoperability between container tools. But multi-arch is only very useful for the Docker-registry style of distribution (rather than other possible ways of accomplishing distribution). Also the OCI doesn't have a distribution specification at the moment (though it is being worked on as we speak).
It basically takes the images from OBS (built by kiwi, using umoci and skopeo), uploads all blobs (layers, config (why is that a blob...)) and the manifest to the registry. It notes down the digest of the image manifests and uploads a manifestlist combining all those images and assigns it the "latest" tag. The version number of the images is written into the mainfestlist (per-arch) as well, which makes lookup quick and easy.
Creation of multi-arch images is something that is on umoci's roadmap. The main hang-up is that nobody has a good idea for what the UX should be (and other multi-arch image builders have the same problem), and that it becomes harder to reason about operating on an image once there is more than one rootfs associated with it. Not to mention that it's not clear whether there is a usecase that isn't handled by the above outlined skopeo UX.
AFAIK OCI images can be multi-arch as well, so umoci should support that in some way at least. For our purpose at least it would be enough to have per-arch OCI images which we can independently push with skopeo.
umoci does support interoperating with multi-arch images (on a best-effort basis as there are some ambiguous cases I don't really want to get into in this thread), but it doesn't support creating new ones because of the UX problem I mentioned. Internally umoci can trivially support modifying, reading, or creating one. The problem is how should the UX look for doing that? Here are examples of some of the thought-experiments I went through when first looking at this problem when OCI added multi-arch: * Should umoci automatically use the architecture/operating-system of the machine it's being run on? Maybe, but even then there might be more than one image that matches just those requirements. And how should they specify they want to act on a different set of architecture/operating-system? * How should a user specify additional requirements (such as annotations)? Maybe by adding filters on annotations. Should those filters be treated the same as architecture/operating-system (which are not annotations)? What if a user wants to filter non-annotations? * Should we have a UX for dealing with ambiguous cases (effectively giving users the option to select which image they actually meant)? What if umoci is not being run interactively? * Are annotations considered to be applied to children of an Index? After I brought this up and a lot of discussion informally, the conclusion was effectively "no" but this is not mentioned in the specification. * There are certain cases where you may have a "multi-arch" image that doesn't use the Index construct explicitly (by having multiple top-level images that have the same refname). umoci treats this the same as those that have an explicit Index, but it's not clear in the specification if this is meant to be how it should be treated. None of the above properties are specified in the specification, and reference resolution is also unspecified (despite discussions before 1.0 saying that we will specify it). Which means that different tools may have different opinions about refname resolution, but they all would still be considered compliant. So from my point of view, "multi-arch" in OCI is effectively not specified enough in order to make a sane UX decision about it. And while I do have some ideas about what the least insane UX for it would look like (and eventually we will have to implement it if people need to use multi-arch images like this), I'm worried that this will result in more confusion for OCI users. -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH <https://www.cyphar.com/>
Hi Fabian,
What does this mean to you? - In a week, opensuse:tumbleweed and opensuse:42.2 (EOL) will be removed - You'll have to use opensuse/tumbleweed instead of opensuse:tumbleweed
I've been using opensuse:leap successfully for a while. What about that name? Is that going to removed, too? opensuse/leap does not seem to exist. Best regards, Peter -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2018-03-18, Peter Simons <psimons@suse.de> wrote:
Hi Fabian,
What does this mean to you? - In a week, opensuse:tumbleweed and opensuse:42.2 (EOL) will be removed - You'll have to use opensuse/tumbleweed instead of opensuse:tumbleweed
I've been using opensuse:leap successfully for a while. What about that name? Is that going to removed, too? opensuse/leap does not seem to exist.
opensuse/amd64:42.3 (or opensuse/amd64:leap) is what we would've suggested using earlier, but with this new setup I would hope that means we're going to push everything to opensuse/<flavor>:<version>. I'm also wondering if there will be snapshot tags for <version> in opensuse/tumbleweed so people can pin what tumbleweed snapshot they want to start from (though pushing every snapshot might cause us some issues with storing too many images in Docker Hub). -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH <https://www.cyphar.com/>
Hi, Am Sonntag, 18. März 2018, 21:03:59 CET schrieb Aleksa Sarai:
On 2018-03-18, Peter Simons <psimons@suse.de> wrote:
Hi Fabian,
What does this mean to you? - In a week, opensuse:tumbleweed and opensuse:42.2 (EOL) will be removed - You'll have to use opensuse/tumbleweed instead of opensuse:tumbleweed
I've been using opensuse:leap successfully for a while. What about that name? Is that going to removed, too? opensuse/leap does not seem to exist.
opensuse:leap is an alias for 42.3 currently. IMO the tag is quite bad, as switching it to 15.0 after release might be surprising for some. opensuse:latest is currently eqivalent, pointing to 42.3 as well.
opensuse/amd64:42.3 (or opensuse/amd64:leap) is what we would've suggested using earlier, but with this new setup I would hope that means we're going to push everything to opensuse/<flavor>:<version>.
My original plan was to have a single source for official images. Leap will be available only as part of the official library (so opensuse:15.0) whereas Tumbleweed only as opensuse/tumbleweed. For Leap there's IMO no reason to move it from the official images, except the convoluted submission process, sending a pull request with a text file containing git repos and hashes...
I'm also wondering if there will be snapshot tags for <version> in opensuse/tumbleweed so people can pin what tumbleweed snapshot they want to start from (though pushing every snapshot might cause us some issues with storing too many images in Docker Hub).
I thought about that as well, but it's not terribly useful. There's only a single Tumbleweed snapshot's repo available on download.opensuse.org, so even if you run an older snapshot container, "zypper install" would get you the latest published package. Also, the Tumbleweed Ports architecture means the snapshots aren't in sync. In fact, not even close. This means you'll have (ex.) 20180314 for x86_64, 20180309 for ppc64le and aarch64 and no snapshot for s390x at all. The images are openQA tested, which (hopefully ;-) ) means that there won't be any broken images pushed to the Docker Hub. Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2018-03-19, Fabian Vogt <fvogt@suse.com> wrote:
opensuse/amd64:42.3 (or opensuse/amd64:leap) is what we would've suggested using earlier, but with this new setup I would hope that means we're going to push everything to opensuse/<flavor>:<version>.
My original plan was to have a single source for official images. Leap will be available only as part of the official library (so opensuse:15.0) whereas Tumbleweed only as opensuse/tumbleweed.
For Leap there's IMO no reason to move it from the official images, except the convoluted submission process, sending a pull request with a text file containing git repos and hashes...
The issue is actually larger than this. The official-library doesn't use the image we create in OBS directly. Rather it takes a tar of the rootfs and then applies them *inside* a Docker build. Because they (still!) use AUFS, this causes the rootfs to not be identical to the one we built (in particular this strips certain xattrs like "security.capability"). We discovered this when the official-library build broke when we fixed "security.capability" support in KIWI (they didn't notice the breakage during the PR process because they test PRs on non-AUFS Docker, but build it on AUFS Docker), and I had to write some Docker patches to "fix" it so that our images would be in official-library again (the build had been failing since). I would always recommend that people use the images we build in OBS, for the same reason we recommend people use packages built in OBS rather than packages built manually via rpmbuild(1) on a random machine in some stranger's basement. The obvious downside with only having "opensuse/..." images is that people won't know why "docker pull opensuse" doesn't work anymore. -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH <https://www.cyphar.com/>
Hi, Am Montag, 19. März 2018, 16:32:17 CET schrieb Aleksa Sarai:
On 2018-03-19, Fabian Vogt <fvogt@suse.com> wrote:
opensuse/amd64:42.3 (or opensuse/amd64:leap) is what we would've suggested using earlier, but with this new setup I would hope that means we're going to push everything to opensuse/<flavor>:<version>.
My original plan was to have a single source for official images. Leap will be available only as part of the official library (so opensuse:15.0) whereas Tumbleweed only as opensuse/tumbleweed.
For Leap there's IMO no reason to move it from the official images, except the convoluted submission process, sending a pull request with a text file containing git repos and hashes...
The issue is actually larger than this. The official-library doesn't use the image we create in OBS directly. Rather it takes a tar of the rootfs and then applies them *inside* a Docker build.
Yes, that's part of the "convoluted submission process"...
Because they (still!) use AUFS, this causes the rootfs to not be identical to the one we built (in particular this strips certain xattrs like "security.capability").
Ouch, I did not know that. Does https://github.com/docker-library/official-images/blob/master/library/opensu... ("Constraints: !aufs") not have an effect? I assume it would apply not only to the users but also to the builder itself. If it doesn't, it's actually a reason to pull all of the images from the library and not just tumbleweed.
We discovered this when the official-library build broke when we fixed "security.capability" support in KIWI (they didn't notice the breakage during the PR process because they test PRs on non-AUFS Docker, but build it on AUFS Docker), and I had to write some Docker patches to "fix" it so that our images would be in official-library again (the build had been failing since).
I would always recommend that people use the images we build in OBS, for the same reason we recommend people use packages built in OBS rather than packages built manually via rpmbuild(1) on a random machine in some stranger's basement.
We'll likely have an openSUSE registry and notary, which means this won't be an issue in the long term anymore. I'll have to read up on the notary subject though. I don't think it's an issue for the opensuse/ NS on docker hub either, as there we control every bit of the blobs landing on the client and they can verify that by comparing the digest. We'll just have to make the list easy to access and provide a guide for verification. It's basically identical to the downloading of .iso images from download.opensuse.org - the download itself and the file can be modified, but we provide a checksum for (sadly voluntary) verification. There's a few comments about that in the osc-plugin-factory PR.
The obvious downside with only having "opensuse/..." images is that people won't know why "docker pull opensuse" doesn't work anymore.
Yes, but that just needs to be communicated. Who knows, if done properly it might serve as motivation for the official library to get some restructuring. Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2018-03-19, Fabian Vogt <fvogt@suse.com> wrote:
The issue is actually larger than this. The official-library doesn't use the image we create in OBS directly. Rather it takes a tar of the rootfs and then applies them *inside* a Docker build.
Yes, that's part of the "convoluted submission process"...
My point was that the *build and publishing* process was also worryingly convoluted. Submission processes only really affect us, but if the build is convoluted and may break things then it affects users.
Because they (still!) use AUFS, this causes the rootfs to not be identical to the one we built (in particular this strips certain xattrs like "security.capability").
Ouch, I did not know that. Does https://github.com/docker-library/official-images/blob/master/library/opensu... ("Constraints: !aufs") not have an effect? I assume it would apply not only to the users but also to the builder itself.
I fear I may be getting a bit old, I completely forgot about this change. :facepalm: On paper that should have fixed the AUFS issue, yes. However the image builds still do a Docker build which extracts a tar archive. They are somewhat careful about reproducibility, but my experience with it has left me a little jaded (not to mention that it all goes through an entire round-trip inside Docker's tar handling -- which is not using tar-split to ensure layer IDs are consistent).
If it doesn't, it's actually a reason to pull all of the images from the library and not just tumbleweed.
I agree. If we're going to move tumbleweed, I don't see why we wouldn't move all of them.
I don't think it's an issue for the opensuse/ NS on docker hub either, as there we control every bit of the blobs landing on the client and they can verify that by comparing the digest. We'll just have to make the list easy to access and provide a guide for verification.
It's basically identical to the downloading of .iso images from download.opensuse.org - the download itself and the file can be modified, but we provide a checksum for (sadly voluntary) verification. There's a few comments about that in the osc-plugin-factory PR.
The "opensuse/..." namespace is what I meant by "images we built in OBS", because those bits are exact copies of the files we build inside OBS. The official-library makes the files go through a round-trip build before publishing them (breaking the reproducibility). This is one of the reasons why I set up "opensuse/amd64" once we discovered the above Docker breakage. The main downside with not using official-library is that we cannot sign images in "opensuse/...". -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH <https://www.cyphar.com/>
Hi, Am Dienstag, 20. März 2018, 00:03:20 CET schrieb Aleksa Sarai:
On 2018-03-19, Fabian Vogt <fvogt@suse.com> wrote:
The issue is actually larger than this. The official-library doesn't use the image we create in OBS directly. Rather it takes a tar of the rootfs and then applies them *inside* a Docker build.
Yes, that's part of the "convoluted submission process"...
My point was that the *build and publishing* process was also worryingly convoluted. Submission processes only really affect us, but if the build is convoluted and may break things then it affects users.
The build is part of the submission from my viewpoint. The submission is every step until the image appears at its destination. Especially as the image remains even if the source git repo got deleted.
Because they (still!) use AUFS, this causes the rootfs to not be identical to the one we built (in particular this strips certain xattrs like "security.capability").
Ouch, I did not know that. Does https://github.com/docker-library/official-images/blob/master/library/opensu... ("Constraints: !aufs") not have an effect? I assume it would apply not only to the users but also to the builder itself.
I fear I may be getting a bit old, I completely forgot about this change. :facepalm: On paper that should have fixed the AUFS issue, yes.
However the image builds still do a Docker build which extracts a tar archive. They are somewhat careful about reproducibility, but my experience with it has left me a little jaded (not to mention that it all goes through an entire round-trip inside Docker's tar handling -- which is not using tar-split to ensure layer IDs are consistent).
Yes, I guess to verify the integrity we'd need to download the published image and compare it with the files we pushed. That's not good (i.e. unacceptable).
If it doesn't, it's actually a reason to pull all of the images from the library and not just tumbleweed.
I agree. If we're going to move tumbleweed, I don't see why we wouldn't move all of them.
Tumbleweed as rolling release is an exception - but if you think that leap should be moved as well (it also makes publishing easier), it's probably the best option. Currently there's no built+tested+published Leap 15.0 image, once there is I'll set up a repository. I'll try to add updated images for 42.3 as well.
I don't think it's an issue for the opensuse/ NS on docker hub either, as there we control every bit of the blobs landing on the client and they can verify that by comparing the digest. We'll just have to make the list easy to access and provide a guide for verification.
It's basically identical to the downloading of .iso images from download.opensuse.org - the download itself and the file can be modified, but we provide a checksum for (sadly voluntary) verification. There's a few comments about that in the osc-plugin-factory PR.
The "opensuse/..." namespace is what I meant by "images we built in OBS", because those bits are exact copies of the files we build inside OBS. The official-library makes the files go through a round-trip build before publishing them (breaking the reproducibility).
Yes, they should be - yet can't be 100% certain that they are as we don't have control over the docker hub either. That's why signing or some other way of verification is important to offer.
This is one of the reasons why I set up "opensuse/amd64" once we discovered the above Docker breakage. The main downside with not using official-library is that we cannot sign images in "opensuse/...".
We can sign images in the official-library? Cheers, Fabian -- Fabian Vogt - Release Engineer SUSE Linux GmbH GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg) Maxfeldstr. 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2018-03-20, Fabian Vogt <fvogt@suse.com> wrote:
This is one of the reasons why I set up "opensuse/amd64" once we discovered the above Docker breakage. The main downside with not using official-library is that we cannot sign images in "opensuse/...".
We can sign images in the official-library?
We don't (and can't) sign them, Docker Inc signs them. But images outside official-library are not signed at all. I'm not really sure which is better overall (especially given that they are signing artefacts which are not actually traceable to the artefacts we produce which have detached and repomd signatures). -- Aleksa Sarai Senior Software Engineer (Containers) SUSE Linux GmbH <https://www.cyphar.com/>
Hi Fabian,
I've been using opensuse:leap successfully for a while. What about that name? Is that going to removed, too? opensuse/leap does not seem to exist.
opensuse:leap is an alias for 42.3 currently. IMO the tag is quite bad, as switching it to 15.0 after release might be surprising for some.
actually, I am using that tag precisely because I want it to switch to 15.0 when the new version comes out. If I didn't want that, than I'd be using a tag that contains an explicit version number. Anyhow, what tag am I supposed to use to get Leap 15.0 (once it's released)? Best regards, Peter -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (4)
-
Aleksa Sarai -
Fabian Vogt -
Jimmy Berry -
Peter Simons