rke2 and cilium/envoy packaging
Hi, I would like to talk about packages related to Cilium and Envoy. As probably many of you know, Envoy-related packages in openSUSE: - envoy-proxy - cilium-proxy are quite hard to maintain. That's due to Envoy using Bazel build system and bundling a lot of dependencies (mostly C and C++ projects) through it. Some time ago, we agreed to pack the most of those dependencies into a tarball (vendor.tar.gz) while force dynamic linking of libraries which are anyhow cryptography- or security-related (boringssl, curl, nghttp2, zlib). Doing so automatically was achieved by this project: https://github.com/kubic-project/obs-service-bazel_repositories The problem is, that this project doesn't work for newest versions of Envoy anymore (at least >=1.17). Now Envoy also pulls some Python dependencies even for building the main binary and it requires concrete versions with concrete hashes through pip. For example: https://github.com/envoyproxy/envoy/blob/main/configs/requirements.txt Honestly speaking, I'm getting really frustrated by chasing over all the new Envoy/Bazel build features and trying to get them working with OBS offline builds. That's why I'm thinking about removing those packages (also Cilium, since it doesn't provide the whole functionality without Envoy) from Kubic. Since we already describe on the Kubic blog how to use rke2, Rancher is a SUSE product and rke2/rancher is based on mirrored (+ sometimes slightly patched) upstream container images, I think it's reasonable to stick to this approach. As well as, for example, we trust Flathub in openSUSE MicroOS Desktop. If curious about how Rancher does that: https://github.com/rancher/image-mirror I would be happy to keep Cilium in Kubic if there would be any possibility for using Dockerfiles (online when building), in that case I would be even happy to provide some set of patches to convert the base image layer to openSUSE TW and replace `apt` with `microdnf`/`zypper`. But if there is no such possibility of plan of supporting such workflow, I think I would rather remove Cilium from Kubic packaging and focus on maintaining it in rke2 and providing it to MicroOS users through Rancher tools. Please let me know what are your thoughts on that. Cheers, Michal
On 6/9/21 2:44 PM, Michal Rostecki wrote:
Hi,
I would like to talk about packages related to Cilium and Envoy. As probably many of you know, Envoy-related packages in openSUSE:
- envoy-proxy - cilium-proxy
are quite hard to maintain. That's due to Envoy using Bazel build system and bundling a lot of dependencies (mostly C and C++ projects) through it.
Some time ago, we agreed to pack the most of those dependencies into a tarball (vendor.tar.gz) while force dynamic linking of libraries which are anyhow cryptography- or security-related (boringssl, curl, nghttp2, zlib). Doing so automatically was achieved by this project:
https://github.com/kubic-project/obs-service-bazel_repositories
The problem is, that this project doesn't work for newest versions of Envoy anymore (at least >=1.17). Now Envoy also pulls some Python dependencies even for building the main binary and it requires concrete versions with concrete hashes through pip. For example:
https://github.com/envoyproxy/envoy/blob/main/configs/requirements.txt
Honestly speaking, I'm getting really frustrated by chasing over all the new Envoy/Bazel build features and trying to get them working with OBS offline builds.
That's why I'm thinking about removing those packages (also Cilium, since it doesn't provide the whole functionality without Envoy) from Kubic.
Since we already describe on the Kubic blog how to use rke2, Rancher is a SUSE product and rke2/rancher is based on mirrored (+ sometimes slightly patched) upstream container images, I think it's reasonable to stick to this approach. As well as, for example, we trust Flathub in openSUSE MicroOS Desktop.
If curious about how Rancher does that:
https://github.com/rancher/image-mirror
I would be happy to keep Cilium in Kubic if there would be any possibility for using Dockerfiles (online when building), in that case I would be even happy to provide some set of patches to convert the base image layer to openSUSE TW and replace `apt` with `microdnf`/`zypper`. But if there is no such possibility of plan of supporting such workflow, I think I would rather remove Cilium from Kubic packaging and focus on maintaining it in rke2 and providing it to MicroOS users through Rancher tools.
Please let me know what are your thoughts on that.
Cheers, Michal
I take the lack of replies as lack of opposition towards removing cilium packages and images from OBS. Here are the requests: https://build.opensuse.org/request/show/899929 https://build.opensuse.org/request/show/899930 https://build.opensuse.org/request/show/899931 https://build.opensuse.org/request/show/899932 https://build.opensuse.org/request/show/899933 I will write up some docs on openSUSE Wiki this week about using rke2 on MicroOS to deploy Cilium. Cheers, Michal
Hello,
In my opinion I find this okay. Since official OpenSUSE cilium packages
were slow to catch up to upstream I usually favored Cilium's official
images instead. The biggest appeal with OBS images was that they were
multi-arch. With that being a thing in Cilium 1.10.0 I don't see much of a
reason for an offline container build?
One question though. I think kubbictl relies on deploying Cilium with I
assume OBS's image of Cilium. Does that need to be swapped?
Thanks,
Anthony
On Mon, Jun 14, 2021 at 10:02 AM Michal Rostecki
On 6/9/21 2:44 PM, Michal Rostecki wrote:
Hi,
I would like to talk about packages related to Cilium and Envoy. As probably many of you know, Envoy-related packages in openSUSE:
- envoy-proxy - cilium-proxy
are quite hard to maintain. That's due to Envoy using Bazel build system and bundling a lot of dependencies (mostly C and C++ projects) through it.
Some time ago, we agreed to pack the most of those dependencies into a tarball (vendor.tar.gz) while force dynamic linking of libraries which are anyhow cryptography- or security-related (boringssl, curl, nghttp2, zlib). Doing so automatically was achieved by this project:
https://github.com/kubic-project/obs-service-bazel_repositories
The problem is, that this project doesn't work for newest versions of Envoy anymore (at least >=1.17). Now Envoy also pulls some Python dependencies even for building the main binary and it requires concrete versions with concrete hashes through pip. For example:
https://github.com/envoyproxy/envoy/blob/main/configs/requirements.txt
Honestly speaking, I'm getting really frustrated by chasing over all the new Envoy/Bazel build features and trying to get them working with OBS offline builds.
That's why I'm thinking about removing those packages (also Cilium, since it doesn't provide the whole functionality without Envoy) from Kubic.
Since we already describe on the Kubic blog how to use rke2, Rancher is a SUSE product and rke2/rancher is based on mirrored (+ sometimes slightly patched) upstream container images, I think it's reasonable to stick to this approach. As well as, for example, we trust Flathub in openSUSE MicroOS Desktop.
If curious about how Rancher does that:
https://github.com/rancher/image-mirror
I would be happy to keep Cilium in Kubic if there would be any possibility for using Dockerfiles (online when building), in that case I would be even happy to provide some set of patches to convert the base image layer to openSUSE TW and replace `apt` with `microdnf`/`zypper`. But if there is no such possibility of plan of supporting such workflow, I think I would rather remove Cilium from Kubic packaging and focus on maintaining it in rke2 and providing it to MicroOS users through Rancher tools.
Please let me know what are your thoughts on that.
Cheers, Michal
I take the lack of replies as lack of opposition towards removing cilium packages and images from OBS.
Here are the requests:
https://build.opensuse.org/request/show/899929 https://build.opensuse.org/request/show/899930 https://build.opensuse.org/request/show/899931 https://build.opensuse.org/request/show/899932 https://build.opensuse.org/request/show/899933
I will write up some docs on openSUSE Wiki this week about using rke2 on MicroOS to deploy Cilium.
Cheers, Michal
On 6/14/21 4:17 PM, Anthony J Rabbito wrote:
Hello,
In my opinion I find this okay. Since official OpenSUSE cilium packages were slow to catch up to upstream I usually favored Cilium's official images instead. The biggest appeal with OBS images was that they were multi-arch. With that being a thing in Cilium 1.10.0 I don't see much of a reason for an offline container build?
One question though. I think kubbictl relies on deploying Cilium with I assume OBS's image of Cilium. Does that need to be swapped?
Probably the right thing to do would be removing Cilium from kubic-control and offering it only through rke2 (or upstream helm chart). I will make a PR later today.
On Mon, 2021-06-14 at 16:43 +0200, Michal Rostecki wrote:
On 6/14/21 4:17 PM, Anthony J Rabbito wrote:
Hello,
In my opinion I find this okay. Since official OpenSUSE cilium packages were slow to catch up to upstream I usually favored Cilium's official images instead. The biggest appeal with OBS images was that they were multi-arch. With that being a thing in Cilium 1.10.0 I don't see much of a reason for an offline container build?
One question though. I think kubbictl relies on deploying Cilium with I assume OBS's image of Cilium. Does that need to be swapped?
Probably the right thing to do would be removing Cilium from kubic-control and offering it only through rke2 (or upstream helm chart). I will make a PR later today.
Hi Michal, All the PR's you submitted were unacceptable on the grounds that patterns and kubicd still required cilium/envoy Please make sure to remove any dependency between cilium/envoy and other core parts of the distribution before trying to remove cilium/envoy Regards, -- Richard Brown Linux Distribution Engineer - Future Technology Team Phone +4991174053-361 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, D-90409 Nuernberg (HRB 36809, AG Nürnberg) Geschäftsführer: Felix Imendörffer
On 6/21/21 11:09 AM, Richard Brown wrote:
On Mon, 2021-06-14 at 16:43 +0200, Michal Rostecki wrote:
On 6/14/21 4:17 PM, Anthony J Rabbito wrote:
Hello,
In my opinion I find this okay. Since official OpenSUSE cilium packages were slow to catch up to upstream I usually favored Cilium's official images instead. The biggest appeal with OBS images was that they were multi-arch. With that being a thing in Cilium 1.10.0 I don't see much of a reason for an offline container build?
One question though. I think kubbictl relies on deploying Cilium with I assume OBS's image of Cilium. Does that need to be swapped?
Probably the right thing to do would be removing Cilium from kubic-control and offering it only through rke2 (or upstream helm chart). I will make a PR later today.
Hi Michal,
All the PR's you submitted were unacceptable on the grounds that patterns and kubicd still required cilium/envoy
Please make sure to remove any dependency between cilium/envoy and other core parts of the distribution before trying to remove cilium/envoy
Regards,
Hi Richard, Here is my kubic-control PR which removes dependency on cilium: https://github.com/thkukuk/kubic-control/pull/34 I guess that after merging it, we need to make a new kubic-control release and then remove the dependency in packages and patterns? Thanks, Michal
participants (3)
-
Anthony J Rabbito -
Michal Rostecki -
Richard Brown