[opensuse-buildservice] Running chroot based OBS worker in kubernetes cluster and how?
Hey, guys: I am an infrastructure member from openEuler community (openEuler is an open source, free Linux distribution platform. The platform provides an open community for global developers to build an open, diversified, and architecture-inclusive software ecosystem. [1]), we use OBS for our distribution packaging and releasing. since our worker doesn't utilize any virtualization technologies (at least in worker service, only chroot is used). we are wondering whether we could containerize our obs worker into kubernetes clusters. Within several hours of research, I found there is a guide [2] which utilizes the kubernetes device plugin and announces it's alpha implementation. I reckon this might be a correct direction but still have some questions. 1. Are there any known issues or disadvantages running an obs worker in a container? 2. all of the container images are tagged with unstable (registry.opensuse.org/obs/server/unstable/container/leap151/containers/openbuildservice/backend for instance), do we have any official docker images? and where can I get the dockerfile of the image? 3. if we only containerize our obs worker that means we need to route requests from backend back to obs worker via nodeport, therefore we need to update the ip address of the worker to node's. Is there a way to achieve this? Thanks TommyLike [1]: https://openeuler.org/en/ [2]: https://openbuildservice.org/files/manuals/obs-admin-guide.pdf -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Montag, 7. September 2020, 11:35:20 CEST TommyLike Hu wrote:
Hey, guys: I am an infrastructure member from openEuler community (openEuler is an open source, free Linux distribution platform. The platform provides an open community for global developers to build an open, diversified, and architecture-inclusive software ecosystem. [1]), we use OBS for our distribution packaging and releasing. since our worker doesn't utilize any virtualization technologies (at least in worker service, only chroot is used). we are wondering whether we could containerize our obs worker into kubernetes clusters. Within several hours of research, I found there is a guide [2] which utilizes the kubernetes device plugin and announces it's alpha implementation. I reckon this might be a correct direction but still have some questions. 1. Are there any known issues or disadvantages running an obs worker in a container?
you can run them in a container, but you most likely want to run a KVM build inside it. At least if you build untrusted code and/or want to ensure that the right kernel is used for the build target.
2. all of the container images are tagged with unstable (registry.opensuse.org/obs/server/unstable/container/leap151/containers/openbuildservice/backend for instance), do we have any official docker images? and where can I get the dockerfile of the image?
WIP, but I am happy to hear feedback about this one: # podman pull registry.opensuse.org/home/adriansuse/branches/opensuse/templates/images/15.2/containers/osc # podman run -ti --device /dev/kvm:/dev/kvm $IMAGE_ID that container is in first place for osc, but you should be able to install obs-worker inside as well and run it in kvm mode.
3. if we only containerize our obs worker that means we need to route requests from backend back to obs worker via nodeport, therefore we need to update the ip address of the worker to node's. Is there a way to achieve this?
the worker has to register with the address where it is reachable atm. -- Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mon, Sep 7, 2020 at 5:49 PM Adrian Schröter <adrian@suse.de> wrote:
On Montag, 7. September 2020, 11:35:20 CEST TommyLike Hu wrote:
Hey, guys: I am an infrastructure member from openEuler community (openEuler is an open source, free Linux distribution platform. The platform provides an open community for global developers to build an open, diversified, and architecture-inclusive software ecosystem. [1]), we use OBS for our distribution packaging and releasing. since our worker doesn't utilize any virtualization technologies (at least in worker service, only chroot is used). we are wondering whether we could containerize our obs worker into kubernetes clusters. Within several hours of research, I found there is a guide [2] which utilizes the kubernetes device plugin and announces it's alpha implementation. I reckon this might be a correct direction but still have some questions. 1. Are there any known issues or disadvantages running an obs worker in a container?
you can run them in a container, but you most likely want to run a KVM build inside it. At least if you build untrusted code and/or want to ensure that the right kernel is used for the build target.
hmm. you are absolutely right, now if we want to build some package which has some special requirements on kernel, we have to upgrade the kernel on nodes. Considering the case that our kubernetes cluster running on VM, guess we need to think of running worker directly on vm or use kata runtime instead?
2. all of the container images are tagged with unstable (registry.opensuse.org/obs/server/unstable/container/leap151/containers/openbuildservice/backend for instance), do we have any official docker images? and where can I get the dockerfile of the image?
WIP, but I am happy to hear feedback about this one:
# podman pull registry.opensuse.org/home/adriansuse/branches/opensuse/templates/images/15.2/containers/osc # podman run -ti --device /dev/kvm:/dev/kvm $IMAGE_ID
We both need the x86 and aarch64 images, but the provided image can only run x86. it would be great if the dockerfile can be provided, maybe we can help on it.
that container is in first place for osc, but you should be able to install obs-worker inside as well and run it in kvm mode.
3. if we only containerize our obs worker that means we need to route requests from backend back to obs worker via nodeport, therefore we need to update the ip address of the worker to node's. Is there a way to achieve this?
the worker has to register with the address where it is reachable atm.
I checked the source code of bs_worker, it doesn't support overwrite the host IP of workers'?
--
Adrian Schroeter <adrian@suse.de> Build Infrastructure Project Manager
SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany (HRB 247165, AG München), Geschäftsführer: Felix Imendörffer
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (2)
-
Adrian Schröter -
TommyLike Hu