[opensuse-buildservice] HowTo build with LXC for OBS
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi OBSers, Last week I played with the current obs unstable version and used the LXC build backend. In this course I was stumbling over some problems, so to make it easier for people, I document my findings in this OBS LXC HowTo Hint: to build with LXC without all the OBS magic, you just run osc build --vm-type=lxc on any OBS checkout. Good for testing & debugging. First, when you just install the obs-worker package or the worker-appliance, it lacks the LXC user-space tools, complaing about not finding lxc-create. LXC also needs the special cgroup pseudo-fs mounted to work. So you need to run once as root: zypper -n install lxc mkdir -p /var/lib/lxc /cgroup echo none /cgroup cgroup defaults 2 0 >> /etc/fstab mount /cgroup echo mount /cgroup >> /etc/init.d/boot.local # note: openSUSE's /etc/init.d/boot.cgroup did not help for me To make OBS build with it, you then edit /etc/sysconfig/obs-worker OBS_VM_TYPE="lxc" This allows to build most (>95%) packages alright and if it works for you or if you are reading this after below fix went upstream, you can stop here. However I found some packages failing for two different reasons. One reason is that packages like udev and mdadm contain device nodes and the /usr/lib/build/lxc.conf forbids most operations on devices. This results in failure messages like Preparing packages for installation... mdadm-3.0.3-0.22.4 error: unpacking of archive failed on file /lib/udev/devices/md0;4e311c7f: cpio: mknod failed - Operation not permitted The other problem I encountered is with packages like yast2-core and perl-IO-Tty that run testsuites as part of their build script and complain about openpty failing. To fix both those problems, I needed to add these lines to /usr/lib/build/lxc.conf : # allow to create any device nodes - but not access lxc.cgroup.devices.allow = c *:* m lxc.cgroup.devices.allow = b *:* m # /dev/pts/* lxc.cgroup.devices.allow = c 136:* rw lxc.tty = 1 but since this file would be replaced on next update of the "build" rpm, those need to be added to the package by the maintainer. I also added this line to lxc.conf: # forbid dangerous operations lxc.cap.drop = sys_module sys_boot sys_rawio sys_time net_raw but I am not sure if all of them are needed. Finally I want to thank Dinar for his work on LXC and to Adrian and all the others making OBS as good as it already is. Ciao Bernhard M. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAk4xMI0ACgkQSTYLOx37oWTpVgCgh2UaE9gCjPt7Ysh8nWJiTtgm L6QAoOAx1MrXMgogXHTZQwnG/AurYEJZ =X9g/ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Jul 28, 2011 at 11:49 AM, Bernhard M. Wiedemann <bernhardout@lsmod.de> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi OBSers,
Last week I played with the current obs unstable version and used the LXC build backend. In this course I was stumbling over some problems, so to make it easier for people, I document my findings in this OBS LXC HowTo
Hint: to build with LXC without all the OBS magic, you just run osc build --vm-type=lxc on any OBS checkout. Good for testing & debugging.
First, when you just install the obs-worker package or the worker-appliance, it lacks the LXC user-space tools, complaing about not finding lxc-create. That's could be added to OBS Worker appliance. (Done for ppc appliance)
LXC also needs the special cgroup pseudo-fs mounted to work. So you need to run once as root:
zypper -n install lxc mkdir -p /var/lib/lxc /cgroup echo none /cgroup cgroup defaults 2 0 >> /etc/fstab mount /cgroup echo mount /cgroup >> /etc/init.d/boot.local
# note: openSUSE's /etc/init.d/boot.cgroup did not help for me I'm not yet found a way how to put it in appliance.
To make OBS build with it, you then edit /etc/sysconfig/obs-worker OBS_VM_TYPE="lxc"
This allows to build most (>95%) packages alright and if it works for you or if you are reading this after below fix went upstream, you can stop here.
However I found some packages failing for two different reasons. One reason is that packages like udev and mdadm contain device nodes and the /usr/lib/build/lxc.conf forbids most operations on devices. This results in failure messages like Preparing packages for installation... mdadm-3.0.3-0.22.4 error: unpacking of archive failed on file /lib/udev/devices/md0;4e311c7f: cpio: mknod failed - Operation not permitted
The other problem I encountered is with packages like yast2-core and perl-IO-Tty that run testsuites as part of their build script and complain about openpty failing.
Good catch!
To fix both those problems, I needed to add these lines to /usr/lib/build/lxc.conf :
# allow to create any device nodes - but not access lxc.cgroup.devices.allow = c *:* m lxc.cgroup.devices.allow = b *:* m # /dev/pts/* lxc.cgroup.devices.allow = c 136:* rw lxc.tty = 1
Is this secure?
but since this file would be replaced on next update of the "build" rpm, those need to be added to the package by the maintainer.
I also added this line to lxc.conf: # forbid dangerous operations lxc.cap.drop = sys_module sys_boot sys_rawio sys_time net_raw
but I am not sure if all of them are needed.
Finally I want to thank Dinar for his work on LXC and to Adrian and all the others making OBS as good as it already is.
Ciao Bernhard M. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/
iEYEARECAAYFAk4xMI0ACgkQSTYLOx37oWTpVgCgh2UaE9gCjPt7Ysh8nWJiTtgm L6QAoOAx1MrXMgogXHTZQwnG/AurYEJZ =X9g/ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Jul 28, 2011 at 12:59 PM, Dinar Valeev <dinarv@gmail.com> wrote:
On Thu, Jul 28, 2011 at 11:49 AM, Bernhard M. Wiedemann <bernhardout@lsmod.de> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Hi OBSers,
Last week I played with the current obs unstable version and used the LXC build backend. In this course I was stumbling over some problems, so to make it easier for people, I document my findings in this OBS LXC HowTo
Hint: to build with LXC without all the OBS magic, you just run osc build --vm-type=lxc on any OBS checkout. Good for testing & debugging.
First, when you just install the obs-worker package or the worker-appliance, it lacks the LXC user-space tools, complaing about not finding lxc-create. That's could be added to OBS Worker appliance. (Done for ppc appliance)
LXC also needs the special cgroup pseudo-fs mounted to work. So you need to run once as root:
zypper -n install lxc mkdir -p /var/lib/lxc /cgroup echo none /cgroup cgroup defaults 2 0 >> /etc/fstab mount /cgroup echo mount /cgroup >> /etc/init.d/boot.local
# note: openSUSE's /etc/init.d/boot.cgroup did not help for me I'm not yet found a way how to put it in appliance.
To make OBS build with it, you then edit /etc/sysconfig/obs-worker OBS_VM_TYPE="lxc"
This allows to build most (>95%) packages alright and if it works for you or if you are reading this after below fix went upstream, you can stop here.
However I found some packages failing for two different reasons. One reason is that packages like udev and mdadm contain device nodes and the /usr/lib/build/lxc.conf forbids most operations on devices. This results in failure messages like Preparing packages for installation... mdadm-3.0.3-0.22.4 error: unpacking of archive failed on file /lib/udev/devices/md0;4e311c7f: cpio: mknod failed - Operation not permitted
The other problem I encountered is with packages like yast2-core and perl-IO-Tty that run testsuites as part of their build script and complain about openpty failing.
Good catch!
To fix both those problems, I needed to add these lines to /usr/lib/build/lxc.conf :
# allow to create any device nodes - but not access lxc.cgroup.devices.allow = c *:* m lxc.cgroup.devices.allow = b *:* m # /dev/pts/* lxc.cgroup.devices.allow = c 136:* rw lxc.tty = 1
Is this secure? i beleive this could be very dangerous allowing container to modify all devices you should figure out which devices are commonly needed and decide based on that
but since this file would be replaced on next update of the "build" rpm, those need to be added to the package by the maintainer.
I also added this line to lxc.conf: # forbid dangerous operations lxc.cap.drop = sys_module sys_boot sys_rawio sys_time net_raw
but I am not sure if all of them are needed.
Finally I want to thank Dinar for his work on LXC and to Adrian and all the others making OBS as good as it already is.
Ciao Bernhard M. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/
iEYEARECAAYFAk4xMI0ACgkQSTYLOx37oWTpVgCgh2UaE9gCjPt7Ysh8nWJiTtgm L6QAoOAx1MrXMgogXHTZQwnG/AurYEJZ =X9g/ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/28/2011 11:59 AM, Dinar Valeev wrote:
On Thu, Jul 28, 2011 at 11:49 AM, Bernhard M. Wiedemann <bernhardout@lsmod.de> wrote: Hi OBSers,
Last week I played with the current obs unstable version and used the LXC build backend. In this course I was stumbling over some problems, so to make it easier for people, I document my findings in this OBS LXC HowTo
Hint: to build with LXC without all the OBS magic, you just run osc build --vm-type=lxc on any OBS checkout. Good for testing & debugging.
First, when you just install the obs-worker package or the worker-appliance, it lacks the LXC user-space tools, complaing about not finding lxc-create.
That's could be added to OBS Worker appliance. (Done for ppc appliance)
LXC also needs the special cgroup pseudo-fs mounted to work. So you need to run once as root:
zypper -n install lxc mkdir -p /var/lib/lxc /cgroup echo none /cgroup cgroup defaults 2 0 >> /etc/fstab mount /cgroup echo mount /cgroup >> /etc/init.d/boot.local
# note: openSUSE's /etc/init.d/boot.cgroup did not help for me
I'm not yet found a way how to put it in appliance.
I think, you can add it to your Kiwi's .../root/build-custom script
However I found some packages failing for two different reasons. One reason is that packages like udev and mdadm contain device nodes and the /usr/lib/build/lxc.conf forbids most operations on devices. This results in failure messages like Preparing packages for installation... mdadm-3.0.3-0.22.4 error: unpacking of archive failed on file /lib/udev/devices/md0;4e311c7f: cpio: mknod failed - Operation not permitted
The other problem I encountered is with packages like yast2-core and perl-IO-Tty that run testsuites as part of their build script and complain about openpty failing.
Good catch!
To fix both those problems, I needed to add these lines to /usr/lib/build/lxc.conf :
# allow to create any device nodes - but not access lxc.cgroup.devices.allow = c *:* m lxc.cgroup.devices.allow = b *:* m # /dev/pts/* lxc.cgroup.devices.allow = c 136:* rw lxc.tty = 1
Is this secure?
I understood the lxc config format to have "rw" for read+write access to devices but the top two lines only have the "m" flag to allow only mknod - unluckily man lxc.conf does not tell. The lower two lines _could_ allow access to the host's pseudo terminals. Not sure how dangerous that is. Ciao Bernhard M. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAk4xOw0ACgkQSTYLOx37oWT3pQCeNeLCC/n9NdZXL5OPQZ0J+OkB Gy8AoJNczNtun0/Be17p/3ICFIwuEmjp =0NI4 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/28/2011 12:33 PM, Bernhard M. Wiedemann wrote:
On 07/28/2011 11:59 AM, Dinar Valeev wrote:
/usr/lib/build/lxc.conf :
# allow to create any device nodes - but not access lxc.cgroup.devices.allow = c *:* m lxc.cgroup.devices.allow = b *:* m # /dev/pts/* lxc.cgroup.devices.allow = c 136:* rw lxc.tty = 1
Is this secure?
I understood the lxc config format to have "rw" for read+write access to devices but the top two lines only have the "m" flag to allow only mknod - unluckily man lxc.conf does not tell. The lower two lines _could_ allow access to the host's pseudo terminals. Not sure how dangerous that is.
I did some more researching. The first lines are secure: lxc-start -n build-root -- /bin/mknod /tmp/devnode c 199 199 lxc-start -n build-root -- /bin/cat /tmp/devnode /bin/cat: /tmp/devnode: Operation not permitted the pts devs did get to the host, so could be problematic, but this stopped when I added to lxc.conf: lxc.pts = 1024 but since I am not much into LXC, this might break other things. I also noticed that when building with LXC, the live log stops early after copying packages... reordering... and when you click "Start Refresh" you see that a new log was started at: processing specfile /.build-srcdir/binutils.spec Ciao Bernhard M. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAk42OMMACgkQSTYLOx37oWTZggCg2MTFImB9kG6Uy7nsuyFzWAai YEgAnAgySnMP0kj2JY7rhh+/289mEInd =cecq -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Mon, Aug 1, 2011 at 7:25 AM, Bernhard M. Wiedemann <bernhardout@lsmod.de> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 07/28/2011 12:33 PM, Bernhard M. Wiedemann wrote:
On 07/28/2011 11:59 AM, Dinar Valeev wrote:
/usr/lib/build/lxc.conf :
# allow to create any device nodes - but not access lxc.cgroup.devices.allow = c *:* m lxc.cgroup.devices.allow = b *:* m # /dev/pts/* lxc.cgroup.devices.allow = c 136:* rw lxc.tty = 1
Is this secure?
I understood the lxc config format to have "rw" for read+write access to devices but the top two lines only have the "m" flag to allow only mknod - unluckily man lxc.conf does not tell. The lower two lines _could_ allow access to the host's pseudo terminals. Not sure how dangerous that is.
I did some more researching. The first lines are secure:
lxc-start -n build-root -- /bin/mknod /tmp/devnode c 199 199 lxc-start -n build-root -- /bin/cat /tmp/devnode /bin/cat: /tmp/devnode: Operation not permitted
the pts devs did get to the host, so could be problematic, but this stopped when I added to lxc.conf: lxc.pts = 1024
but since I am not much into LXC, this might break other things.
I also noticed that when building with LXC, the live log stops early after copying packages... reordering...
and when you click "Start Refresh" you see that a new log was started at: processing specfile /.build-srcdir/binutils.spec
That's weird I haven't notice that. AFAIK, log should be collected through screen log. And there is not so much differences for lxc/kvm/chroot. I'll check that.
Ciao Bernhard M. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/
iEYEARECAAYFAk42OMMACgkQSTYLOx37oWTZggCg2MTFImB9kG6Uy7nsuyFzWAai YEgAnAgySnMP0kj2JY7rhh+/289mEInd =cecq -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (3)
-
Bernhard M. Wiedemann -
Dinar Valeev -
Ramez Hanna