[opensuse-packaging] Image based installation
Hi, There were some concerns what's behind "Image based installation" that you may hear related to Alpha3+, so I would like to explain shortly what it means and how it may affect bug reports you see. The whole story is a bit longer, but basically Lukas Ocilka, Steffen Winterfeld and me had mexican dinner in Prague downtown - and it turned out to be a productive meeting :) The idea was: installing a live system is about twice as fast as an rpm based install and there has to be a way to use that fact for a "normal" DVD. So we decided it has to be simple to install a basic system from tar and then treat the installed system as if we would do an update. Jiri Srain and Lukas prototyped that system within yast during hackweek II and I spent all easter trying to get a working imaging solution. So that's what we use now in Alpha3+ - and saving roughly 50% installation time compared to 10.3 (http://www.kdedevelopers.org/node/3385). This is done by the package openSUSE-images. It takes 4 lists of packages (patterns enhanced_base, x11, gnome and kde for the very architecture) and installs them with zypper in a chroot. After that it splits out the meta data and the common parts and generates compressed tar files that end up on the DVD. Yast will see a description file on what pattern selection, what set of tar files to unpack and will do so. This takes a fraction of the time it would take with rpm, still after that you have a correctly installed system. The only difference noticed so far is the install time in rpm header (which is frozen at the time openSUSE-images built). So how does this affect you as packager? If your package is not in the default patterns: not at all. If it is, there are special rules your package has to follow not to break this optimization. Most importantly: %post scripts should not create host specific configurations that can't be updated later. This rule already applies as we provide live CDs, so no change. But important to remember. Other than that? I can't think of any, but the method is pretty new and I'm not sure it was tried before. If you have further questions, feel free to contact me. Greetings, Stephan --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hello, On Apr 9 21:05 Stephan Kulow wrote (shortened):
%post scripts should not create host specific configurations that can't be updated later.
What does this mean? Could you give an example what to avoid and what is allowed? Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany AG Nuernberg, HRB 16746, GF: Markus Rex --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Thursday 10 April 2008 09:34:17 wrote Johannes Meixner:
Hello,
On Apr 9 21:05 Stephan Kulow wrote (shortened):
%post scripts should not create host specific configurations that can't be updated later.
What does this mean? Could you give an example what to avoid and what is allowed?
For example reading /etc/HOSTNAME and writing it to some config via %pre or %post scripts is not allowed. Actually such kind of actions should be never done, since you do not know, who is using your package for a mass deployment (via images or other mechanisms). bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Donnerstag 10 April 2008 schrieb Johannes Meixner:
Hello,
On Apr 9 21:05 Stephan Kulow wrote (shortened):
%post scripts should not create host specific configurations that can't be updated later.
What does this mean? Could you give an example what to avoid and what is allowed?
allowed: changing the files in the file system not allowed: reading from other file systems as proc/dev/sys or other means of getting informations about the computer/kernel/network Does this clarify it or do you have some cases where it's not clear on what side it is? Greetings, Stephan -- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hello, On Apr 10 10:55 Stephan Kulow wrote (shortened):
Am Donnerstag 10 April 2008 schrieb Johannes Meixner:
On Apr 9 21:05 Stephan Kulow wrote (shortened):
%post scripts should not create host specific configurations that can't be updated later.
What does this mean? Could you give an example what to avoid and what is allowed?
allowed: changing the files in the file system
As far as I see this is sufficient for my packages.
not allowed: reading from other file systems as proc/dev/sys or other means of getting informations about the computer/kernel/network
Perhaps there are hidden ways to read proc/dev/sys e.g. when calling certain commands which do it implicitly so that RPM scripts look clean but actually aren't. Could you provide background information what is going on behind the scenes (i.e. the difference to using plain rpm) so that I would better understand what the root cause is why it is not allowed. The problem is that I can only test my packages (with reasonable effort) by using plain rpm for installation and update (i.e. I build it for my workstation and then install or update it using plain rpm). If it works well this way, I consider my package to be o.k. For example if I got a bug report about whatever failure and I cannot reproduce it, neither I nor the reporter may have the idea that the root cause could be a subtle difference during system installation compared to plain rpm. In the worst case we might end up with Windows-like behaviour: "If a package doesn't work, first of all try to re-install it". Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany AG Nuernberg, HRB 16746, GF: Markus Rex --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Am Donnerstag 10 April 2008 schrieb Johannes Meixner:
Could you provide background information what is going on behind the scenes (i.e. the difference to using plain rpm) so that I would better understand what the root cause is why it is not allowed.
It's an image. Perhaps the live cd we provide for quite some time already makes it more clear to you. We install rpms at time X, then make a CD out of it and ship it to the user. The user boots the live cd and if then this live system would always preconfigure print queue Meixners because it happened to be in the system that created the live cd, this would be pretty bad. And this is exactly what I'm talking about. And no, I can't give you a way to check your packages, you will have to read/write the %post scripts in a way that they do not assume anything. It's just not their job. Greetings, Stephan -- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Hello, On Apr 10 11:31 Stephan Kulow wrote (shortened):
We install rpms at time X, then make a CD out of it and ship it to the user.
I.e. the RPM scripts run at time X on a machine Y (which is not the end-user's machine). It is just the same kind of problem as what happens during RPM build time in %install in the build system versus what happens during RPM installation time on the end-user's machine. Thanks for the info, now it is clear! Kind Regards Johannes Meixner -- SUSE LINUX Products GmbH, Maxfeldstrasse 5, 90409 Nuernberg, Germany AG Nuernberg, HRB 16746, GF: Markus Rex --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Stephan Kulow wrote:
Yast will see a description file on what pattern selection, what set of tar files to unpack and will do so. This takes a fraction of the time it would take with rpm, still after that you have a correctly installed system. The only difference noticed so far is the install time in rpm header (which is frozen at the time openSUSE-images built).
Are a few minutes installation time really that important? Sure it's 50% according to your numbers but it's still minutes. Has anyone actually tried to optimize the traditional installation method? Like preloading rpms or reorder them on the medium to avoid seeks on the cdrom?
If you have further questions, feel free to contact me.
How does this mechanism add the rpms of the second and following images to the rpm database? cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.de/ SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Dne Thursday 10 of April 2008 10:39:45 Ludwig Nussel napsal(a):
Stephan Kulow wrote:
Yast will see a description file on what pattern selection, what set of tar files to unpack and will do so. This takes a fraction of the time it would take with rpm, still after that you have a correctly installed system. The only difference noticed so far is the install time in rpm header (which is frozen at the time openSUSE-images built).
Are a few minutes installation time really that important? Sure it's 50% according to your numbers but it's still minutes. Has anyone actually tried to optimize the traditional installation method? Like preloading rpms or reorder them on the medium to avoid seeks on the cdrom?
If you have further questions, feel free to contact me.
How does this mechanism add the rpms of the second and following images to the rpm database?
There is a special image, which contains the RPM database (and other software management data), the list of images is bound with this one. You cannot combine images in any way, only in the way it was intended by the creator of images. The advantage compared to having only one image is that you can share some data between images (KDE vs. GNOME) to save media space. Jiri -- Regards, Jiri Srain YaST Team Leader --------------------------------------------------------------------- SUSE LINUX, s.r.o. e-mail: jsrain@suse.cz Lihovarska 1060/12 tel: +420 284 028 959 190 00 Praha 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz
Am Donnerstag 10 April 2008 schrieb Ludwig Nussel:
Stephan Kulow wrote:
Yast will see a description file on what pattern selection, what set of tar files to unpack and will do so. This takes a fraction of the time it would take with rpm, still after that you have a correctly installed system. The only difference noticed so far is the install time in rpm header (which is frozen at the time openSUSE-images built).
Are a few minutes installation time really that important? Sure it's 50% according to your numbers but it's still minutes. Oh, if the unit is your problem: it saves half an hour.
Greetings, Stephan -- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
On Thu, Apr 10, Stephan Kulow wrote:
Am Donnerstag 10 April 2008 schrieb Ludwig Nussel:
Stephan Kulow wrote:
Yast will see a description file on what pattern selection, what set of tar files to unpack and will do so. This takes a fraction of the time it would take with rpm, still after that you have a correctly installed system. The only difference noticed so far is the install time in rpm header (which is frozen at the time openSUSE-images built).
Are a few minutes installation time really that important? Sure it's 50% according to your numbers but it's still minutes. Oh, if the unit is your problem: it saves half an hour.
Ok, on the weekend I installed openSUSE 10.3 (default KDE installation) in less than half an hour on an old ThinkPad T40. Now we improve the installation time to save 30 minutes. Does this now mean, that if I start an installation, the system is already completly installed since some minutes? Or did we double the necessary time to install from 10.3 to 11.0? In that case, we should find the reason and solve that problem. Thorsten -- Thorsten Kukuk, Project Manager/Release Manager SLES SUSE LINUX Products GmbH, Maxfeldstr. 5, D-90409 Nuernberg GF: Markus Rex, HRB 16746 (AG Nuernberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
Ludwig Nussel napsal(a):
Stephan Kulow wrote:
Yast will see a description file on what pattern selection, what set of tar files to unpack and will do so. This takes a fraction of the time it would take with rpm, still after that you have a correctly installed system. The only difference noticed so far is the install time in rpm header (which is frozen at the time openSUSE-images built).
Are a few minutes installation time really that important? Sure it's 50% according to your numbers but it's still minutes. Has anyone actually tried to optimize the traditional installation method? Like preloading rpms or reorder them on the medium to avoid seeks on the cdrom?
Reordering RPMs should be done according to dependencies but libzypp solver can 'randomize' them a bit ;) There are some more important features that would make the traditional installation method faster: * Downloading and Installation in more threads (installation of RPM doesn't block downloading) * Installing more RPMs at once. (a lot of small RPMs have to initialize RPM database, lock it, unlock it..., and again...) And frankly, even if the installation doesn't seem to be so important as we should not install so often, every speed-up counts :D L.
On Thu, 10 Apr 2008 at 11:05, Lukas Ocilka wrote:
* Downloading and Installation in more threads (installation of RPM doesn't block downloading)
I've suggested that years ago, and since it gets postponded from release to release, first in Bugzilla, then in FATE: https://bugzilla.novell.com/show_bug.cgi?id=60844 cu Reinhard --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-packaging+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-packaging+help@opensuse.org
participants (8)
-
Adrian Schröter -
Jiri Srain -
Johannes Meixner -
Ludwig Nussel -
Lukas Ocilka -
Reinhard Max -
Stephan Kulow -
Thorsten Kukuk