[New: openFATE 308837] performance: jail creation ...
Feature added by: Michael Meeks (michael_meeks) Feature #308837, revision 1 Title: performance: jail creation ... Buildservice: New Priority Requester: Mandatory Requested by: Michael Meeks (michael_meeks) Description: For leaf GNOME packages - partly due to the granular packaging (probably) it takes a long time - sometimes 30%+ of build time, just to populate the jail from clean. Furthermore, the distinction between 'clean' and 'unclean' jails is highly problematic - with local builds often failing mid-way due to jail un-cleanness. It would be ideal to make jail construction lightening fast, such that cleaning is never necessary. This should be possible with a stackable FUSE filing system, keyed on the RPMs that are installed: ultimately, given a set of packages to install - we should end up with a know-able / cache-able state. Better - this should allow multiple local builds to be performed, concurrently on the local host without the excessive resourcing (for leaf packages) of installing two massive, duplicate chroot jails, with the problems of shared-nothing in memory. -- openSUSE Feature: https://features.opensuse.org/308837
Feature changed by: Jan Engelhardt (jengelh) Feature #308837, revision 2 Title: performance: jail creation ... Buildservice: New Priority Requester: Mandatory Requested by: Michael Meeks (michael_meeks) Description: For leaf GNOME packages - partly due to the granular packaging (probably) it takes a long time - sometimes 30%+ of build time, just to populate the jail from clean. Furthermore, the distinction between 'clean' and 'unclean' jails is highly problematic - with local builds often failing mid-way due to jail un-cleanness. It would be ideal to make jail construction lightening fast, such that cleaning is never necessary. This should be possible with a stackable FUSE filing system, keyed on the RPMs that are installed: ultimately, given a set of packages to install - we should end up with a know-able / cache-able state. Better - this should allow multiple local builds to be performed, concurrently on the local host without the excessive resourcing (for leaf packages) of installing two massive, duplicate chroot jails, with the problems of shared-nothing in memory. + Discussion: + #1: Jan Engelhardt (jengelh) (2010-01-19 01:50:16) + Removing --clean from the argument list to build is a nice shortcut. + Additionally, you want to make sure to use an fs that is reasonably + fast for lots of metadata updates, such as it happens with buildroot + initialization. /tmp, /var/tmp (latter could be a symlink) could be + made btrfs. + + I am not convinced a stackable filesystem helps much, as each buildroot + may have package sets that can almost totally differ. Beyond the + Preinstalls, I do not expect any real package sharing. -- openSUSE Feature: https://features.opensuse.org/308837
Feature changed by: Michael Meeks (michael_meeks) Feature #308837, revision 3 Title: performance: jail creation ... Buildservice: New Priority Requester: Mandatory Requested by: Michael Meeks (michael_meeks) Description: For leaf GNOME packages - partly due to the granular packaging (probably) it takes a long time - sometimes 30%+ of build time, just to populate the jail from clean. Furthermore, the distinction between 'clean' and 'unclean' jails is highly problematic - with local builds often failing mid-way due to jail un-cleanness. It would be ideal to make jail construction lightening fast, such that cleaning is never necessary. This should be possible with a stackable FUSE filing system, keyed on the RPMs that are installed: ultimately, given a set of packages to install - we should end up with a know-able / cache-able state. Better - this should allow multiple local builds to be performed, concurrently on the local host without the excessive resourcing (for leaf packages) of installing two massive, duplicate chroot jails, with the problems of shared-nothing in memory. Discussion: #1: Jan Engelhardt (jengelh) (2010-01-19 01:50:16) Removing --clean from the argument list to build is a nice shortcut. Additionally, you want to make sure to use an fs that is reasonably fast for lots of metadata updates, such as it happens with buildroot initialization. /tmp, /var/tmp (latter could be a symlink) could be made btrfs. I am not convinced a stackable filesystem helps much, as each buildroot may have package sets that can almost totally differ. Beyond the Preinstalls, I do not expect any real package sharing. + #2: Michael Meeks (michael_meeks) (2010-01-19 11:17:49) (reply to #1) + Well - when we get to leaf packages; there is a lot of sharing possible + - at least conceptually. + As an example, the entire GNOME stack - will require all the + dependencies of gtk+, gconf, dbus, X11 - and so on; there is a huge + common set of packages that that will pull in. KDE will be similar I + guess, likewise for Moblin: everything up to clutter-devel :-) Even for + simple stuff - we tend to need a few compilers, perl, cmake, autotools + (etc.) installed to build things. Consider OpenOffice_org package + builds - we have 20+ source packages, all of which require the same + base stack - base-system+gnome+kde+X11+OpenOffice_org-bootstrap. + Clearly the win accumulates to the leaf packages though. + So - sure, while providing a perfect file-system image with the right + packages pre-installed is prolly impossible; having a set of stacked + layered images - that include our common development stacks, should + save a large number of individual package installs - all of which take + significant time & space. Naturally, a few more packages will be + required to be installed on top, but we can cut out 90% of the package + installation phase [ I hope ]. That should give enough file-system + space to easily allow multiple parallel package builds on the same + laptop as a bonus ;-) -- openSUSE Feature: https://features.opensuse.org/308837
Feature changed by: Adrian Schröter (adrianSuSE) Feature #308837, revision 4 Title: performance: jail creation ... Buildservice: New Priority Requester: Mandatory Requested by: Michael Meeks (michael_meeks) Description: For leaf GNOME packages - partly due to the granular packaging (probably) it takes a long time - sometimes 30%+ of build time, just to populate the jail from clean. Furthermore, the distinction between 'clean' and 'unclean' jails is highly problematic - with local builds often failing mid-way due to jail un-cleanness. It would be ideal to make jail construction lightening fast, such that cleaning is never necessary. This should be possible with a stackable FUSE filing system, keyed on the RPMs that are installed: ultimately, given a set of packages to install - we should end up with a know-able / cache-able state. Better - this should allow multiple local builds to be performed, concurrently on the local host without the excessive resourcing (for leaf packages) of installing two massive, duplicate chroot jails, with the problems of shared-nothing in memory. Discussion: #1: Jan Engelhardt (jengelh) (2010-01-19 01:50:16) Removing --clean from the argument list to build is a nice shortcut. Additionally, you want to make sure to use an fs that is reasonably fast for lots of metadata updates, such as it happens with buildroot initialization. /tmp, /var/tmp (latter could be a symlink) could be made btrfs. - I am not convinced a stackable filesystem helps much, as each buildroot may have package sets that can almost totally differ. Beyond the Preinstalls, I do not expect any real package sharing. #2: Michael Meeks (michael_meeks) (2010-01-19 11:17:49) (reply to #1) Well - when we get to leaf packages; there is a lot of sharing possible - at least conceptually. As an example, the entire GNOME stack - will require all the dependencies of gtk+, gconf, dbus, X11 - and so on; there is a huge common set of packages that that will pull in. KDE will be similar I guess, likewise for Moblin: everything up to clutter-devel :-) Even for simple stuff - we tend to need a few compilers, perl, cmake, autotools (etc.) installed to build things. Consider OpenOffice_org package builds - we have 20+ source packages, all of which require the same base stack - base-system+gnome+kde+X11+OpenOffice_org-bootstrap. Clearly the win accumulates to the leaf packages though. So - sure, while providing a perfect file-system image with the right packages pre-installed is prolly impossible; having a set of stacked layered images - that include our common development stacks, should save a large number of individual package installs - all of which take significant time & space. Naturally, a few more packages will be required to be installed on top, but we can cut out 90% of the package installation phase [ I hope ]. That should give enough file-system space to easily allow multiple parallel package builds on the same laptop as a bonus ;-) + #3: Adrian Schröter (adriansuse) (2010-01-21 12:11:37) + This is not going to work on the server side easily, since we have too + many different build projects in parallel in most cases. But could help + in some cases. However, we have some dispatcher/worker changes in the + queue which needs to be done bfore. + On the client side, I think one need a concept to save build roots and + use them again. + you can do this manually by using the repostory name as part of your + build root. + Is this solving your issue ? -- openSUSE Feature: https://features.opensuse.org/308837
Feature changed by: Adrian Schröter (adrianSuSE) Feature #308837, revision 5 Title: performance: jail creation ... Buildservice: New Priority Requester: Mandatory Requested by: Michael Meeks (michael_meeks) Description: For leaf GNOME packages - partly due to the granular packaging (probably) it takes a long time - sometimes 30%+ of build time, just to populate the jail from clean. Furthermore, the distinction between 'clean' and 'unclean' jails is highly problematic - with local builds often failing mid-way due to jail un-cleanness. It would be ideal to make jail construction lightening fast, such that cleaning is never necessary. This should be possible with a stackable FUSE filing system, keyed on the RPMs that are installed: ultimately, given a set of packages to install - we should end up with a know-able / cache-able state. Better - this should allow multiple local builds to be performed, concurrently on the local host without the excessive resourcing (for leaf packages) of installing two massive, duplicate chroot jails, with the problems of shared-nothing in memory. Discussion: #1: Jan Engelhardt (jengelh) (2010-01-19 01:50:16) Removing --clean from the argument list to build is a nice shortcut. Additionally, you want to make sure to use an fs that is reasonably fast for lots of metadata updates, such as it happens with buildroot initialization. /tmp, /var/tmp (latter could be a symlink) could be made btrfs. I am not convinced a stackable filesystem helps much, as each buildroot may have package sets that can almost totally differ. Beyond the Preinstalls, I do not expect any real package sharing. #2: Michael Meeks (michael_meeks) (2010-01-19 11:17:49) (reply to #1) Well - when we get to leaf packages; there is a lot of sharing possible - at least conceptually. As an example, the entire GNOME stack - will require all the dependencies of gtk+, gconf, dbus, X11 - and so on; there is a huge common set of packages that that will pull in. KDE will be similar I guess, likewise for Moblin: everything up to clutter-devel :-) Even for simple stuff - we tend to need a few compilers, perl, cmake, autotools (etc.) installed to build things. Consider OpenOffice_org package builds - we have 20+ source packages, all of which require the same base stack - base-system+gnome+kde+X11+OpenOffice_org-bootstrap. Clearly the win accumulates to the leaf packages though. So - sure, while providing a perfect file-system image with the right packages pre-installed is prolly impossible; having a set of stacked layered images - that include our common development stacks, should save a large number of individual package installs - all of which take significant time & space. Naturally, a few more packages will be required to be installed on top, but we can cut out 90% of the package installation phase [ I hope ]. That should give enough file-system space to easily allow multiple parallel package builds on the same laptop as a bonus ;-) #3: Adrian Schröter (adriansuse) (2010-01-21 12:11:37) This is not going to work on the server side easily, since we have too many different build projects in parallel in most cases. But could help in some cases. However, we have some dispatcher/worker changes in the queue which needs to be done bfore. On the client side, I think one need a concept to save build roots and use them again. you can do this manually by using the repostory name as part of your build root. Is this solving your issue ? + #4: Adrian Schröter (adriansuse) (2010-01-22 11:09:28) + is it (just for request information from michael) ? -- openSUSE Feature: https://features.opensuse.org/308837
Feature changed by: Michael Meeks (michael_meeks) Feature #308837, revision 6 Title: performance: jail creation ... Buildservice: New Priority Requester: Mandatory Requested by: Michael Meeks (michael_meeks) Description: For leaf GNOME packages - partly due to the granular packaging (probably) it takes a long time - sometimes 30%+ of build time, just to populate the jail from clean. Furthermore, the distinction between 'clean' and 'unclean' jails is highly problematic - with local builds often failing mid-way due to jail un-cleanness. It would be ideal to make jail construction lightening fast, such that cleaning is never necessary. This should be possible with a stackable FUSE filing system, keyed on the RPMs that are installed: ultimately, given a set of packages to install - we should end up with a know-able / cache-able state. Better - this should allow multiple local builds to be performed, concurrently on the local host without the excessive resourcing (for leaf packages) of installing two massive, duplicate chroot jails, with the problems of shared-nothing in memory. Discussion: #1: Jan Engelhardt (jengelh) (2010-01-19 01:50:16) Removing --clean from the argument list to build is a nice shortcut. Additionally, you want to make sure to use an fs that is reasonably fast for lots of metadata updates, such as it happens with buildroot initialization. /tmp, /var/tmp (latter could be a symlink) could be made btrfs. I am not convinced a stackable filesystem helps much, as each buildroot may have package sets that can almost totally differ. Beyond the Preinstalls, I do not expect any real package sharing. #2: Michael Meeks (michael_meeks) (2010-01-19 11:17:49) (reply to #1) Well - when we get to leaf packages; there is a lot of sharing possible - at least conceptually. As an example, the entire GNOME stack - will require all the dependencies of gtk+, gconf, dbus, X11 - and so on; there is a huge common set of packages that that will pull in. KDE will be similar I guess, likewise for Moblin: everything up to clutter-devel :-) Even for simple stuff - we tend to need a few compilers, perl, cmake, autotools (etc.) installed to build things. Consider OpenOffice_org package builds - we have 20+ source packages, all of which require the same base stack - base-system+gnome+kde+X11+OpenOffice_org-bootstrap. Clearly the win accumulates to the leaf packages though. So - sure, while providing a perfect file-system image with the right packages pre-installed is prolly impossible; having a set of stacked layered images - that include our common development stacks, should save a large number of individual package installs - all of which take significant time & space. Naturally, a few more packages will be required to be installed on top, but we can cut out 90% of the package installation phase [ I hope ]. That should give enough file-system space to easily allow multiple parallel package builds on the same laptop as a bonus ;-) #3: Adrian Schröter (adriansuse) (2010-01-21 12:11:37) This is not going to work on the server side easily, since we have too many different build projects in parallel in most cases. But could help in some cases. However, we have some dispatcher/worker changes in the queue which needs to be done bfore. On the client side, I think one need a concept to save build roots and use them again. you can do this manually by using the repostory name as part of your build root. Is this solving your issue ? + #5: Michael Meeks (michael_meeks) (2010-01-22 14:17:56) (reply to #3) + On the server side - I agree, there are issues: your scheduler would + need to schedule similar tasks (eg. building kdebase4 and subsequently + preferring to put kdepim4 on the same node. But even with rather un- + related GNOME vs. KDE packages (I suspect) there is some substantial + overlap. Presumably we try to put similar types of build job on the + same node now - [ or do we use no local caching of eg. packages at all + ? ]. Perhaps this is what you mean by dispatcher / worker changes ? :-) + On the client side - my problem is disk space; an intelligent fuse file- + system would be able to share large amounts of data between different + jails, saving hundreds of Mb per jail (for leaf packages) - and that + quite aside from the much improved performance of jail creation. So -no + - whacking each build in it's own directory is not going to solve my + problem: but blow my local disk :-) #4: Adrian Schröter (adriansuse) (2010-01-22 11:09:28) is it (just for request information from michael) ? -- openSUSE Feature: https://features.opensuse.org/308837
participants (1)
-
fate_noreply@suse.de