[opensuse-factory] Optimizing the package manager
Saw this brilliant post from Marcus Meissner about speeding up the package management stack (link: http://marcusmeissner.livejournal.com/13428.html ) I tried this myself, and saw a dramatic improvement. I've used openSUSE 10.3 since Alpha 5 or so, and I guess all the refreshes of factory and KDE4 cluttered the cache files, because when I ran the steps above it pretty much cut my file sizes in half and Yast package management once again ran as fast as when I initially installed 10.3. I also posted it on suseforums.net for feedback, and it seems to be positive. I understand that factory refreshes place a non-standard "load" on the package manager, and that most users of 10.3 Final will not be placing as high a load on the package manager, but if this addresses a flaw that is intrinsic to the way zypper and rpm operate, is it worth maybe creating a standard script via cron job that "cleans" the package manager on a regular basis (weekly, monthly, whatever) ? I'm really, really impressed with the improvements to package management for 10.3, and I think the vast majority of existing users will be as well. I'm just wondering if we can cut down potential problems in the default install, without relying on pointing to a page in the wiki for de-cluttering the package manager after time ? Maybe I'm missing something and this won't wind up being an issue at all, but my own experience and the feedback I've received indicates otherwise. Is this something that will already happen, or is it worth opening a feature request for? Cheers, KV --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Friday 28 September 2007 04:35:22 Kevin Valko wrote:
I'm really, really impressed with the improvements to package management for 10.3, and I think the vast majority of existing users will be as well. I'm just wondering if we can cut down potential problems in the default install, without relying on pointing to a page in the wiki for de-cluttering the package manager after time ?
Nice to see you like the improvements! We are already discussing how to vacuum the zypp database, and also the rpm database fragmentation number found by Marcus are interesting. We will work on it and try to release an online update for 10.3, at least for the zypp part. For rpm we will see. The biggest "?" is about operation time. This cleanup also takes lot of time and is cpu and IO intensive. The right moment to do it is after heavy database writting, that is refresh. Cron job also was suggested as an idea. But consideer the cron job too random about "when" to do it, and also problematic f it start working when you are using YaST or the applet is checking for updates (it is safe, but you will get a lock message). Our current idea is to do it on refresh but using a time threshold. And making it configurable of course. More ideas are welcome. Cheers Duncan --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Duncan Mac-Vicar Prett schreef:
On Friday 28 September 2007 04:35:22 Kevin Valko wrote:
I'm really, really impressed with the improvements to package management for 10.3, and I think the vast majority of existing users will be as well. I'm just wondering if we can cut down potential problems in the default install, without relying on pointing to a page in the wiki for de-cluttering the package manager after time ?
Nice to see you like the improvements!
We are already discussing how to vacuum the zypp database, and also the rpm database fragmentation number found by Marcus are interesting.
We will work on it and try to release an online update for 10.3, at least for the zypp part. For rpm we will see.
The biggest "?" is about operation time. This cleanup also takes lot of time and is cpu and IO intensive. The right moment to do it is after heavy database writting, that is refresh. Cron job also was suggested as an idea. But consideer the cron job too random about "when" to do it, and also problematic f it start working when you are using YaST or the applet is checking for updates (it is safe, but you will get a lock message). Our current idea is to do it on refresh but using a time threshold. And making it configurable of course.
More ideas are welcome.
Cheers Duncan
It is just an idea, but it might be done in the background, when the system is not heavily used, and paused when the sytem is used, or is that not an option? If there could be a limitation to the cpu use of this action, nobody would be anoyed if it happened during normal use...? - -- Have a nice day, M9. Now, is the only time that exists. OS: Linux 2.6.22.5-29-default x86_64 Huidige gebruiker: monkey9@tribal-sfn2 Systeem: openSUSE 10.3 (x86_64) KDE: 3.5.7 "release 71" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4-svn0 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFG/MzUX5/X5X6LpDgRAm17AKCToW2Ce5xav2Ig2xEm9BlNq0XdigCeJ/TF YtKFTFChSysxlWqz16uR3lM= =pMh3 -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
M9. wrote:
It is just an idea, but it might be done in the background, when the system is not heavily used, and paused when the sytem is used, or is that not an option? If there could be a limitation to the cpu use of this action, nobody would be anoyed if it happened during normal use...?
Things done in the background are difficult to see and stop; and killing a database cleanup could be dangerous. Better do it upfront, asking. -- Cheers, Carlos E. R. (from RC1) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
Duncan Mac-Vicar Prett wrote:
On Friday 28 September 2007 04:35:22 Kevin Valko wrote:
I'm really, really impressed with the improvements to package management for 10.3, and I think the vast majority of existing users will be as well. I'm just wondering if we can cut down potential problems in the default install, without relying on pointing to a page in the wiki for de-cluttering the package manager after time ?
Nice to see you like the improvements!
We are already discussing how to vacuum the zypp database, and also the rpm database fragmentation number found by Marcus are interesting.
We will work on it and try to release an online update for 10.3, at least for the zypp part. For rpm we will see.
The biggest "?" is about operation time. This cleanup also takes lot of time and is cpu and IO intensive. The right moment to do it is after heavy database writting, that is refresh. Cron job also was suggested as an idea. But consideer the cron job too random about "when" to do it, and also problematic f it start working when you are using YaST or the applet is checking for updates (it is safe, but you will get a lock message). Our current idea is to do it on refresh but using a time threshold. And making it configurable of course.
More ideas are welcome.
You could just ask whenever the YaST package manage UI comes up. Run some metric that checks the size and/or condition of the zypper database and if it is "too big", put up a dialog box that says "The Package Database needs to be optimize. Do it now? Warning - this could take some time to complete." -- Jonathan Arnold (mailto:jdarnold@buddydog.org) Linux Brain Dump - Linux Notes, HOWTOs and Tutorials: http://www.linuxbraindump.org Daemon Dancing in the Dark, an Open OS weblog: http://freebsd.amazingdev.com/blog/ --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Friday 28 September 2007 15:40:18 Jonathan D. Arnold / Daemon Dancing wrote:
You could just ask whenever the YaST package manage UI comes up. Run some metric that checks the size and/or condition of the zypper database and if it is "too big", put up a dialog box that says "The Package Database needs to be optimize. Do it now? Warning - this could take some time to complete."
Actually, since 3.3.13, sqlite has support for this: 2007 February 13 (3.3.13) * Add a "fragmentation" measurement in the output of sqlite3_analyzer. So I guess we can use this metric and then we don't need to record the last vacuum. -- Duncan Mac-Vicar Prett Novell :: SUSE R&D, Maxfeldstr. 5, 90409 Nürnberg GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Fri, 2007-09-28 at 18:01 +0200, Duncan Mac-Vicar Prett wrote:
On Friday 28 September 2007 15:40:18 Jonathan D. Arnold / Daemon Dancing wrote:
You could just ask whenever the YaST package manage UI comes up. Run some metric that checks the size and/or condition of the zypper database and if it is "too big", put up a dialog box that says "The Package Database needs to be optimize. Do it now? Warning - this could take some time to complete."
Actually, since 3.3.13, sqlite has support for this:
2007 February 13 (3.3.13) * Add a "fragmentation" measurement in the output of sqlite3_analyzer.
So I guess we can use this metric and then we don't need to record the last vacuum.
I tried an Oreck vacuum on my database and it didn't help at all. :-)) Ken --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
The biggest "?" is about operation time. This cleanup also takes lot of time and is cpu and IO intensive. The right moment to do it is after heavy database writting, that is refresh. Cron job also was suggested as an idea. But consideer the cron job too random about "when" to do it, and also problematic f it start working when you are using YaST or the applet is checking for updates (it is safe, but you will get a lock message). Our current idea is to do it on refresh but using a time threshold. And making it configurable of course.
More ideas are welcome.
Is it safe to assume the cleanup intensity is dependent upon the level of fragmentation and time between cleanups? Just a thought, but would it maybe make sense to add the cleanup process as a SuSEconfig function, to run after package installation? Not that we should really strive to make SuSEconfig take even longer than it does already, but users are already conditioned to waiting for that process to run after Yast completes, they'd have a visual indication as to what is occurring, and if it runs regularly then would the cleanup procedure be quicker and less intense? Power users could of course disable it or run it at the time of their choosing, but maybe as a default setting it may not be too much of a load. Cheers, KV --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
On Thu, Sep 27, 2007 at 10:35:22PM -0400, Kevin Valko wrote:
Saw this brilliant post from Marcus Meissner about speeding up the package management stack (link: http://marcusmeissner.livejournal.com/13428.html )
I tried this myself, and saw a dramatic improvement. I've used openSUSE 10.3 since Alpha 5 or so, and I guess all the refreshes of factory and KDE4 cluttered the cache files, because when I ran the steps above it pretty much cut my file sizes in half and Yast package management once again ran as fast as when I initially installed 10.3.
I also posted it on suseforums.net for feedback, and it seems to be positive.
I understand that factory refreshes place a non-standard "load" on the package manager, and that most users of 10.3 Final will not be placing as high a load on the package manager, but if this addresses a flaw that is intrinsic to the way zypper and rpm operate, is it worth maybe creating a standard script via cron job that "cleans" the package manager on a regular basis (weekly, monthly, whatever) ?
Just one side note. My Laptop has 256MB only. A 512MB or 1GB machine would have not that much time difference, just because the files are larger. The machine is also from 2002, newer harddisk might be faster. "rpm --rebuilddb" ... I guess running it once every 1000 package updates is sufficient. THis is however difficult to measure ;) Ciao, Marcus --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-factory+help@opensuse.org
participants (7)
-
Carlos E. R. -
Duncan Mac-Vicar Prett -
Jonathan D. Arnold / Daemon Dancing -
Kenneth Schneider -
Kevin Valko -
M9. -
Marcus Meissner