[Bug 897353] New: db3 full scan slowness - affects rpmdb
https://bugzilla.novell.com/show_bug.cgi?id=897353 https://bugzilla.novell.com/show_bug.cgi?id=897353#c0 Summary: db3 full scan slowness - affects rpmdb Classification: openSUSE Product: openSUSE Factory Version: 201409* Platform: All OS/Version: openSUSE 13.1 Status: NEW Severity: Normal Priority: P5 - None Component: Basesystem AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: klaussfreire@gmail.com QAContact: qa-bugs@suse.de Found By: --- Blocker: --- Created an attachment (id=606898) --> (http://bugzilla.novell.com/attachment.cgi?id=606898) Test scripts User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Firefox/31.0 bsddb3 full scans (those performed by filter-less cursor walking) scan the backing file in random order, causing a much slower scan. This is a problem that affects rpmdb, which uses those kinds of scans while listing packages. Reproducible: Always Steps to Reproduce: Download attached test scripts 1. python bsdcreate.py 2. echo 3 | sudo tee /proc/sys/vm/drop_caches 3. time python bsdtest.py 4. time python bsdtest.py 5. echo 3 | sudo tee /proc/sys/vm/drop_caches 6. time ( cp fruit /dev/null ; python bsdtest.py ) Actual Results: First run (step 3) takes a long time, much longer than second (step 4) and third (step 5), even though step 5 also reads from disk, it does so in sequential order which is much faster Expected Results: First and third run should take about the same time, the scan should be done in physical order to leverage kernel both read-ahead and rotating media's higher pefromance for sequential reads. There's another script, bsdtest_fadvise.py, that issues a posix_fadvise call to preload the file before initiating the scan. This also results in far better performance, and would be an appropriate workaround for "rpm -qa" (which does such a scan). For that to work, python-fadvise must be installed from https://github.com/lamby/python-fadvise As the package database grows, it's becoming important to optimize this. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=897353
https://bugzilla.novell.com/show_bug.cgi?id=897353#c1
Bernhard Wiedemann
https://bugzilla.novell.com/show_bug.cgi?id=897353
https://bugzilla.novell.com/show_bug.cgi?id=897353#c2
Michael Schröder
https://bugzilla.novell.com/show_bug.cgi?id=897353
https://bugzilla.novell.com/show_bug.cgi?id=897353#c3
--- Comment #3 from Claudio Freire
echo 3 | sudo tee /proc/sys/vm/drop_caches ; time ( cat /var/lib/rpm/Packages > /dev/null ; rpm -qa > /dev/null ) 3
real 0m7.481s user 0m1.127s sys 0m0.286s So the difference is 6-fold. Second, this was brought up in the context of etckeeper, which is something that's done during package installation and causes real trouble, see: http://lists.opensuse.org/opensuse-factory/2014-09/msg00223.html Finally, even if you don't care about etckeeper, this is a performance bug from the POV of a bsddb3 user. A sequential scan is supposed to be sequential physically as well, and it should be fixable (the data structure should support it). So, you could do one or more of the following: a) Report it upstream (bsddb3) b) Report it upstream (rpm, they can implement the fadvise workaround, or switch to another bsddb implementation that doesn't have this issue) c) or work around it in everything that does package management -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply@novell.com