[opensuse] Re: [opensuse-factory] How many seconds does "time rpm -qa | wc" cost it?
On 09/15/2014 07:56 AM, Carlos E. R. wrote:
On 2014-09-15 11:04, 1xx wrote:
Hi all:
How many seconds does "time rpm -qa | wc" cost it in your OS?
About a minute and a half. Not "seconds".
Telcontar:~ # time rpm -qa | wc 6154 6154 240152
real 1m20.028s user 0m2.875s sys 0m1.735s Telcontar:~ # Telcontar:~ # time rpm -qa | wc -l 6154
real 0m2.877s user 0m2.665s sys 0m0.213s Telcontar:~ #
This is on 13.1 on real hardware, with 8 GiB RAM, and a quad core2 cpu (Q9550 @ 2.83GHz), and reasonably good rotating disks, almost idling.
Notice the huge difference between the first and second runs, which proves that the slag in on the disk and database, as Jan Engelhardt says.
Telcontar:~ # l -h /var/lib/rpm/Packages -rw-r--r-- 1 root root 348M Sep 15 13:49 /var/lib/rpm/Packages
Why does it take so long? Here are my results: # time rpm -qa | wc 1806 1806 61731 real 0m0.613s user 0m0.504s sys 0m0.050s This is on a quad core with hyperthreading & 16 GB. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2014-09-15 14:00, James Knott wrote:
On 09/15/2014 07:56 AM, Carlos E. R. wrote:
Telcontar:~ # l -h /var/lib/rpm/Packages -rw-r--r-- 1 root root 348M Sep 15 13:49 /var/lib/rpm/Packages
Why does it take so long? Here are my results:
Because the database is 348M on rotating disk. Again, notice the huge difference on the second run, while the files are still cached: 3 seconds. If you run the query after a yast/zypper run, it runs fast, too. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlQW2CMACgkQtTMYHG2NR9UxIACfTrUTyoU4PipdHuhvf8j1nF6Z +/sAoI3fB+NLCxaIxlatBriOa4JYE2di =Xxj5 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Monday 15 September 2014, Carlos E. R. wrote:
Why does it take so long? Here are my results:
Because the database is 348M on rotating disk. Again, notice the huge difference on the second run, while the files are still cached: 3 seconds.
If you run the query after a yast/zypper run, it runs fast, too.
My system's responses: fred@linux-main:~> time rpm -qa | wc 1462 1462 43882 real 0m8.833s user 0m2.291s sys 0m0.320s fred@linux-main:~> time rpm -qa | wc 1462 1462 43882 real 0m2.808s user 0m2.243s sys 0m0.144s fred@linux-main:~> time rpm -qa | wc 1462 1462 43882 real 0m2.787s user 0m2.252s sys 0m0.120s Commands sent with very little delay. My system is Suse 11.2 on an Athlon 64, so no speed demon so I am left to assume that my database is tiny in comparison to yours. Then I tried it on my Suse 13.1 running on an I5-650 and got this: fred@linux-cube --> time rpm --qa | wc 1967 1967 66646 real 0m1.282s user 0m1.100s sys 0m0.093s fred@linux-cube --> time rpm -qa | wc real 0m1.244s user 0m1.111s sys 0m0.070s Obviously I should move everything over to the cube, eh? But kmail still works on the 11.2 and that is for a different thread. So, Carlos, I don't see much caching on the 13.1 system except for the sys. Certainly not much difference between "real" and "user" like there is on the 11.2 system. I find it odd that the second and third time commands return a very slightly higher value than the first. Is that just saying I don't have much in the way of a database on the cube? Fred -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
El 15/09/14 a las #4, James Knott escribió:
Why does it take so long? Here are my results:
# time rpm -qa | wc 1806 1806 61731
real 0m0.613s user 0m0.504s sys 0m0.050s
This is on a quad core with hyperthreading & 16 GB.
That's why rpmqpack exists. because this operation is slow in rotating media. -- Cristian "I don't know the key to success, but the key to failure is trying to please everybody." -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-09-15 18:03, Cristian Rodríguez wrote:
El 15/09/14 a las #4, James Knott escribió:
Why does it take so long? Here are my results:
That's why rpmqpack exists. because this operation is slow in rotating media.
Yes, it does run quick here. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On 09/15/2014 08:52 PM, Carlos E. R. wrote:
On 2014-09-15 18:03, Cristian Rodríguez wrote:
That's why rpmqpack exists. because this operation is slow in rotating media.
Yes, it does run quick here.
Well, it only reads /var/lib/rpm/Name while "rpm -qa" also reads /var/lib/rpm/Packages ... because the latter command also prints the version numbers. Interestingly, "rpm -qa --qf='${NAME}\n'" - which would output the same as 'rpmqpack' - is still much slower than rpmqpack. BTW: the slowness comes from the ~30000-50000 pread(2) invocations reading from /var/lib/rpm/Packages: $ time strace -c rpm -qa | wc % time seconds usecs/call calls errors syscall ------ ----------- ----------- --------- --------- ---------------- 98.21 0.027663 1 33110 pread 0.39 0.000111 1 116 read 0.37 0.000105 0 1869 write 0.32 0.000089 1 114 mmap 0.30 0.000085 1 117 37 open ... ------ ----------- ----------- --------- --------- ---------------- 100.00 0.028167 39408 47 total 1869 1869 63019 real 0m14.889s user 0m0.902s sys 0m0.847s If that file is cached, then the rpm command is quite okay: $ time rpm -qa | wc 1869 1869 63019 real 0m0.500s user 0m0.476s sys 0m0.041s but when the cache is dropped, then the pread()s eat much time again: $ sync $ echo 3 > /proc/sys/vm/drop_caches $ time rpm -qa | wc 1869 1869 63019 real 0m15.246s user 0m0.610s sys 0m0.194s So the question is if the pread()s could be enhanced/avoided. [above number from this system: i5-4570, 20G RAM, Hitachi HDS72101] Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-09-15 23:10, Bernhard Voelker wrote:
So the question is if the pread()s could be enhanced/avoided.
I suppose that the entire database could be loaded in memory and then processed. This was not feasible 15 years ago, but now it is. Probably there are functions to map an entire file in RAM - I think that "map" is the current name. And it only needs to be loaded for some limited time. Or force the system to cache it by copying it to null... Look, it is as simple as this: cer@Telcontar:~> time cp /var/lib/rpm/Packages /dev/null real 0m3.560s user 0m0.003s sys 0m0.188s cer@Telcontar:~> time cp /var/lib/rpm/Name /dev/null real 0m0.067s user 0m0.000s sys 0m0.001s cer@Telcontar:~> time rpm -qa | wc -l 6154 real 0m3.191s user 0m2.685s sys 0m0.200s cer@Telcontar:~> It runs now in seconds, not minutes. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
El 15/09/14 a las #4, Carlos E. R. escribió:
On 2014-09-15 23:10, Bernhard Voelker wrote:
So the question is if the pread()s could be enhanced/avoided.
I suppose that the entire database could be loaded in memory and then processed. This was not feasible 15 years ago, but now it is. Probably there are functions to map an entire file in RAM
Yes, to do that just use fopen(3) with "m" mode.. or mmap() .. -- Cristian "I don't know the key to success, but the key to failure is trying to please everybody." -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (6)
-
Bernhard Voelker -
Carlos E. R. -
Carlos E. R. -
Cristian Rodríguez -
Fred n Sandy -
James Knott