[opensuse-factory] Re: [opensuse] rpmdb2solv extremely slow
On Sunday 19 of January 2014 13:58:02 auxsvr wrote:
Hi,
Since last week, running zypper on oS 13.1 is extremely slow, performing a lot of disk I/O. After some investigation, it seems that it removes and creates again the database at /var/cache/zypp/solv/@System every time it tries to do anything related with package management. According to /var/log/zypper.log, earlier (normal) runs of rpmdb2solv returned after 10 s (from "ExternalProgram.cc(start_program):402 pid 4833 launched" to "ExternalProgram.cc(checkStatus):503 Pid 4833 successfully completed"), whereas lately this lasts about 10 minutes, slower by a factor of 60!
Has anyone else seen something like this? Any ideas?
I am seeing this on Factory also. Cheers, Hrvoje
On Sun, Jan 19, 2014 at 03:26:42PM +0100, ??umski wrote:
On Sunday 19 of January 2014 13:58:02 auxsvr wrote:
Since last week, running zypper on oS 13.1 is extremely slow, performing a lot of disk I/O. After some investigation, it seems that it removes and creates again the database at /var/cache/zypp/solv/@System every time it tries to do anything related with package management. According to /var/log/zypper.log, earlier (normal) runs of rpmdb2solv returned after 10 s (from "ExternalProgram.cc(start_program):402 pid 4833 launched" to "ExternalProgram.cc(checkStatus):503 Pid 4833 successfully completed"),
10s is already slow, it should run faster.
whereas lately this lasts about 10 minutes, slower by a factor of 60!
10 min is absurd. Does strace on the rpmdb2solv process give you a clue where it spends the time?
Has anyone else seen something like this? Any ideas?
I am seeing this on Factory also.
The last change to Factory's libsolv was done in December, so it must be something unrelated. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 20 of January 2014 11:21:34 Michael Schroeder wrote: ...
The last change to Factory's libsolv was done in December, so it must be something unrelated.
It is due to libzypp changes. I've built 14.2.0 and that one works fine. I'll try to build 14.2.1 to narrow it down more (and report it ;-) Cheers, Hrvoje
Cheers, Michael.
On Monday 20 January 2014 13:08:21 šumski wrote:
On Monday 20 of January 2014 11:21:34 Michael Schroeder wrote: ...
The last change to Factory's libsolv was done in December, so it must be something unrelated.
It is due to libzypp changes. I've built 14.2.0 and that one works fine. I'll try to build 14.2.1 to narrow it down more (and report it ;-)
Introduced by commit 2eb2261ba68 for 14.2.1: Cleanup orphaned raw and solv caches (bnc#853065) Mea culpa. It accidentally clears the @System solv file as well, that's why rpmdb2solv is called again and again on every request. I'll fix it asap. Sorry. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres SUSE LINUX Products GmbH, Development, ma@suse.de GF:Jeff Hawn,Jennifer Guild,Felix Imendörffer, HRB16746(AG Nürnberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 20 January 2014 16:56:58 Michael Andres wrote:
It is due to libzypp changes. I've built 14.2.0 and that one works fine. I'll try to build 14.2.1 to narrow it down more (and report it ;-)
Introduced by commit 2eb2261ba68 for 14.2.1: Cleanup orphaned raw and solv caches (bnc#853065)
This of course does not explain why rpmdb2solv takes that much time on some systems. It just explains why it is called that often. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres SUSE LINUX Products GmbH, Development, ma@suse.de GF:Jeff Hawn,Jennifer Guild,Felix Imendörffer, HRB16746(AG Nürnberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 20 January 2014 16:56:58 Michael Andres wrote:
Introduced by commit 2eb2261ba68 for 14.2.1: Cleanup orphaned raw and solv caches (bnc#853065)
Fixed in libzypp-14.5.0 -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres SUSE LINUX Products GmbH, Development, ma@suse.de GF:Jeff Hawn,Jennifer Guild,Felix Imendörffer, HRB16746(AG Nürnberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Michael Andres writes:
On Monday 20 January 2014 16:56:58 Michael Andres wrote:
Introduced by commit 2eb2261ba68 for 14.2.1: Cleanup orphaned raw and solv caches (bnc#853065)
Fixed in libzypp-14.5.0
Hmm. Tumbleweed runs 13.8.5 from openSUSE-Updates, but the solv/@System cache is always removed when zypper exits. Is this the expected behaviour or has this change been backported? Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 20 January 2014 20:28:42 Achim Gratz wrote:
Michael Andres writes:
On Monday 20 January 2014 16:56:58 Michael Andres wrote:
Introduced by commit 2eb2261ba68 for 14.2.1: Cleanup orphaned raw and solv caches (bnc#853065)
Fixed in libzypp-14.5.0
Hmm. Tumbleweed runs 13.8.5 from openSUSE-Updates, but the solv/@System cache is always removed when zypper exits. Is this the expected behaviour or has this change been backported?
It's been backported (in 13.8.4); fix in 13.9.0 is on the way. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres SUSE LINUX Products GmbH, Development, ma@suse.de GF:Jeff Hawn,Jennifer Guild,Felix Imendörffer, HRB16746(AG Nürnberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Michael Andres writes:
Hmm. Tumbleweed runs 13.8.5 from openSUSE-Updates, but the solv/@System cache is always removed when zypper exits. Is this the expected behaviour or has this change been backported?
It's been backported (in 13.8.4); fix in 13.9.0 is on the way.
Ah, that explains it. Thank you. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Michael Andres writes:
Hmm. Tumbleweed runs 13.8.5 from openSUSE-Updates, but the solv/@System cache is always removed when zypper exits. Is this the expected behaviour or has this change been backported?
It's been backported (in 13.8.4); fix in 13.9.0 is on the way.
That update has landed today and zypper keeps that cache alive and is fast again. Thank you! Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 20 of January 2014 16:56:58 Michael Andres wrote:
Introduced by commit 2eb2261ba68 for 14.2.1: Cleanup orphaned raw and solv caches (bnc#853065)
Mea culpa. It accidentally clears the @System solv file as well, that's why rpmdb2solv is called again and again on every request.
I'll fix it asap. Sorry.
That was fast :D Thanks! Note that i didn't have it that extreme as Peter, but it was quite noticeable in contrast to usual speed. zypp:Head now works fine here =) Cheers, Hrvoje
On 20/01/14 10:21, Michael Schroeder wrote:
On Sun, Jan 19, 2014 at 03:26:42PM +0100, ??umski wrote:
On Sunday 19 of January 2014 13:58:02 auxsvr wrote:
Since last week, running zypper on oS 13.1 is extremely slow, performing a lot of disk I/O. After some investigation, it seems that it removes and creates again the database at /var/cache/zypp/solv/@System every time it tries to do anything related with package management. According to /var/log/zypper.log, earlier (normal) runs of rpmdb2solv returned after 10 s (from "ExternalProgram.cc(start_program):402 pid 4833 launched" to "ExternalProgram.cc(checkStatus):503 Pid 4833 successfully completed"), 10s is already slow, it should run faster.
whereas lately this lasts about 10 minutes, slower by a factor of 60! 10 min is absurd. Does strace on the rpmdb2solv process give you a clue where it spends the time?
Has anyone else seen something like this? Any ideas? I am seeing this on Factory also. The last change to Factory's libsolv was done in December, so it must be something unrelated.
Cheers, Michael.
rpm -q libzypp libzypp-14.3.0-1.2.x86_64 Installed on 3 boxes but only one is slow using zypper ref and zypper dup. With "strace -s 256 -f rpmdb2solv" it spends around 4 minutes in pread(3, Regards Sid. -- Sid Boyce ... Hamradio License G3VBV, Licensed Private Pilot Emeritus IBM/Amdahl Mainframes and Sun/Fujitsu Servers Tech Support Senior Staff Specialist, Cricket Coach Microsoft Windows Free Zone - Linux used for all Computing Tasks -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 20.01.2014 16:52, schrieb Sid Boyce:
libzypp-14.3.0-1.2.x86_64 Installed on 3 boxes but only one is slow using zypper ref and zypper dup.
With "strace -s 256 -f rpmdb2solv" it spends around 4 minutes in pread(3, Here it takes 0.4s.
What filesystem do you use for /var/lib ? What does filefrag /var/lib/rpm/* look like? None of that should be affected by the libzypp version as it's between rpm and libsolv though, but I have no other theory. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 20 of January 2014 16:57:11 Stephan Kulow wrote:
Am 20.01.2014 16:52, schrieb Sid Boyce:
Installed on 3 boxes but only one is slow using zypper ref and zypper dup.
With "strace -s 256 -f rpmdb2solv" it spends around 4 minutes in pread(3,
Here it takes 0.4s.
What filesystem do you use for /var/lib ? What does filefrag /var/lib/rpm/* look like?
After defragmenting /var/lib/Packages from ~100 to ~40 extents, rpmdb2solv takes 6 s rather than 10 min on an old PC on ext4, oS 13.1. Since the Packages file is ~400 MB and the disks read at ~80MB/s, this is reasonable. So the cause for the delay apparent when running zypper must be the combination of fragmentation (slowness of rpmdb2solv) and the fact that the solv file is always deleted and rebuilt, fixed by Michael Andres.
Greetings, Stephan
Thanks, Peter -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 20/01/14 15:57, Stephan Kulow wrote:
Am 20.01.2014 16:52, schrieb Sid Boyce:
libzypp-14.3.0-1.2.x86_64 Installed on 3 boxes but only one is slow using zypper ref and zypper dup.
With "strace -s 256 -f rpmdb2solv" it spends around 4 minutes in pread(3, Here it takes 0.4s.
What filesystem do you use for /var/lib ? What does filefrag /var/lib/rpm/* look like?
None of that should be affected by the libzypp version as it's between rpm and libsolv though, but I have no other theory.
Greetings, Stephan
It takes ~0.4s on 2 boxes. All 3 use ext4 I also ran "zypper clean -a" to see if that would have any effect. A good box --------------- tindog:~ # filefrag /var/lib/rpm/* /var/lib/rpm/alternatives: 1 extent found /var/lib/rpm/Basenames: 183 extents found /var/lib/rpm/Conflictname: 2 extents found /var/lib/rpm/Dirnames: 83 extents found /var/lib/rpm/Group: 10 extents found /var/lib/rpm/Installtid: 27 extents found /var/lib/rpm/Name: 30 extents found /var/lib/rpm/Obsoletename: 9 extents found /var/lib/rpm/Packages: 125 extents found /var/lib/rpm/Providename: 98 extents found /var/lib/rpm/Requirename: 54 extents found /var/lib/rpm/Sha1header: 52 extents found /var/lib/rpm/Sigmd5: 30 extents found /var/lib/rpm/Triggername: 1 extent found The slow box ------------------- sabre:~ # filefrag /var/lib/rpm/* /var/lib/rpm/alternatives: 1 extent found /var/lib/rpm/Basenames: 252 extents found /var/lib/rpm/Conflictname: 4 extents found /var/lib/rpm/Dirnames: 39 extents found /var/lib/rpm/Group: 11 extents found /var/lib/rpm/Installtid: 22 extents found /var/lib/rpm/Name: 31 extents found /var/lib/rpm/Obsoletename: 9 extents found /var/lib/rpm/Packages: 118 extents found /var/lib/rpm/Providename: 97 extents found /var/lib/rpm/Requirename: 71 extents found /var/lib/rpm/Sha1header: 61 extents found /var/lib/rpm/Sigmd5: 39 extents found /var/lib/rpm/Triggername: 1 extent found Puzzled! as it took that long previously. After a zypper ref and zypper dup and a reboot, it's normal again after weeks of slow operation. Regards Sid. -- Sid Boyce ... Hamradio License G3VBV, Licensed Private Pilot Emeritus IBM/Amdahl Mainframes and Sun/Fujitsu Servers Tech Support Senior Staff Specialist, Cricket Coach Microsoft Windows Free Zone - Linux used for all Computing Tasks -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Am 20.01.2014 18:17, schrieb Sid Boyce:
------------------- sabre:~ # filefrag /var/lib/rpm/* /var/lib/rpm/Basenames: 252 extents found /var/lib/rpm/Packages: 118 extents found /var/lib/rpm/Providename: 97 extents found
Defrag like that: cd /var/lib/rpm cp Basenames Basenames.new && mv Basenames.new Basenames ... same for the other files Of course you don't want do to that while rpm is running. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Monday 2014-01-20 20:02, Stephan Kulow wrote:
Am 20.01.2014 18:17, schrieb Sid Boyce:
------------------- sabre:~ # filefrag /var/lib/rpm/* /var/lib/rpm/Basenames: 252 extents found /var/lib/rpm/Packages: 118 extents found /var/lib/rpm/Providename: 97 extents found
Defrag like that: cd /var/lib/rpm cp Basenames Basenames.new && mv Basenames.new Basenames ... same for the other files
Or you could run e4defrag or xfs_fsr on the files. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (8)
-
Achim Gratz -
auxsvr＠gmail.com -
Jan Engelhardt -
Michael Andres -
Michael Schroeder -
Sid Boyce -
Stephan Kulow -
šumski