[opensuse-buildservice] osc spikes CPU when downloading kernel-source
I've been testing out 0.123 before we deploy it internally and I've noticed a problem when kernel-source is downloaded from our internal server: Once the download reaches 100%, osc freezes and top reports it sitting at near 100% of the cpu. I've narrowed this down to something PackageQuery is doing when reading the rpm.part file for rpm fields (osc/fetch.py:96). Is this performance issue a known bug? -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
i guess, the kernel-source are a big size, no ? 2010/1/11 Luke Imhoff <luke@cray.com>:
I've been testing out 0.123 before we deploy it internally and I've noticed a problem when kernel-source is downloaded from our internal server: Once the download reaches 100%, osc freezes and top reports it sitting at near 100% of the cpu. I've narrowed this down to something PackageQuery is doing when reading the rpm.part file for rpm fields (osc/fetch.py:96). Is this performance issue a known bug?
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- Cordially. Deploy your softwares for all platforms and finally update them in 3 clicks. Try now the OpenSource MonoOSC tool http://monoosc.sourceforge.net/ http://download.opensuse.org/repositories/home:/surfzoid/ http://download.opensuse.org/repositories/home:/surfzoid:/DebianUbuntu/ http://download.opensuse.org/repositories/home:/surfzoid:/DebianUbuntu:/Mono... windows take you more($), Linux give you more!! Political Power cannot be wisdom! Small Eric Quotations of the days: --------------------------------------------------------------------------- I have no special talents. I am only passionately curious -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Yes, but it didn't do this before. The old implemention (0.119) could handle kernel-source without spiking. On Mon, 2010-01-11 at 09:26 -0600, Petit Eric wrote:
i guess, the kernel-source are a big size, no ?
2010/1/11 Luke Imhoff <luke@cray.com>:
I've been testing out 0.123 before we deploy it internally and I've noticed a problem when kernel-source is downloaded from our internal server: Once the download reaches 100%, osc freezes and top reports it sitting at near 100% of the cpu. I've narrowed this down to something PackageQuery is doing when reading the rpm.part file for rpm fields (osc/fetch.py:96). Is this performance issue a known bug?
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- Cordially.
Deploy your softwares for all platforms and finally update them in 3 clicks. Try now the OpenSource MonoOSC tool http://monoosc.sourceforge.net/ http://download.opensuse.org/repositories/home:/surfzoid/ http://download.opensuse.org/repositories/home:/surfzoid:/DebianUbuntu/ http://download.opensuse.org/repositories/home:/surfzoid:/DebianUbuntu:/Mono...
windows take you more($), Linux give you more!! Political Power cannot be wisdom!
Small Eric Quotations of the days: --------------------------------------------------------------------------- I have no special talents. I am only passionately curious
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
By annotating the for loop at osc.util.rpmquery:85 I was able to determine that only a select few of the headers are causing a problem. Of the 63 headers, these 9 are above a second in parsing: Header 23 Tag 1035 (RPMTAG_FILEMD5S) Elapsed 0:03:10.620126 Header 24 Tag 1036 (RPMTAG_FILELINKTOS) Elapsed 0:01:55.322508 Header 26 Tag 1039 (RPMTAG_FILEUSERNAME) Elapsed 0:01:52.061127 Header 27 Tag 1040 (RPMTAG_FILEGROUPNAME) Elapsed 0:01:44.157788 Header 36 Tag 1081 (RPMTAG_CHANGELOGNAME) Elapsed 0:00:08.699440 Header 37 Tag 1082 (RPMTAG_CHANGELOGTEXT) Elapsed 0:00:38.645766 Header 44 Tag 1097 (RPMTAG_FILELANGS) Elapsed 0:00:39.853684 Header 51 Tag 1117 (RPMTAG_BASENAMES) Elapsed 0:00:39.473591 Header 52 Tag 1118 (RPMTAG_DIRNAMES) Elapsed 0:00:15.796838 On Mon, 2010-01-11 at 09:30 -0600, Luke Imhoff wrote:
Yes, but it didn't do this before. The old implemention (0.119) could handle kernel-source without spiking.
On Mon, 2010-01-11 at 09:26 -0600, Petit Eric wrote:
i guess, the kernel-source are a big size, no ?
2010/1/11 Luke Imhoff <luke@cray.com>:
I've been testing out 0.123 before we deploy it internally and I've noticed a problem when kernel-source is downloaded from our internal server: Once the download reaches 100%, osc freezes and top reports it sitting at near 100% of the cpu. I've narrowed this down to something PackageQuery is doing when reading the rpm.part file for rpm fields (osc/fetch.py:96). Is this performance issue a known bug?
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- Cordially.
Deploy your softwares for all platforms and finally update them in 3 clicks. Try now the OpenSource MonoOSC tool http://monoosc.sourceforge.net/ http://download.opensuse.org/repositories/home:/surfzoid/ http://download.opensuse.org/repositories/home:/surfzoid:/DebianUbuntu/ http://download.opensuse.org/repositories/home:/surfzoid:/DebianUbuntu:/Mono...
windows take you more($), Linux give you more!! Political Power cannot be wisdom!
Small Eric Quotations of the days: --------------------------------------------------------------------------- I have no special talents. I am only passionately curious
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2010-01-11 09:20:27 -0600, Luke Imhoff wrote:
I've been testing out 0.123 before we deploy it internally and I've noticed a problem when kernel-source is downloaded from our internal server: Once the download reaches 100%, osc freezes and top reports it sitting at near 100% of the cpu. I've narrowed this down to something PackageQuery is doing when reading the rpm.part file for rpm fields (osc/fetch.py:96). Is this performance issue a known bug?
By default rpmquery.py reads all tags + data which can take some time (e.g. if you have lots of files in the package). So we just have to limit the read() method to some default tags like name, release, arch etc. if no other tags are needed/specified. I'm going to fix this ASAP. Thanks for the report! Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Tags that are used: {"name" : 1000, "version" : 1001, "release" : 1002, "epoch" : 1003, "arch" : 1002, "summary" : 1004, "description" : 1005, "distribution" : 1010, "provide" : {"name" : 1047, "flags" : 1112, "version" 1113} "require" : {"name" : 1049, "flags" : 1048, "version" 1050} "url" : 1020} And then these 3 that I couldn't find the names: 1044 1051 1052 On Mon, 2010-01-11 at 12:51 -0600, Marcus Hüwe wrote:
On 2010-01-11 09:20:27 -0600, Luke Imhoff wrote:
I've been testing out 0.123 before we deploy it internally and I've noticed a problem when kernel-source is downloaded from our internal server: Once the download reaches 100%, osc freezes and top reports it sitting at near 100% of the cpu. I've narrowed this down to something PackageQuery is doing when reading the rpm.part file for rpm fields (osc/fetch.py:96). Is this performance issue a known bug?
By default rpmquery.py reads all tags + data which can take some time (e.g. if you have lots of files in the package). So we just have to limit the read() method to some default tags like name, release, arch etc. if no other tags are needed/specified. I'm going to fix this ASAP. Thanks for the report!
Marcus
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2010-01-11 19:51:22 +0100, Marcus Hüwe wrote:
On 2010-01-11 09:20:27 -0600, Luke Imhoff wrote:
I've been testing out 0.123 before we deploy it internally and I've noticed a problem when kernel-source is downloaded from our internal server: Once the download reaches 100%, osc freezes and top reports it sitting at near 100% of the cpu. I've narrowed this down to something PackageQuery is doing when reading the rpm.part file for rpm fields (osc/fetch.py:96). Is this performance issue a known bug?
By default rpmquery.py reads all tags + data which can take some time (e.g. if you have lots of files in the package). So we just have to limit the read() method to some default tags like name, release, arch etc. if no other tags are needed/specified. I'm going to fix this ASAP. Thanks for the report!
I just committed a fix for it to git master. Marcus -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
participants (4)
-
Luke Imhoff -
Marcus Hüwe -
Michael Schroeder -
Petit Eric