[Bug 477690] New: osc fetching binaries really slow
https://bugzilla.novell.com/show_bug.cgi?id=477690 Summary: osc fetching binaries really slow Classification: openSUSE Product: openSUSE.org Version: unspecified Platform: Other OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: BuildService AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: dmueller@novell.com QAContact: adrian@novell.com Found By: --- it looks like in osc 0.112 it can not really cache the ichain cookie anymore. osc co openSUSE:Factory/libqca2 osc build staging x86_64 this has to fetch a lot of build packages from api.opensuse.org, and every fetch takes around 2s to process. looking at osc -H build output, it seems like it does the ichain authentication everytime because it gets everytime the HTTP/401 unauthorized back. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User suse-tux@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c1 Marcus Hüwe <suse-tux@gmx.de> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |suse-tux@gmx.de --- Comment #1 from Marcus Hüwe <suse-tux@gmx.de> 2009-02-24 19:05:05 MST --- Yes it seems that the cookie support was disabled some time ago (probably due to #378421 but this is already resolved). I just enabled it again in r6571. Normally it shouldn't cause any trouble. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User dmueller@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c2 --- Comment #2 from Dirk Mueller <dmueller@novell.com> 2009-02-25 15:58:24 MST --- no, it doesn't seem to be that "easy". api.opensuse.org sets a cookie on ".opensuse.org", that is not sent back when asking api.opensuse.org again. it seems there is a bug in the cookie library here. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User suse-tux@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c3 --- Comment #3 from Marcus Hüwe <suse-tux@gmx.de> 2009-02-25 16:55:42 MST --- (In reply to comment #2)
no, it doesn't seem to be that "easy". api.opensuse.org sets a cookie on ".opensuse.org", that is not sent back when asking api.opensuse.org again it seems there is a bug in the cookie library here.
A new cookie is only send back if you get a 401 and authenticate with your credentials. If you afterwards send a new request to the api the cookie is used and no new cookie is send back. In this case the old cookie is kept and will be used until the cookie has expired. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User suse-tux@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c4 --- Comment #4 from Marcus Hüwe <suse-tux@gmx.de> 2009-02-28 06:33:19 MST --- Ah sorry I mixed something up. You're right while fetching the binaries osc doesn't use a cookie - I'm going to fix this ASAP. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User suse-tux@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c5 --- Comment #5 from Marcus Hüwe <suse-tux@gmx.de> 2009-02-28 08:58:53 MST --- I just committed a fix (see r6614). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User dmueller@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c6 --- Comment #6 from Dirk Mueller <dmueller@novell.com> 2009-03-03 08:23:59 MST --- I can verify that the cookie issue is fixed indeed, thanks a lot! however, download is still horribly slow, so the cookie issue was not the actual cause. it looks like the binary server is just overloaded with that many individual requests. can we add api support for the cpio download to fetch all binaries in one go? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User suse-tux@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c7 --- Comment #7 from Marcus Hüwe <suse-tux@gmx.de> 2009-03-04 18:45:35 MST --- (In reply to comment #6)
it looks like the binary server is just overloaded with that many individual requests. can we add api support for the cpio download to fetch all binaries in one go? This is already possible via the public/ route, e.g.: https://api.opensuse.org/public/build/openSUSE:11.1/standard/i586/_repository?view=cpio&binary=sed&binary=gawk Would this speed up the download process? It shouldn't be too hard to implement it.
-- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 Cyril Hrubis <chrubis@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|bnc-team-screening@forge.pr |poeml@novell.com |ovo.novell.com | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User dmueller@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c8 --- Comment #8 from Dirk Mueller <dmueller@novell.com> 2009-03-16 07:57:56 MST --- I don't know yet, we'd have to try. the cookie fix now seems to work (thanks a lot!), but it didn't speed up the download process :( can you try to implement the cpio variant if you have time, please? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User suse-tux@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c9 --- Comment #9 from Marcus Hüwe <suse-tux@gmx.de> 2009-03-16 13:09:27 MST --- Yes I'll have a look at it within the next few days. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User suse-tux@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c10 --- Comment #10 from Marcus Hüwe <suse-tux@gmx.de> 2009-03-19 12:36:14 MST --- It seems that fetching packages is indeed a bit faster. I just attached a small (ugly!) patch which downloads a cpio archive in case the packages aren't found on d.o.o. If you think it's faster too, I'll make a nicer patch and commit it. (note: in order to test it you need a current svn version of osc >= r6881) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=477690 User suse-tux@gmx.de added comment https://bugzilla.novell.com/show_bug.cgi?id=477690#c11 --- Comment #11 from Marcus Hüwe <suse-tux@gmx.de> 2009-03-19 12:39:26 MST --- Created an attachment (id=280735) --> (https://bugzilla.novell.com/attachment.cgi?id=280735) retrieve packages in a cpio archive -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com