[Bug 586960] New: fate #305945 causes extreme update channel metadata overhead
http://bugzilla.novell.com/show_bug.cgi?id=586960 http://bugzilla.novell.com/show_bug.cgi?id=586960#c0 Summary: fate #305945 causes extreme update channel metadata overhead Classification: openSUSE Product: openSUSE 11.2 Version: Final Platform: Other OS/Version: SLES 11 Status: NEW Severity: Major Priority: P5 - None Component: Kernel AssignedTo: kernel-maintainers@forge.provo.novell.com ReportedBy: mmarek@novell.com QAContact: qa@suse.de CC: tschmidt@novell.com, ihno@novell.com, coolo@novell.com, meissner@novell.com, mge@novell.com, okir@novell.com, ro@novell.com, visnov@novell.com, dmueller@novell.com Found By: --- Blocker: No +++ This bug was initially created as a clone of Bug #582907 (for SLE) +++ while investigating why the metadata of the 11.2 update channel grew, we noticed that every kernel-* flavor now exports around 8000-9000 provides, which add to about 3.5 MB of compressed (!) primary.xml.gz overhead. for opensuse 11.2, this overhead is fixed, however for SLE11-SP1 it will grow overtime, because we never delete old packages from SLE update channel. that means if we do kernel updates for 7 years (roughly 1 kernel update per month), we'll have 210 MB (!) of compressed metadata just for the kernel, which means 8 minutes of download time for an average DSL connection. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=586960 http://bugzilla.novell.com/show_bug.cgi?id=586960#c1 Michal Marek <mmarek@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Priority|P5 - None |P1 - Urgent Status|NEW |ASSIGNED --- Comment #1 from Michal Marek <mmarek@novell.com> 2010-03-10 10:23:49 UTC --- Some data from the original bug: The problem is that gzip doesn't cope well with the duplication because the text only repeats after several hundred kilobytes, it has a dictionary of 64kb. Ideas for the future were: * make primary.xml.gz zsyncable (https://features.opensuse.org/309088) * implement some rotation for primary.xml.gz * use lzma, which compresses the duplicate dependencies just fine (all three options assume that the original primary.xml.gz stays for compatibility reasons). The more urgent thing is what to do with 11.2. In git, I already blacklisted some of the provides, removed provides for kernel-vanilla completely (because nobody is building KMPs for vanilla) and removed the firmware(...) provides, which are not used anywhere. This reduced the number to about 44% (total of 83856 provides for all kernel-{debug,default,desktop,pae,trace,vanilla,xen}{,-base} for both architectures). -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=586960 http://bugzilla.novell.com/show_bug.cgi?id=586960#c Michal Marek <mmarek@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|kernel-maintainers@forge.pr |mmarek@novell.com |ovo.novell.com | -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=586960 http://bugzilla.novell.com/show_bug.cgi?id=586960#c2 --- Comment #2 from Marcus Meissner <meissner@novell.com> 2010-03-10 10:43:26 UTC --- in 11.2 yesterday the following change was deployed that reduces the impact on the download servers: repomd.xml refers now to tagged primary.xml.gz files, so that they are mirrorable. (e.g. it now refers to: 4c3e8fba6e823d3175670d78dbafea3db4cebc04-primary.xml.gz ) libzypp on 11.2 can handle this fortunately. primary.xml.gz are generated and kept too for tools that do not read repomd.xml correctly. HOWEVER. * The sheer size of primary.xml and the generated resolvables however are still a concern. The users will download this huge files on every update anew, it will pass through the converters and the stack. * For 11.2 we cannot move away from the function export due to released KMPs requiring it already. (Or at least we need a transition phase.) -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=586960 http://bugzilla.novell.com/show_bug.cgi?id=586960#c3 Michal Marek <mmarek@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|ASSIGNED |RESOLVED Resolution| |FIXED --- Comment #3 from Michal Marek <mmarek@novell.com> 2010-08-12 12:21:59 UTC --- For 11.2, we have the above workaround. 11.3 has symsets again, in 11.4, we should have zsyncable metadata and a libzypp that supports zsync downloads. -- Configure bugmail: http://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=586960 https://bugzilla.novell.com/show_bug.cgi?id=586960#c4 Michael Schröder <mls@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED CC| |mls@suse.com Resolution|FIXED | --- Comment #4 from Michael Schröder <mls@suse.com> 2011-12-14 17:07:19 UTC --- For 11.4 it currently takes more than 8 seconds just to convert the downloaded xml file to the solv format. At the end of the 11.4 lifetime it'll probably be around 20 seconds. I regret adding the zsync option, updates would be faster without it because we would still use the symset approach. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=586960 https://bugzilla.novell.com/show_bug.cgi?id=586960#c5 Jeff Mahoney <jeffm@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED CC| |jeffm@suse.com Resolution| |WONTFIX --- Comment #5 from Jeff Mahoney <jeffm@suse.com> 2012-08-02 11:57:36 EDT --- With the coming release of openSUSE 12.2, openSUSE kernel developers are focusing their efforts there. Reports against openSUSE 11.4 and prior will not get the attention needed to resolve them before openSUSE 12.2 is release and openSUSE 11.4 becomes unmaintained. Please re-test with openSUSE 12.1 or openSUSE RC2+ and re-open with an updated Product if you still encounter your issue. We apologize for this issue not getting the attention it deserves but we are focusing our resources in the area where they will have the most impact for our users. We're working hard to make openSUSE 12.2 the best openSUSE release yet! -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=586960 https://bugzilla.novell.com/show_bug.cgi?id=586960#c6 Michael Schröder <mls@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Component|Kernel |Kernel Resolution|WONTFIX | AssignedTo|mmarek@suse.com |kernel-maintainers@forge.pr | |ovo.novell.com Product|openSUSE 11.2 |openSUSE 12.2 --- Comment #6 from Michael Schröder <mls@suse.com> 2012-08-03 08:32:51 UTC --- Ok, re-opened against 12.2. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=586960 https://bugzilla.novell.com/show_bug.cgi?id=586960#c Jeff Mahoney <jeffm@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- AssignedTo|kernel-maintainers@forge.pr |mmarek@suse.com |ovo.novell.com | -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=586960 https://bugzilla.novell.com/show_bug.cgi?id=586960#c7 Jeff Mahoney <jeffm@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |RESOLVED Resolution| |WONTFIX --- Comment #7 from Jeff Mahoney <jeffm@suse.com> 2014-08-08 16:28:49 EDT --- This report is against openSUSE 12.2 which is no longer under maintenance. If you are able to reproduce it with openSUSE 13.1 or openSUSE Factory, please re-open and reset the the "Product" field to the appropriate release. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com