[Bug 370054] New: support parallel downloads
https://bugzilla.novell.com/show_bug.cgi?id=370054 User stephan.kleine@googlemail.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c369531 Summary: support parallel downloads Product: openSUSE 11.0 Version: Alpha 2plus Platform: Other OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: libzypp AssignedTo: zypp-maintainers@forge.provo.novell.com ReportedBy: stephan.kleine@googlemail.com QAContact: dmacvicar@novell.com Found By: --- Currently libzypp/yast2/zypper downloads one file after the other instead of downloading multiple files at the same time to maximize the usage of available bandwith like e.g. smart does. This is especially painful if one gets repeatedly directed to a slow mirror during bigger updates and could actually work hand in hand with the support of mirror functionality (as proposed in #369531). Last but not least you could even introduce the ability to split up bigger files (e.g. kernel-source) into several parts and then to download those parts in parallel from different mirrors like it is possible with aria2. Perhaps you can reuse it? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 Stephan Binner <stbinner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Severity|Normal |Enhancement -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User nicekiwi9@gmail.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c1 Ezra Sharp <nicekiwi9@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |nicekiwi9@gmail.com --- Comment #1 from Ezra Sharp <nicekiwi9@gmail.com> 2008-04-17 07:58:00 MST --- very good idea, slow downloads have been the biggest annoyance for me in updating any Linux distros i have tried... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User funtasyspace@yahoo.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c2 --- Comment #2 from Jörg Hermsdorf <funtasyspace@yahoo.com> 2008-04-17 08:50:24 MST --- I think this enhancement is the same as I suggested in Bug 370457. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User linux.learner@gmail.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c3 Jon Rocker <linux.learner@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |linux.learner@gmail.com --- Comment #3 from Jon Rocker <linux.learner@gmail.com> 2008-04-19 21:01:37 MST --- You could even make this configurable via yast. Wither to enable it or not. How many parallel downloads is wanted, with 5 being the default. If you want zypp/YaST to be fast, then this is a must. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User stbinner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c4 Stephan Binner <stbinner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |DUPLICATE --- Comment #4 from Stephan Binner <stbinner@novell.com> 2008-04-20 00:39:00 MST --- *** This bug has been marked as a duplicate of bug 370457 *** https://bugzilla.novell.com/show_bug.cgi?id=370457 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User stbinner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c5 Stephan Binner <stbinner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|DUPLICATE | --- Comment #5 from Stephan Binner <stbinner@novell.com> 2008-04-21 01:50:54 MST --- No dupe. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User aos@att.net added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c6 Andrew Sorensen <aos@att.net> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aos@att.net --- Comment #6 from Andrew Sorensen <aos@att.net> 2008-05-05 20:08:04 MST --- please do it! my connection is fast but the servers are not. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User ma@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c7 Michael Andres <ma@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |jakub.rusinek@gmail.com --- Comment #7 from Michael Andres <ma@novell.com> 2008-05-08 03:16:29 MST --- *** Bug 385711 has been marked as a duplicate of this bug. *** https://bugzilla.novell.com/show_bug.cgi?id=385711 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User poeml@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c8 Peter Poeml <poeml@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |poeml@novell.com --- Comment #8 from Peter Poeml <poeml@novell.com> 2008-05-19 07:46:35 MST --- Segmented downloading with aria2 in parallel would make most sense, especially as download.opensuse.org supports metalinks transparently. See http://en.opensuse.org/Metalink for more information. At the same time, it would fix numerous problems with download failures (connection errors, timeouts, broken replies, ...). Refer also to http://en.opensuse.org/Libzypp/Failover) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 Michael Andres <ma@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ma@novell.com AssignedTo|zypp-maintainers@forge.provo.novell.com |ma@novell.com Status|REOPENED |ASSIGNED -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 Michael Andres <ma@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Component|libzypp |libzypp Product|openSUSE 11.0 |openSUSE 11.1 Version|Alpha 2plus |Alpha 0 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 Stanislav Visnovsky <visnov@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |visnov@novell.com Priority|P5 - None |P3 - Medium -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User jakub.rusinek@gmail.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c9 Jakub Rusinek <jakub.rusinek@gmail.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |xenoterracide@gmail.com --- Comment #9 from Jakub Rusinek <jakub.rusinek@gmail.com> 2008-10-06 10:41:05 MDT --- *** Bug 425031 has been marked as a duplicate of this bug. *** https://bugzilla.novell.com/show_bug.cgi?id=425031 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User matt@genesi-usa.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c10 Matt Sealey <matt@genesi-usa.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |matt@genesi-usa.com --- Comment #10 from Matt Sealey <matt@genesi-usa.com> 2008-11-26 12:22:05 MST --- What is the status of this for 11.1 or zypper 1.0? I would like to put my oar in here :D The most useful "parallel download" solution is to prefetch packages in the background - not to "maximize bandwidth" by fetching many packages simultaneously, but to get the next package while you are installing the previous. I have seen these effects on Gentoo Portage, and to be honest, a lot of workloads do not produce an appreciable effect. What alleviates it in Gentoo is compiling packages takes a long, long time, far longer in MOST cases than simply extracting/installing them. So here goes; ~ * Imagine you have some packages which are going to be installed. The first few packages are very very small (deltas or just small updates) - these can be downloaded in parallel (as in multiple HTTP downloads simultaneously) very fast and be ready as soon as zypper needs them. It usually takes far longer to extract and update the RPM DB than to download them. This is the ideal scenario. * Imagine that the 5th package in the list is huge - kernel-source or something of that ilk. You may be able to download the first 4 packages while the first is installing, and then take a fast path to installing 3 cached packages, but will still be waiting on kernel-source to download by the time they're installed. This scenario does not offer an appreciable speedup. * What happens if the next packages after kernel-source are also rather large, and the next set of TINY packages are too far away from the "parallel download limit"? Then, this does not offer any appreciable speedup. * It may even slow down the installation as downloading many large packages in parallel (as in many HTTP requests at once), on a moderate speed connection (~3Mbit/s) will actually increase the wait time while packages are downloaded. There really is not a safe default here as connections vary both in quality AND user behaviour. * And also when the packages to install are large but relatively easy to install (source rpms especially) then they can complete far quicker, leaving zypper with a wait for a package anyway. ~ Perhaps the most prudent solution is; - Implement a package prefetch cache which can download packages in the background during installation of others ("background downloading"). - Limits on prefetch cache - an example might be a maximum of 100MB and 5 files in the cache, and simultaneous downloading of 2 packages plus one prioritized foreground download (total 3). - Limits can be temporarily relieved on the basis of need - if you have kernel-source and can opportunistically fetch 30 smaller packages while it is installing, then the 5 file limit can be discarded in favour of the 100MB limit. In the event that these 30 packages can be fetched far less time than the installation, the limit on simultaneous downloads may be increased to increase the number of cached items. - Where the package sizes are small these can be favoured over larger packages, if only to reduce the latency of initiating a download task through curl, fetching it, and then installing it. Where this hits a wall by stumbling on a large package which has not even attempted to be prefetched yet, it will simply download as before, and not affect performance (no improvement, but no slowdown either). - Limits must be scaled back at the first, best opportunity to free up resources. Hard maximum limits should be enforced. - Prefetching should be based on the solver results such that packages which are to be installed could their major dependencies prefetched first - for instance there are two paths to walk in installing both KDE and GNOME components at the same time. In this scenario GNOME packages and KDE packages may prefetch at the same time and zypper can opportunistically switch between the two paths depending on availability in the cache, rather than a fixed route. Zypper should endeavour to install dependencies of both paths first so as to make way for parallel downloads and switching of dependency paths to the final packages. Thoughts? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User dmacvicar@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c11 --- Comment #11 from Duncan Mac-Vicar <dmacvicar@novell.com> 2008-11-27 01:49:58 MST --- We agree with that. We want to make installing and fetching in parallel. The multiple connection stuff we get it for free with the aria2c backend. Sadly this feature shows up all the time as high priority together with a bug bunch of other features with high priority. I hope we can do it for 11.2 with the planned ZYpp transaction rework. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=370054 User matt@genesi-usa.com added comment https://bugzilla.novell.com/show_bug.cgi?id=370054#c12 --- Comment #12 from Matt Sealey <matt@genesi-usa.com> 2008-11-27 09:33:32 MST --- I look forward to it if you get to it :) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply@novell.com