Mailinglist Archive: opensuse-features (253 mails)

< Previous Next >
[openFATE 309561] Delta downloads of reposoitory metadata
  • From: fate_noreply@xxxxxxx
  • Date: Tue, 21 Sep 2010 13:21:02 +0200 (CEST)
  • Message-id: <feature-309561-6@xxxxxxxxxxxxxx>
Feature changed by: Stephan Kulow (coolo)
Feature #309561, revision 6
Title: Delta downloads of reposoitory metadata

Hackweek V: Done
Requester: Important

openSUSE-11.4: Done
Requester: Mandatory

Info Provider: (Novell)
Requested by: Michael Schröder (mlschroe)

We already use deltarpms do speed up downloading of rpms when doing an
update. Unfortunately, the repository metadata is also quite a chunk of
data which considerably slows down update speed.
The idea is to use a zsync like algorithm to fetch only the changed
parts of the metadata. The implementation will use libcurl's multi
interface to work with multiple connections in parallel.
Another benefit is that the code can replace the aria2c interface,
which does not fit well into the current code (different proxy handling
and the like).

#1: Jan Engelhardt (jengelh) (2010-06-02 12:41:45)
Say, if we were to (finally) use solv-style metadata instead of rpm-md,
wouldn't the problem reduce itself?
Also, if it were possible to have an aggregate metadata instead that
can reference multiple subrepositories, big repositories (like packman
and mine) could split theirs up into grouped repositories without
having to add extra .repo entries to /etc/zypp/repos.d.
Each group (multimedia, games) would have its own metadata, that is,
being a self-contained repository. And aggregate.xml containins then:
<link ref="multimedia/" />
<link ref="games/" />
So that when there is an update in games, only games's md needs to be
downloaded, instead of the big one.

#2: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:57:21)
Closing this one as done.

#3: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:59:25)
Coolo, giving the fact that aria is causing problems in 11.3 and this
code is very isolated, would you accept a backport to 11.3 as online
update in the context of:
- enabled only explicitly (env variable)
May serve as a good test for 11.4 as not many people is using Factory.

+ #4: Stephan Kulow (coolo) (2010-09-21 13:20:58) (reply to #3)
+ if it's not default, it might be worthwhile. Michael asked for it and I
+ considered a danger, but he forgot the detail about explicitly turning
+ it on.

openSUSE Feature:

< Previous Next >
This Thread