Mailinglist Archive: opensuse-features (214 mails)

< Previous Next >
[openFATE 309561] Delta downloads of repository metadata
Feature changed by: Duncan Mac-Vicar (dmacvicar)
Feature #309561, revision 24
Title: Delta downloads of repository metadata

Hackweek V: Done
Priority
Requester: Important

openSUSE-11.4: Done
Priority
Requester: Mandatory

Info Provider: (Novell)
Requested by: Michael Schröder (mlschroe)
Product Manager: (Novell)
Project Manager: (Novell)
Project Manager: (Novell)
Engineering Manager: (Novell)
Developer: (Novell)
Developer: (Novell)
Partner organization: openSUSE.org

Description:
We already use deltarpms do speed up downloading of rpms when doing an
update. Unfortunately, the repository metadata is also quite a chunk of
data which considerably slows down update speed.
The idea is to use a zsync like algorithm to fetch only the changed
parts of the metadata. The implementation will use libcurl's multi
interface to work with multiple connections in parallel.
Another benefit is that the code can replace the aria2c interface,
which does not fit well into the current code (different proxy handling
and the like).

References:
packages: libzypp

Discussion:
#1: Jan Engelhardt (jengelh) (2010-06-02 12:41:45)
Say, if we were to (finally) use solv-style metadata instead of rpm-md,
wouldn't the problem reduce itself?
Also, if it were possible to have an aggregate metadata instead that
can reference multiple subrepositories, big repositories (like packman
and mine) could split theirs up into grouped repositories without
having to add extra .repo entries to /etc/zypp/repos.d.
E.g.
/multimedia/i586/MPlayer.rpm
/multimedia/i586/ffmpeg.rpm
/multimedia/repodata/*
/games/i586/trackball.rpm
/games/i586/...
/games/repodata/*
/repodata/aggregate.xml
Each group (multimedia, games) would have its own metadata, that is,
being a self-contained repository. And aggregate.xml containins then:
<link ref="multimedia/" />
<link ref="games/" />
So that when there is an update in games, only games's md needs to be
downloaded, instead of the big one.

#2: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:57:21)
Closing this one as done.

#3: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:59:25)
Coolo, giving the fact that aria is causing problems in 11.3 and this
code is very isolated, would you accept a backport to 11.3 as online
update in the context of:
- enabled only explicitly (env variable)
May serve as a good test for 11.4 as not many people is using Factory.

#4: Stephan Kulow (coolo) (2010-09-21 13:20:58) (reply to #3)
if it's not default, it might be worthwhile. Michael asked for it and I
considered a danger, but he forgot the detail about explicitly turning
it on.

#6: Michal Marek (michal-m) (2010-10-08 17:04:31)
BTW, what changes need to be done on the repository side? Or is this
already fully implemented in Factory?

#7: Stephan Kulow (coolo) (2010-10-11 11:05:08) (reply to #6)
afaik we patched the way the meta data is compressed (gzip --rsyncable)
and the download redirector now gives out more details about the file
blocks.
The first part is easy to do for older products, the metalink for SLES
updates might be a bit challenging.

+ #14: Duncan Mac-Vicar (dmacvicar) (2011-05-06 12:28:45)
+ libzypp part is done when request id 69748 is accepted




--
openSUSE Feature:
https://features.opensuse.org/309561

< Previous Next >
This Thread
  • No further messages