Feature changed by: jpxviii jpxviii (jpxviii) Feature #309561, revision 4 Title: Delta downloads of reposoitory metadata Hackweek V: Unconfirmed Priority Requester: Important + openSUSE-11.4: Unconfirmed + Priority + Requester: Mandatory Requested by: Michael Schröder (mlschroe) Description: We already use deltarpms do speed up downloading of rpms when doing an update. Unfortunately, the repository metadata is also quite a chunk of data which considerably slows down update speed. The idea is to use a zsync like algorithm to fetch only the changed parts of the metadata. The implementation will use libcurl's multi interface to work with multiple connections in parallel. Another benefit is that the code can replace the aria2c interface, which does not fit well into the current code (different proxy handling and the like). Discussion: #1: Jan Engelhardt (jengelh) (2010-06-02 12:41:45) Say, if we were to (finally) use solv-style metadata instead of rpm-md, wouldn't the problem reduce itself?   Also, if it were possible to have an aggregate metadata instead that can reference multiple subrepositories, big repositories (like packman and mine) could split theirs up into grouped repositories without having to add extra .repo entries to /etc/zypp/repos.d. E.g. /multimedia/i586/MPlayer.rpm /multimedia/i586/ffmpeg.rpm /multimedia/repodata/* /games/i586/trackball.rpm /games/i586/... /games/repodata/* /repodata/aggregate.xml   Each group (multimedia, games) would have its own metadata, that is, being a self-contained repository. And aggregate.xml containins then:   <link ref="multimedia/" /> <link ref="games/" />   So that when there is an update in games, only games's md needs to be downloaded, instead of the big one. -- openSUSE Feature: https://features.opensuse.org/309561

Feature changed by: Stephan Kulow (coolo) Feature #309561, revision 6 Title: Delta downloads of reposoitory metadata Hackweek V: Done Priority Requester: Important openSUSE-11.4: Done Priority Requester: Mandatory Info Provider: (Novell) Requested by: Michael Schröder (mlschroe) Description: We already use deltarpms do speed up downloading of rpms when doing an update. Unfortunately, the repository metadata is also quite a chunk of data which considerably slows down update speed. The idea is to use a zsync like algorithm to fetch only the changed parts of the metadata. The implementation will use libcurl's multi interface to work with multiple connections in parallel. Another benefit is that the code can replace the aria2c interface, which does not fit well into the current code (different proxy handling and the like). Discussion: #1: Jan Engelhardt (jengelh) (2010-06-02 12:41:45) Say, if we were to (finally) use solv-style metadata instead of rpm-md, wouldn't the problem reduce itself? Also, if it were possible to have an aggregate metadata instead that can reference multiple subrepositories, big repositories (like packman and mine) could split theirs up into grouped repositories without having to add extra .repo entries to /etc/zypp/repos.d. E.g. /multimedia/i586/MPlayer.rpm /multimedia/i586/ffmpeg.rpm /multimedia/repodata/* /games/i586/trackball.rpm /games/i586/... /games/repodata/* /repodata/aggregate.xml Each group (multimedia, games) would have its own metadata, that is, being a self-contained repository. And aggregate.xml containins then: <link ref="multimedia/" /> <link ref="games/" /> So that when there is an update in games, only games's md needs to be downloaded, instead of the big one. #2: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:57:21) Closing this one as done. #3: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:59:25) Coolo, giving the fact that aria is causing problems in 11.3 and this code is very isolated, would you accept a backport to 11.3 as online update in the context of: - enabled only explicitly (env variable) May serve as a good test for 11.4 as not many people is using Factory. + #4: Stephan Kulow (coolo) (2010-09-21 13:20:58) (reply to #3) + if it's not default, it might be worthwhile. Michael asked for it and I + considered a danger, but he forgot the detail about explicitly turning + it on. -- openSUSE Feature: https://features.opensuse.org/309561

Feature changed by: Jiri Benc (jbenc) Feature #309561, revision 9 - Title: Delta downloads of reposoitory metadata + Title: Delta downloads of repository metadata Hackweek V: Done Priority Requester: Important openSUSE-11.4: Done Priority Requester: Mandatory Info Provider: (Novell) Requested by: Michael Schröder (mlschroe) Description: We already use deltarpms do speed up downloading of rpms when doing an update. Unfortunately, the repository metadata is also quite a chunk of data which considerably slows down update speed. The idea is to use a zsync like algorithm to fetch only the changed parts of the metadata. The implementation will use libcurl's multi interface to work with multiple connections in parallel. Another benefit is that the code can replace the aria2c interface, which does not fit well into the current code (different proxy handling and the like). Discussion: #1: Jan Engelhardt (jengelh) (2010-06-02 12:41:45) Say, if we were to (finally) use solv-style metadata instead of rpm-md, wouldn't the problem reduce itself? Also, if it were possible to have an aggregate metadata instead that can reference multiple subrepositories, big repositories (like packman and mine) could split theirs up into grouped repositories without having to add extra .repo entries to /etc/zypp/repos.d. E.g. /multimedia/i586/MPlayer.rpm /multimedia/i586/ffmpeg.rpm /multimedia/repodata/* /games/i586/trackball.rpm /games/i586/... /games/repodata/* /repodata/aggregate.xml Each group (multimedia, games) would have its own metadata, that is, being a self-contained repository. And aggregate.xml containins then: <link ref="multimedia/" /> <link ref="games/" /> So that when there is an update in games, only games's md needs to be downloaded, instead of the big one. #2: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:57:21) Closing this one as done. #3: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:59:25) Coolo, giving the fact that aria is causing problems in 11.3 and this code is very isolated, would you accept a backport to 11.3 as online update in the context of: - enabled only explicitly (env variable) May serve as a good test for 11.4 as not many people is using Factory. #4: Stephan Kulow (coolo) (2010-09-21 13:20:58) (reply to #3) if it's not default, it might be worthwhile. Michael asked for it and I considered a danger, but he forgot the detail about explicitly turning it on. #6: Michal Marek (michal-m) (2010-10-08 17:04:31) BTW, what changes need to be done on the repository side? Or is this already fully implemented in Factory? -- openSUSE Feature: https://features.opensuse.org/309561

Feature changed by: Matthias Eckermann (mge1512) Feature #309561, revision 16 Title: Delta downloads of repository metadata Hackweek V: Done Priority Requester: Important openSUSE-11.4: Done Priority Requester: Mandatory Info Provider: (Novell) Requested by: Michael Schröder (mlschroe) + Product Manager: (Novell) Project Manager: (Novell) + Project Manager: (Novell) Partner organization: openSUSE.org Description: We already use deltarpms do speed up downloading of rpms when doing an update. Unfortunately, the repository metadata is also quite a chunk of data which considerably slows down update speed. The idea is to use a zsync like algorithm to fetch only the changed parts of the metadata. The implementation will use libcurl's multi interface to work with multiple connections in parallel. Another benefit is that the code can replace the aria2c interface, which does not fit well into the current code (different proxy handling and the like). Discussion: #1: Jan Engelhardt (jengelh) (2010-06-02 12:41:45) Say, if we were to (finally) use solv-style metadata instead of rpm-md, wouldn't the problem reduce itself? Also, if it were possible to have an aggregate metadata instead that can reference multiple subrepositories, big repositories (like packman and mine) could split theirs up into grouped repositories without having to add extra .repo entries to /etc/zypp/repos.d. E.g. /multimedia/i586/MPlayer.rpm /multimedia/i586/ffmpeg.rpm /multimedia/repodata/* /games/i586/trackball.rpm /games/i586/... /games/repodata/* /repodata/aggregate.xml Each group (multimedia, games) would have its own metadata, that is, being a self-contained repository. And aggregate.xml containins then: <link ref="multimedia/" /> <link ref="games/" /> So that when there is an update in games, only games's md needs to be downloaded, instead of the big one. #2: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:57:21) Closing this one as done. #3: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:59:25) Coolo, giving the fact that aria is causing problems in 11.3 and this code is very isolated, would you accept a backport to 11.3 as online update in the context of: - enabled only explicitly (env variable) May serve as a good test for 11.4 as not many people is using Factory. #4: Stephan Kulow (coolo) (2010-09-21 13:20:58) (reply to #3) if it's not default, it might be worthwhile. Michael asked for it and I considered a danger, but he forgot the detail about explicitly turning it on. #6: Michal Marek (michal-m) (2010-10-08 17:04:31) BTW, what changes need to be done on the repository side? Or is this already fully implemented in Factory? #7: Stephan Kulow (coolo) (2010-10-11 11:05:08) (reply to #6) afaik we patched the way the meta data is compressed (gzip --rsyncable) and the download redirector now gives out more details about the file blocks. The first part is easy to do for older products, the metalink for SLES updates might be a bit challenging. -- openSUSE Feature: https://features.opensuse.org/309561

Feature changed by: Lukas Ocilka (locilka) Feature #309561, revision 19 Title: Delta downloads of repository metadata Hackweek V: Done Priority Requester: Important openSUSE-11.4: Done Priority Requester: Mandatory Info Provider: (Novell) Requested by: Michael Schröder (mlschroe) Product Manager: (Novell) Project Manager: (Novell) Project Manager: (Novell) Engineering Manager: (Novell) - Engineering Manager: (Novell) Partner organization: openSUSE.org Description: We already use deltarpms do speed up downloading of rpms when doing an update. Unfortunately, the repository metadata is also quite a chunk of data which considerably slows down update speed. The idea is to use a zsync like algorithm to fetch only the changed parts of the metadata. The implementation will use libcurl's multi interface to work with multiple connections in parallel. Another benefit is that the code can replace the aria2c interface, which does not fit well into the current code (different proxy handling and the like). Discussion: #1: Jan Engelhardt (jengelh) (2010-06-02 12:41:45) Say, if we were to (finally) use solv-style metadata instead of rpm-md, wouldn't the problem reduce itself? Also, if it were possible to have an aggregate metadata instead that can reference multiple subrepositories, big repositories (like packman and mine) could split theirs up into grouped repositories without having to add extra .repo entries to /etc/zypp/repos.d. E.g. /multimedia/i586/MPlayer.rpm /multimedia/i586/ffmpeg.rpm /multimedia/repodata/* /games/i586/trackball.rpm /games/i586/... /games/repodata/* /repodata/aggregate.xml Each group (multimedia, games) would have its own metadata, that is, being a self-contained repository. And aggregate.xml containins then: <link ref="multimedia/" /> <link ref="games/" /> So that when there is an update in games, only games's md needs to be downloaded, instead of the big one. #2: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:57:21) Closing this one as done. #3: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:59:25) Coolo, giving the fact that aria is causing problems in 11.3 and this code is very isolated, would you accept a backport to 11.3 as online update in the context of: - enabled only explicitly (env variable) May serve as a good test for 11.4 as not many people is using Factory. #4: Stephan Kulow (coolo) (2010-09-21 13:20:58) (reply to #3) if it's not default, it might be worthwhile. Michael asked for it and I considered a danger, but he forgot the detail about explicitly turning it on. #6: Michal Marek (michal-m) (2010-10-08 17:04:31) BTW, what changes need to be done on the repository side? Or is this already fully implemented in Factory? #7: Stephan Kulow (coolo) (2010-10-11 11:05:08) (reply to #6) afaik we patched the way the meta data is compressed (gzip --rsyncable) and the download redirector now gives out more details about the file blocks. The first part is easy to do for older products, the metalink for SLES updates might be a bit challenging. -- openSUSE Feature: https://features.opensuse.org/309561

Feature changed by: Duncan Mac-Vicar (dmacvicar) Feature #309561, revision 22 Title: Delta downloads of repository metadata Hackweek V: Done Priority Requester: Important openSUSE-11.4: Done Priority Requester: Mandatory Info Provider: (Novell) Requested by: Michael Schröder (mlschroe) Product Manager: (Novell) Project Manager: (Novell) Project Manager: (Novell) Engineering Manager: (Novell) Developer: (Novell) Developer: (Novell) Partner organization: openSUSE.org Description: We already use deltarpms do speed up downloading of rpms when doing an update. Unfortunately, the repository metadata is also quite a chunk of data which considerably slows down update speed. The idea is to use a zsync like algorithm to fetch only the changed parts of the metadata. The implementation will use libcurl's multi interface to work with multiple connections in parallel. Another benefit is that the code can replace the aria2c interface, which does not fit well into the current code (different proxy handling and the like). + References: + packages: libzypp Discussion: #1: Jan Engelhardt (jengelh) (2010-06-02 12:41:45) Say, if we were to (finally) use solv-style metadata instead of rpm-md, wouldn't the problem reduce itself? Also, if it were possible to have an aggregate metadata instead that can reference multiple subrepositories, big repositories (like packman and mine) could split theirs up into grouped repositories without having to add extra .repo entries to /etc/zypp/repos.d. E.g. /multimedia/i586/MPlayer.rpm /multimedia/i586/ffmpeg.rpm /multimedia/repodata/* /games/i586/trackball.rpm /games/i586/... /games/repodata/* /repodata/aggregate.xml Each group (multimedia, games) would have its own metadata, that is, being a self-contained repository. And aggregate.xml containins then: <link ref="multimedia/" /> <link ref="games/" /> So that when there is an update in games, only games's md needs to be downloaded, instead of the big one. #2: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:57:21) Closing this one as done. #3: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:59:25) Coolo, giving the fact that aria is causing problems in 11.3 and this code is very isolated, would you accept a backport to 11.3 as online update in the context of: - enabled only explicitly (env variable) May serve as a good test for 11.4 as not many people is using Factory. #4: Stephan Kulow (coolo) (2010-09-21 13:20:58) (reply to #3) if it's not default, it might be worthwhile. Michael asked for it and I considered a danger, but he forgot the detail about explicitly turning it on. #6: Michal Marek (michal-m) (2010-10-08 17:04:31) BTW, what changes need to be done on the repository side? Or is this already fully implemented in Factory? #7: Stephan Kulow (coolo) (2010-10-11 11:05:08) (reply to #6) afaik we patched the way the meta data is compressed (gzip --rsyncable) and the download redirector now gives out more details about the file blocks. The first part is easy to do for older products, the metalink for SLES updates might be a bit challenging. -- openSUSE Feature: https://features.opensuse.org/309561

Feature changed by: Michael Andres (mlandres) Feature #309561, revision 29 Title: Delta downloads of repository metadata Hackweek V: Done Priority Requester: Important openSUSE-11.4: Done Priority Requester: Mandatory Info Provider: (Novell) Requested by: Michael Schröder (mlschroe) Product Manager: (Novell) Project Manager: (Novell) Project Manager: (Novell) Engineering Manager: (Novell) Developer: (Novell) - Developer: (Novell) Partner organization: openSUSE.org Description: We already use deltarpms do speed up downloading of rpms when doing an update. Unfortunately, the repository metadata is also quite a chunk of data which considerably slows down update speed. The idea is to use a zsync like algorithm to fetch only the changed parts of the metadata. The implementation will use libcurl's multi interface to work with multiple connections in parallel. Another benefit is that the code can replace the aria2c interface, which does not fit well into the current code (different proxy handling and the like). References: packages: libzypp Discussion: #1: Jan Engelhardt (jengelh) (2010-06-02 12:41:45) Say, if we were to (finally) use solv-style metadata instead of rpm-md, wouldn't the problem reduce itself? Also, if it were possible to have an aggregate metadata instead that can reference multiple subrepositories, big repositories (like packman and mine) could split theirs up into grouped repositories without having to add extra .repo entries to /etc/zypp/repos.d. E.g. /multimedia/i586/MPlayer.rpm /multimedia/i586/ffmpeg.rpm /multimedia/repodata/* /games/i586/trackball.rpm /games/i586/... /games/repodata/* /repodata/aggregate.xml Each group (multimedia, games) would have its own metadata, that is, being a self-contained repository. And aggregate.xml containins then: <link ref="multimedia/" /> <link ref="games/" /> So that when there is an update in games, only games's md needs to be downloaded, instead of the big one. #2: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:57:21) Closing this one as done. #3: Duncan Mac-Vicar (dmacvicar) (2010-09-21 12:59:25) Coolo, giving the fact that aria is causing problems in 11.3 and this code is very isolated, would you accept a backport to 11.3 as online update in the context of: - enabled only explicitly (env variable) May serve as a good test for 11.4 as not many people is using Factory. #4: Stephan Kulow (coolo) (2010-09-21 13:20:58) (reply to #3) if it's not default, it might be worthwhile. Michael asked for it and I considered a danger, but he forgot the detail about explicitly turning it on. #6: Michal Marek (michal-m) (2010-10-08 17:04:31) BTW, what changes need to be done on the repository side? Or is this already fully implemented in Factory? #7: Stephan Kulow (coolo) (2010-10-11 11:05:08) (reply to #6) afaik we patched the way the meta data is compressed (gzip --rsyncable) and the download redirector now gives out more details about the file blocks. The first part is easy to do for older products, the metalink for SLES updates might be a bit challenging. #14: Duncan Mac-Vicar (dmacvicar) (2011-05-06 12:28:45) libzypp part is done when request id 69748 is accepted + #19: Michael Andres (mlandres) (2011-06-09 14:36:44) (reply to #14) + This libzypp is available in SUSE:SLE-11:SP2 -- openSUSE Feature: https://features.opensuse.org/309561