[zypp-devel] [PATCH 0/2] Support for primary.xml.lzma (fate#309167)
Hi, this series tries to implement fate#309167. In short: The 11.2+ kernel provides a dependency for each exported symbol (fate#305945), which results in thousands of provides. Zypp / sat-solver cope well with this, but the problem is that the primary.xml.gz file grows beyond limits. This is especially an issue in the update repository, which contains also previous versions of released updates (bnc#582907). The idea is to sort the entries in primary.xml and provide a lzma-compressed version allongside with primary.xml.gz, e.g.: <data type="primary"> <checksum type="sha256">5af6293d33986c74084639d228cab0eb66dcd16e1b2782e79447520055a256ee</checksum> <timestamp>1272657997</timestamp> <size>1747219</size> <open-size>12401312</open-size> <open-checksum type="sha256">c450888395ecc54345a0d0bc152b5bc774cea727e9d285bf6d88dda2d6bc0e0c</open-checksum> <location href="repodata/primary.xml.gz"/> </data> <data type="primary_lzma"> <checksum type="sha256">e15e7b0ae7756ca345580f5639f3512bcdfe23aa21c0355a7147af356a18bf22</checksum> <timestamp>1272657997</timestamp> <size>750408</size> <open-size>12401312</open-size> <open-checksum type="sha256">c450888395ecc54345a0d0bc152b5bc774cea727e9d285bf6d88dda2d6bc0e0c</open-checksum> <location href="repodata/primary.xml.lzma"/> </data> The createrepo changes are in http://gitorious.org/mmarek-misc/createrepo/commits/lzma-support, a package is in home:michal-m:branches:system:packagemanager/createrepo. These two patches change zypp to skip downloading primary.xml.gz if primary.xml.lzma is available. A third patch against sat-solver adds support for primary.xml.lzma to repo2solv.sh. libzypp.spec will need a versioned requires on satsolver-tools so that an up-to-date repo2solv.sh script is available. This part is not included. Please review, it would be great to have support for this in 11.3. Michal Michal Marek (2): Add Fetcher::cancelJob() Download primary.xml.lzma if available (fate#309167) zypp/Fetcher.cc | 52 ++++++++++++++++++++++++++-------------- zypp/Fetcher.h | 17 ++++++++++--- zypp/repo/yum/Downloader.cc | 23 +++++++++++++++++- zypp/repo/yum/Downloader.h | 1 + zypp/repo/yum/ResourceType.cc | 3 ++ zypp/repo/yum/ResourceType.h | 2 + 6 files changed, 75 insertions(+), 23 deletions(-) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
--- Note: this changes the abi, so a version bump is in order. zypp/Fetcher.cc | 52 ++++++++++++++++++++++++++++++++++------------------ zypp/Fetcher.h | 17 +++++++++++++---- 2 files changed, 47 insertions(+), 22 deletions(-) diff --git a/zypp/Fetcher.cc b/zypp/Fetcher.cc index 14b2325..c0c9820 100644 --- a/zypp/Fetcher.cc +++ b/zypp/Fetcher.cc @@ -88,6 +88,7 @@ namespace zypp // check checksums even if there is no such // checksum (warns of no checksum) AlwaysVerifyChecksum = 0x0004, + Cancelled = 0x0008, }; ZYPP_DECLARE_FLAGS(Flags, Flag); @@ -111,7 +112,6 @@ namespace zypp }; ZYPP_DECLARE_OPERATORS_FOR_FLAGS(FetcherJob::Flags); - typedef shared_ptr<FetcherJob> FetcherJob_Ptr; std::ostream & operator<<( std::ostream & str, const FetcherJob_Ptr & obj ) { @@ -137,12 +137,13 @@ namespace zypp void addIndex( const OnMediaLocation &resource ); - void enqueueDir( const OnMediaLocation &resource, bool recursive, const FileChecker &checker = FileChecker() ); - void enqueueDigestedDir( const OnMediaLocation &resource, bool recursive, const FileChecker &checker = FileChecker() ); + FetcherJob_Ptr enqueueDir( const OnMediaLocation &resource, bool recursive, const FileChecker &checker = FileChecker() ); + FetcherJob_Ptr enqueueDigestedDir( const OnMediaLocation &resource, bool recursive, const FileChecker &checker = FileChecker() ); - void enqueue( const OnMediaLocation &resource, const FileChecker &checker = FileChecker() ); - void enqueueDigested( const OnMediaLocation &resource, const FileChecker &checker = FileChecker() ); + FetcherJob_Ptr enqueue( const OnMediaLocation &resource, const FileChecker &checker = FileChecker() ); + FetcherJob_Ptr enqueueDigested( const OnMediaLocation &resource, const FileChecker &checker = FileChecker() ); void addCachePath( const Pathname &cache_dir ); + void cancelJob(FetcherJob_Ptr job); void reset(); void start( const Pathname &dest_dir, MediaSetAccess &media, @@ -232,12 +233,13 @@ namespace zypp }; /////////////////////////////////////////////////////////////////// - void Fetcher::Impl::enqueueDigested( const OnMediaLocation &resource, const FileChecker &checker ) + FetcherJob_Ptr Fetcher::Impl::enqueueDigested( const OnMediaLocation &resource, const FileChecker &checker ) { FetcherJob_Ptr job; job.reset(new FetcherJob(resource)); job->flags |= FetcherJob:: AlwaysVerifyChecksum; _resources.push_back(job); + return job; } Fetcher::Impl::Impl() @@ -251,7 +253,7 @@ namespace zypp Fetcher::Options Fetcher::Impl::options() const { return _options; } - void Fetcher::Impl::enqueueDir( const OnMediaLocation &resource, + FetcherJob_Ptr Fetcher::Impl::enqueueDir( const OnMediaLocation &resource, bool recursive, const FileChecker &checker ) { @@ -264,9 +266,10 @@ namespace zypp job->flags |= FetcherJob::Directory; _resources.push_back(job); + return job; } - void Fetcher::Impl::enqueueDigestedDir( const OnMediaLocation &resource, + FetcherJob_Ptr Fetcher::Impl::enqueueDigestedDir( const OnMediaLocation &resource, bool recursive, const FileChecker &checker ) { @@ -280,16 +283,17 @@ namespace zypp job->flags |= FetcherJob::AlwaysVerifyChecksum; _resources.push_back(job); - + return job; } - void Fetcher::Impl::enqueue( const OnMediaLocation &resource, const FileChecker &checker ) + FetcherJob_Ptr Fetcher::Impl::enqueue( const OnMediaLocation &resource, const FileChecker &checker ) { FetcherJob_Ptr job; job.reset(new FetcherJob(resource)); if ( checker ) job->checkers.push_back(checker); _resources.push_back(job); + return job; } void Fetcher::Impl::addIndex( const OnMediaLocation &resource ) @@ -330,6 +334,11 @@ namespace zypp } + void Fetcher::Impl::cancelJob(FetcherJob_Ptr job) + { + job->flags |= FetcherJob::Cancelled; + } + // tries to provide resource to dest_dir from any of the configured additional // cache paths where the file may already be present. returns true if the // file was provided from the cache. @@ -750,6 +759,8 @@ namespace zypp for ( list<FetcherJob_Ptr>::const_iterator it_res = _resources.begin(); it_res != _resources.end(); ++it_res ) { + if ( (*it_res)->flags & FetcherJob::Cancelled ) + continue; if ( (*it_res)->flags & FetcherJob::Directory ) { const OnMediaLocation location((*it_res)->location); @@ -860,23 +871,23 @@ namespace zypp return _pimpl->options(); } - void Fetcher::enqueueDigested( const OnMediaLocation &resource, const FileChecker &checker ) + FetcherJob_Ptr Fetcher::enqueueDigested( const OnMediaLocation &resource, const FileChecker &checker ) { - _pimpl->enqueueDigested(resource, checker); + return _pimpl->enqueueDigested(resource, checker); } - void Fetcher::enqueueDir( const OnMediaLocation &resource, + FetcherJob_Ptr Fetcher::enqueueDir( const OnMediaLocation &resource, bool recursive, const FileChecker &checker ) { - _pimpl->enqueueDir(resource, recursive, checker); + return _pimpl->enqueueDir(resource, recursive, checker); } - void Fetcher::enqueueDigestedDir( const OnMediaLocation &resource, + FetcherJob_Ptr Fetcher::enqueueDigestedDir( const OnMediaLocation &resource, bool recursive, const FileChecker &checker ) { - _pimpl->enqueueDigestedDir(resource, recursive, checker); + return _pimpl->enqueueDigestedDir(resource, recursive, checker); } @@ -886,9 +897,9 @@ namespace zypp } - void Fetcher::enqueue( const OnMediaLocation &resource, const FileChecker &checker ) + FetcherJob_Ptr Fetcher::enqueue( const OnMediaLocation &resource, const FileChecker &checker ) { - _pimpl->enqueue(resource, checker); + return _pimpl->enqueue(resource, checker); } void Fetcher::addCachePath( const Pathname &cache_dir ) @@ -896,6 +907,11 @@ namespace zypp _pimpl->addCachePath(cache_dir); } + void Fetcher::cancelJob(FetcherJob_Ptr job) + { + _pimpl->cancelJob(job); + } + void Fetcher::reset() { _pimpl->reset(); diff --git a/zypp/Fetcher.h b/zypp/Fetcher.h index 90babd8..4e407a5 100644 --- a/zypp/Fetcher.h +++ b/zypp/Fetcher.h @@ -29,6 +29,9 @@ namespace zypp { ///////////////////////////////////////////////////////////////// + class FetcherJob; + typedef shared_ptr<FetcherJob> FetcherJob_Ptr; + /** * This class allows to retrieve a group of files in a confortable * way, providing some smartness that does not belong to the @@ -102,6 +105,7 @@ namespace zypp public: /** Implementation */ class Impl; + public: /** @@ -177,7 +181,7 @@ namespace zypp * be transfered until \ref start() is called * */ - void enqueue( const OnMediaLocation &resource, + FetcherJob_Ptr enqueue( const OnMediaLocation &resource, const FileChecker &checker = FileChecker() ); /** @@ -191,7 +195,7 @@ namespace zypp * * \todo FIXME implement checker == operator to avoid this. */ - void enqueueDigested( const OnMediaLocation &resource, + FetcherJob_Ptr enqueueDigested( const OnMediaLocation &resource, const FileChecker &checker = FileChecker() ); @@ -234,7 +238,7 @@ namespace zypp * not be transfered and will be ignored. * */ - void enqueueDir( const OnMediaLocation &resource, + FetcherJob_Ptr enqueueDir( const OnMediaLocation &resource, bool recursive = false, const FileChecker &checker = FileChecker() ); @@ -278,7 +282,7 @@ namespace zypp * not be transfered and will be ignored. * */ - void enqueueDigestedDir( const OnMediaLocation &resource, + FetcherJob_Ptr enqueueDigestedDir( const OnMediaLocation &resource, bool recursive = false, const FileChecker &checker = FileChecker() ); @@ -287,6 +291,11 @@ namespace zypp * where to look for cached files */ void addCachePath( const Pathname &cache_dir ); + + /** + * Cancels a previously enqueued job + */ + void cancelJob(FetcherJob_Ptr job); /** * Reset the transfer (jobs) list -- 1.7.0.3 -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
--- zypp/repo/yum/Downloader.cc | 23 ++++++++++++++++++++++- zypp/repo/yum/Downloader.h | 1 + zypp/repo/yum/ResourceType.cc | 3 +++ zypp/repo/yum/ResourceType.h | 2 ++ 4 files changed, 28 insertions(+), 1 deletions(-) diff --git a/zypp/repo/yum/Downloader.cc b/zypp/repo/yum/Downloader.cc index a73b72f..3f51728 100644 --- a/zypp/repo/yum/Downloader.cc +++ b/zypp/repo/yum/Downloader.cc @@ -13,6 +13,7 @@ #include "zypp/base/Function.h" #include "zypp/Date.h" +#include "zypp/PathInfo.h" #include "zypp/parser/yum/RepomdFileReader.h" #include "zypp/parser/yum/PatchesFileReader.h" @@ -25,6 +26,7 @@ using namespace std; using namespace zypp::xml; using namespace zypp::parser::yum; +using namespace zypp::filesystem; namespace zypp { @@ -87,8 +89,27 @@ bool Downloader::repomd_Callback( const OnMediaLocation &loc, MIL << "Skipping filelists.xml.gz" << endl; return true; } + // skip primary.xml.gz if we already have primary.xml.lzma + if ( dtype == ResourceType::PRIMARY && _primary_lzma_job) + { + MIL << "Skipping primary.xml.gz" << endl; + return true; + } + + FetcherJob_Ptr job = this->enqueueDigested(loc_with_path); + if ( dtype == ResourceType::PRIMARY ) + _primary_gz_job = job; + // FIXME: search for 'lzma' in $PATH + if ( dtype == ResourceType::PRIMARY_LZMA && PathInfo("/usr/bin/lzma").isX()) + { + _primary_lzma_job = job; + if (_primary_gz_job) + { + MIL << "Skipping primary.xml.gz" << endl; + this->cancelJob(_primary_gz_job); + } + } - this->enqueueDigested(loc_with_path); // We got a patches file we need to read, to add patches listed // there, so we transfer what we have in the queue, and diff --git a/zypp/repo/yum/Downloader.h b/zypp/repo/yum/Downloader.h index e9b717a..4615db6 100644 --- a/zypp/repo/yum/Downloader.h +++ b/zypp/repo/yum/Downloader.h @@ -74,6 +74,7 @@ namespace zypp private: Pathname _dest_dir; std::list<OnMediaLocation> _patches_files; + FetcherJob_Ptr _primary_lzma_job, _primary_gz_job; MediaSetAccess *_media_ptr; }; diff --git a/zypp/repo/yum/ResourceType.cc b/zypp/repo/yum/ResourceType.cc index 650ba9f..8a22828 100644 --- a/zypp/repo/yum/ResourceType.cc +++ b/zypp/repo/yum/ResourceType.cc @@ -25,6 +25,7 @@ namespace zypp const ResourceType ResourceType::NONE(ResourceType::NONE_e); const ResourceType ResourceType::REPOMD(ResourceType::REPOMD_e); const ResourceType ResourceType::PRIMARY(ResourceType::PRIMARY_e); + const ResourceType ResourceType::PRIMARY_LZMA(ResourceType::PRIMARY_LZMA_e); const ResourceType ResourceType::OTHER(ResourceType::OTHER_e); const ResourceType ResourceType::FILELISTS(ResourceType::FILELISTS_e); const ResourceType ResourceType::GROUP(ResourceType::GROUP_e); @@ -46,6 +47,7 @@ namespace zypp // initialize it _table["repomd"] = ResourceType::REPOMD_e; _table["primary"] = ResourceType::PRIMARY_e; + _table["primary_lzma"] = ResourceType::PRIMARY_LZMA_e; _table["other"] = ResourceType::OTHER_e; _table["filelists"] = ResourceType::FILELISTS_e; _table["group"] = ResourceType::GROUP_e; @@ -76,6 +78,7 @@ namespace zypp // initialize it _table[REPOMD_e] = "repomd"; _table[PRIMARY_e] = "primary"; + _table[PRIMARY_LZMA_e] = "primary_lzma"; _table[OTHER_e] = "other"; _table[FILELISTS_e] = "filelists"; _table[GROUP_e] = "group"; diff --git a/zypp/repo/yum/ResourceType.h b/zypp/repo/yum/ResourceType.h index 2d5cf9e..6965195 100644 --- a/zypp/repo/yum/ResourceType.h +++ b/zypp/repo/yum/ResourceType.h @@ -29,6 +29,7 @@ namespace zypp static const ResourceType NONE; // unknown static const ResourceType REPOMD; static const ResourceType PRIMARY; + static const ResourceType PRIMARY_LZMA; // suse extension static const ResourceType OTHER; static const ResourceType FILELISTS; static const ResourceType GROUP; @@ -45,6 +46,7 @@ namespace zypp NONE_e, REPOMD_e, PRIMARY_e, + PRIMARY_LZMA_e, OTHER_e, FILELISTS_e, GROUP_e, -- 1.7.0.3 -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
--- tools/repo2solv.sh | 9 ++++++++- 1 files changed, 8 insertions(+), 1 deletions(-) diff --git a/tools/repo2solv.sh b/tools/repo2solv.sh index 4af01e8..7d66e4f 100755 --- a/tools/repo2solv.sh +++ b/tools/repo2solv.sh @@ -48,6 +48,8 @@ repomd_decompress() { case $1 in *.gz) gzip -dc "$1" ;; *.bz2) bzip2 -dc "$1" ;; + *.lzma) lzma -dc "$1" ;; + *.xz) xz -dc "$1" ;; *) cat "$1" ;; esac } @@ -95,7 +97,12 @@ if test "$repotype" = rpmmd ; then cd repodata || exit 2 primfile= - primxml=`repomd_findfile primary primary.xml` + if type lzma >/dev/null 2>&1; then + primxml=`repomd_findfile primary_lzma primary.xml` + fi + if test -z "$primxml"; then + primxml=`repomd_findfile primary primary.xml` + fi if test -n "$primxml" -a -s "$primxml" ; then primfile=`mktemp` || exit 3 ( -- 1.7.0.3 -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Mon, May 03, 2010 at 02:24:13PM +0200, Michal Marek wrote:
this series tries to implement fate#309167. In short: The 11.2+ kernel provides a dependency for each exported symbol (fate#305945), which results in thousands of provides.
Wait wait, didn't we agree on reverting this change and going back to symsets?
Zypp / sat-solver cope well with this, but the problem is that the primary.xml.gz file grows beyond limits.
I'm not really in favor of this. I would rather have zsync downloads for primary, which would need a "rsyncable" compression format, i.e. "gzip --rsyncable". Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On 3.5.2010 15:31, Michael Schroeder wrote:
On Mon, May 03, 2010 at 02:24:13PM +0200, Michal Marek wrote:
this series tries to implement fate#309167. In short: The 11.2+ kernel provides a dependency for each exported symbol (fate#305945), which results in thousands of provides.
Wait wait, didn't we agree on reverting this change and going back to symsets?
Yes, but I hope that's not the ultimate solution.
Zypp / sat-solver cope well with this, but the problem is that the primary.xml.gz file grows beyond limits.
I'm not really in favor of this. I would rather have zsync downloads for primary, which would need a "rsyncable" compression format, i.e. "gzip --rsyncable".
I see pros and cons of both approaches (rsyncable primary.xml.gz would probably result in smaller incremental updates, with primary.xml.lzma, the first download would be smaller) and I don't really insist on either solution. But support for primary.xml.lzma is indeed trivial to add, I have no idea how much effort would zsync support take. Michal -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On 3.5.2010 17:32, Michal Marek wrote:
I'm not really in favor of this. I would rather have zsync downloads for primary, which would need a "rsyncable" compression format, i.e. "gzip --rsyncable". I see pros and cons of both approaches (rsyncable primary.xml.gz would
On 3.5.2010 15:31, Michael Schroeder wrote: probably result in smaller incremental updates, with primary.xml.lzma, the first download would be smaller) and I don't really insist on either solution. But support for primary.xml.lzma is indeed trivial to add, I have no idea how much effort would zsync support take.
Michael, is there chance of having zsync support in 11.3? Thanks, Michal -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Wed, May 05, 2010 at 02:43:50PM +0200, Michal Marek wrote:
On 3.5.2010 17:32, Michal Marek wrote:
I'm not really in favor of this. I would rather have zsync downloads for primary, which would need a "rsyncable" compression format, i.e. "gzip --rsyncable". I see pros and cons of both approaches (rsyncable primary.xml.gz would
On 3.5.2010 15:31, Michael Schroeder wrote: probably result in smaller incremental updates, with primary.xml.lzma, the first download would be smaller) and I don't really insist on either solution. But support for primary.xml.lzma is indeed trivial to add, I have no idea how much effort would zsync support take.
Michael,
is there chance of having zsync support in 11.3?
It'll be my Hackweek project. I doubt that this is early enough for 11.3, though. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
participants (2)
-
Michael Schroeder -
Michal Marek