October 2007 - Zypp Development - openSUSE Mailing Lists

[zypp-devel] Re: [zypp-commit] r7504 - in /trunk/libzypp: CMakeLists.txt cmake/modules/FindSatsolver.cmake libzypp.spec.cmake
by Michael Matz 12 Oct '07

12 Oct '07

Hi, On Fri, 12 Oct 2007, mlandres(a)svn.opensuse.org wrote: > +FIND_LIBRARY(SATSOLVER_LIBRARY NAMES satsolver0 > + PATHS > + /usr/lib > + /usr/local/lib > +) The library now is called libsatsolver again (without the '0'). For development it would be nice if the includes and library could also be searched in ${CMAKE_INSTALL_PREFIX}/{lib*,include}/ Ciao, Michael. -- To unsubscribe, e-mail: zypp-devel+unsubscribe(a)opensuse.org For additional commands, e-mail: zypp-devel+help(a)opensuse.org

1 0

[zypp-devel] Re: [yast-devel] Zypp Patterns
by Lukas Ocilka 10 Oct '07

10 Oct '07

Ricardo Cruz napsal(a): > Hey, > > libzypp Pattern's install_packages() is marked obsolete as of 10.3. > yast-gtk keeps using it, and so does -qt. Can you guys point me to the > new API? It seems that zypp-devel at opensuse.org might be more suitable for asking this question :) Lukas -- Lukas Ocilka, YaST Developer (xn--luk-gla45d) ----------------------------------------------------------------- SUSE LINUX, s. r. o., Lihovarska 1060/12, Praha 9, Czech Republic

1 0

[zypp-devel] my plan: SOLV files for libzypp
by Michael Matz 08 Oct '07

08 Oct '07

Aloha, for your viewing pleasure here are my weekend hacks (and this time it's really really horrible, don't look at them) to make libzypp use SOLV files instead of sqlite and rpm database as backends. In it's current form quite some information about the resolvable details are missing (no description, no summary, no changelog, nothing except what a NVRAD contains). The connection between the resolvable and its SQLite ID is lost too (hence zypper if bash will give a funny summary/description from a different resolvable). The category of patches is lost (because that's a simple text attribute), as are many other things. And it doesn't automatically update the .solv files with the repositories, you have to do that by hand (files are /var/cache/zypp/$ID.solv, where ID is the repository ID from the sqlite repositories table). Except for the rpmdatabase shadow, that's kept in sync (in /tmp/system.solv, I wanted to develop as user, who can't write in /var/cache/zypp). And it duplicates code. And it doesn't do error checking. And it kill kittens, when someone reads it. The nice thing is, that even without underlying changes in libzypp I get my testcase of "zypper lu" down to 5 seconds, and in the profiles it's now realitively obvious where to improve next. If we go the route of .solv files (and we should) we can make much of the underlying representation of Edition/Atom/Name/Caps and many other things be just Id, i.e. an integer. CapSets will just be zero-terminated lists of Ids. That could be implemented directly in the zypp classes, instead of the current insane Impl indirection, saving memory and new calls (currently we do about 6 millions (!)). I'm personally planning to do more about this. In particular working with mls on extending the .solv format a bit to store also some other information about resolvables. I haven't yet made up my mind if the SQLite database will stay or not. It's quite nice to have for some information (as it's easily extensible), but the necessity to then maintain a mapping from resolvable-ID (in zypp.db) to source/resolvable-id (in SOLV) makes this less attractive. Anyway, so you know about my plans, here's my current patch^Whack^Wunbelievable bloody crap. Ciao, Michael. -- Index: target/rpm/RpmPackageImpl.cc =================================================================== --- target/rpm/RpmPackageImpl.cc (revision 7438) +++ target/rpm/RpmPackageImpl.cc (working copy) @@ -48,17 +48,45 @@ RPMPackageImpl::RPMPackageImpl( _license(data->tag_license()), _packager(data->tag_packager()), _group(data->tag_group()), - _changelog(data->tag_changelog()), + //_changelog(data->tag_changelog()), + _changelog(Changelog()), _type("rpm"), // FIXME in the future - _filenames(data->tag_filenames()), + //_filenames(data->tag_filenames()), + _filenames(std::list<std::string>()), _size(data->tag_size()) { // we know we are reading english. - _description.setText(data->tag_description(), Locale("en")); + //_description.setText(data->tag_description(), Locale("en")); + _description.setText(std::string(), Locale("en")); data->tag_du(_disk_usage); _location.setDownloadSize(data->tag_archivesize()); } +RPMPackageImpl::RPMPackageImpl() + : _summary("XXXXXXXXXXX", Locale("en")), + _description("XXX"), + _buildtime(Date()), + _installtime(Date()), + _buildhost(""), + _url(""), + _vendor(""), + _license(""), + _packager(""), + _group(PackageGroup()), + //_changelog(data->tag_changelog()), + _changelog(Changelog()), + _type("rpm"), // FIXME in the future + //_filenames(data->tag_filenames()), + _filenames(std::list<std::string>()), + _size(0) +{ + // we know we are reading english. + //_description.setText(data->tag_description(), Locale("en")); + _description.setText(std::string(), Locale("en")); + //data->tag_du(_disk_usage); + //_location.setDownloadSize(data->tag_archivesize()); +} + /** Package summary */ TranslatedText RPMPackageImpl::summary() const { @@ -128,6 +156,7 @@ PackageGroup RPMPackageImpl::group() con /** */ Changelog RPMPackageImpl::changelog() const { +std::cerr << "Argh" << std::endl; return _changelog; } @@ -189,6 +218,7 @@ ByteCount RPMPackageImpl::sourcesize() c /** */ std::list<std::string> RPMPackageImpl::filenames() const { +//std::cerr << "Argh2" << std::endl; return _filenames; } Index: target/rpm/RpmPackageImpl.h =================================================================== --- target/rpm/RpmPackageImpl.h (revision 7438) +++ target/rpm/RpmPackageImpl.h (working copy) @@ -40,6 +40,8 @@ public: const RpmHeader::constPtr data ); + RPMPackageImpl(); + /** Package summary */ virtual TranslatedText summary() const; /** Package description */ Index: target/rpm/RpmDb.cc =================================================================== --- target/rpm/RpmDb.cc (revision 7438) +++ target/rpm/RpmDb.cc (working copy) @@ -28,11 +28,15 @@ #include "zypp/base/String.h" #include "zypp/base/Regex.h" +#include "zypp/data/RecordId.h" + #include "zypp/Date.h" #include "zypp/Pathname.h" #include "zypp/PathInfo.h" #include "zypp/PublicKey.h" +#include "zypp/capability/Capabilities.h" + #include "zypp/target/rpm/RpmDb.h" #include "zypp/target/rpm/RpmCallbacks.h" @@ -55,6 +59,7 @@ using namespace zypp::filesystem; namespace zypp { +extern Pool * the_pool; namespace target { namespace rpm @@ -277,6 +282,7 @@ public: RpmDb::RpmDb() : _dbStateInfo( DbSI_NO_INIT ) , _packages( * new Packages ) // delete in destructor + , source(0) #warning Check for obsolete memebers , _backuppath ("/var/adm/backup") , _packagebackups(false) @@ -1254,6 +1260,117 @@ Package::Ptr RpmDb::makePackageFromHeade return pptr; } +static bool +update_solv (string dbpath) +{ + cerr << "trying " << dbpath << endl; + if (dbpath != "//var/lib/rpm" && dbpath != "/var/lib/rpm") + return false; + cerr << "Building system.solv" << endl; + system ("if test -f /tmp/system.solv; then /matz/projects/solver/klaus/dev/tools/rpmdb2solv /tmp/system.solv > /tmp/system.solv.new; else /matz/projects/solver/klaus/dev/tools/rpmdb2solv > /tmp/system.solv.new; fi; mv /tmp/system.solv.new /tmp/system.solv"); + return true; +} + +static inline Resolvable::Kind +kind_for_name (const char *name, Id arch) +{ + switch (name[0]) + { + case 'a': + if (!strncmp (name, "atom:", 5)) + return ResTraits<Atom>::kind; + break; + case 'l': + if (!strncmp (name, "language:", 9)) + return ResTraits<Language>::kind; + break; + case 'm': + if (!strncmp (name, "message:", 8)) + return ResTraits<Message>::kind; + break; + case 'p': + if (!strncmp (name, "patch:", 6)) + return ResTraits<Patch>::kind; + if (!strncmp (name, "pattern:", 8)) + return ResTraits<Pattern>::kind; + if (!strncmp (name, "product:", 8)) + return ResTraits<Product>::kind; + break; + case 's': + if (!strncmp (name, "selection:", 10)) + return ResTraits<Selection>::kind; + if (!strncmp (name, "script:", 7)) + return ResTraits<Script>::kind; + break; + } + if (arch == ARCH_SRC) + return ResTraits<SrcPackage>::kind; + return ResTraits<Package>::kind; +} + +map<Id, Capability> id2cap; +map<Id, Edition> id2ed; +static const Rel flag2rel[8] = { + Rel::NONE, + Rel::GT, + Rel::EQ, + Rel::GE, + Rel::LT, + Rel::NE, + Rel::LE, + Rel::ANY +}; +static inline void +add_deps (CapFactory &factory, Pool *pool, CapSet &caps, Id *ids) +{ + Id id; + if (!ids) + return; + while((id = *ids++) != 0) + { + if (id2cap.find(id) != id2cap.end()) + { + caps.insert (id2cap[id]); + } + else + { + const char *name = id2str (pool, id); + capability::CapabilityImpl::Ptr capi; + if (name[0] == '/') + { + capi = new capability::FileCap (ResTraits<Package>::kind, name); + } + /* ??? The normal target::rpm code doesn't look at the specific + kind of the dependencies (hal/modalias/filesystem). */ + else if (!ISRELDEP (id)) + { + Resolvable::Kind refer = kind_for_name (name, 1); + capi = new capability::NamedCap (refer, name); + } + else + { + Reldep *rd = GETRELDEP (pool, id); + Resolvable::Kind refer = kind_for_name (name, 1); + Rel rel = flag2rel[rd->flags & 7]; + Edition ed; + if (id2ed.find (rd->evr) != id2ed.end()) + ed = id2ed[rd->evr]; + else + { + /* ??? catch exception here? */ + ed = Edition (id2str (pool, rd->evr)); + id2ed[rd->evr] = ed; + } + capi = new capability::VersionedCap( refer, name, rel, ed ); + } + + Capability cap = factory.fromImpl(capi); + id2cap[id] = cap; + caps.insert (cap); + } + } +} + const list<Package::Ptr> & RpmDb::doGetPackages(callback::SendReport<ScanDBReport> & report) { if ( packagesValid() ) @@ -1262,19 +1379,88 @@ const list<Package::Ptr> & RpmDb::doGetP } _packages.clear(); + if (source) + { + pool_freesource (the_pool, source); + source = 0; + } + if (update_solv ((root() + dbPath()).asString())) + { + FILE *fp = fopen ("/tmp/system.solv", "r"); + if (fp) + { + source = pool_addsource_solv(the_pool, fp, "system"); + fclose (fp); + } + if (source) + { + int i; + cerr << "SOLV: have " << source->nsolvables << " solvables" << endl; + CapFactory capfactory; + for (i = source->start; i < source->start + source->nsolvables; i++) + { + Solvable *s = the_pool->solvables + i; + data::RecordId id(i); + Resolvable::Kind kind; + Edition ed; + const char *name = id2str (the_pool, s->name); + if (id2ed.find (s->evr) != id2ed.end()) + ed = id2ed[s->evr]; + else + { + /* ??? catch exception here? */ + /* XXX Edition ctor does a regexp if given a string :( */ + ed = Edition (id2str (the_pool, s->evr)); + id2ed[s->evr] = ed; + } + NVRAD nvrad ( name, + ed, + /* XXX Arch needs to become Id based. */ + Arch(id2str (the_pool, s->arch))); + add_deps (capfactory, the_pool, nvrad[Dep::PROVIDES], s->provides); + add_deps (capfactory, the_pool, nvrad[Dep::REQUIRES], s->requires); + add_deps (capfactory, the_pool, nvrad[Dep::CONFLICTS], s->conflicts); + add_deps (capfactory, the_pool, nvrad[Dep::OBSOLETES], s->obsoletes); + add_deps (capfactory, the_pool, nvrad[Dep::RECOMMENDS], s->recommends); + add_deps (capfactory, the_pool, nvrad[Dep::SUGGESTS], s->suggests); + add_deps (capfactory, the_pool, nvrad[Dep::FRESHENS], s->freshens); + add_deps (capfactory, the_pool, nvrad[Dep::ENHANCES], s->enhances); + add_deps (capfactory, the_pool, nvrad[Dep::SUPPLEMENTS], s->supplements); + //add_deps (capfactory, nvrad[Dep::PREREQUIRES], s->prerequires); + detail::ResImplTraits<RPMPackageImpl>::Ptr impl( new RPMPackageImpl() ); + + Package::Ptr pptr; + try + { + // create package from dataprovider + pptr = detail::makeResolvableFromImpl( nvrad, impl ); + } + catch (Exception & excpt_r) + { + ZYPP_CAUGHT( excpt_r ); + ERR << "Can't create Package::Ptr" << endl; + } + if (pptr) + _packages._list.push_back (pptr); + /* ??? deal with filerequires somehow? Should be done in the solver. */ + } + } + } /////////////////////////////////////////////////////////////////// // Collect package data. /////////////////////////////////////////////////////////////////// + librpmDb::db_const_iterator iter; + CapFactory _f; + + if (!source) { unsigned expect = 0; librpmDb::constPtr dbptr; librpmDb::dbAccess( dbptr ); expect = dbptr->size(); DBG << "Expecting " << expect << " packages" << endl; - librpmDb::db_const_iterator iter; unsigned current = 0; - CapFactory _f; Pathname location; for ( iter.findAll(); *iter; ++iter, ++current, report->progress( (100*current)/expect)) @@ -1298,6 +1484,7 @@ const list<Package::Ptr> & RpmDb::doGetP _packages._list.push_back( pptr ); } + } _packages.buildIndex(); DBG << "Found installed packages: " << _packages._list.size() << endl; Index: target/rpm/RpmDb.h =================================================================== --- target/rpm/RpmDb.h (revision 7438) +++ target/rpm/RpmDb.h (working copy) @@ -30,6 +30,9 @@ #include "zypp/target/rpm/RpmCallbacks.h" #include "zypp/ZYppCallbacks.h" +#include "pool.h" +#include "source_solv.h" + namespace zypp { namespace target @@ -239,6 +242,7 @@ private: class Packages; Packages & _packages; + Source *source; std::set<std::string> _filerequires; Index: repo/cached/RepoImpl.cc =================================================================== --- repo/cached/RepoImpl.cc (revision 7438) +++ repo/cached/RepoImpl.cc (working copy) @@ -38,6 +38,9 @@ #include "zypp/repo/cached/AtomImpl.h" #include "zypp/cache/CacheAttributes.h" +#include "pool.h" +#include "source_solv.h" + using namespace zypp::detail; using namespace zypp::cache; using namespace std; @@ -66,6 +69,232 @@ RepoImpl::~RepoImpl() MIL << "Destroying repo '" << info().alias() << "'" << endl; } +void RepoImpl::insertNVRAD (data::RecordId id, Resolvable::Kind kind, const NVRAD &nvrad) +{ + if ( kind == ResTraits<Package>::kind ) + { + ResImplTraits<cached::PackageImpl>::Ptr impl = new cached::PackageImpl(id, this); + Package::Ptr package = detail::makeResolvableFromImpl( nvrad, impl ); + _store.insert( package ); + } + else if ( kind == ResTraits<SrcPackage>::kind ) + { + ResImplTraits<cached::SrcPackageImpl>::Ptr impl = new cached::SrcPackageImpl(id, this); + SrcPackage::Ptr srcpackage = detail::makeResolvableFromImpl( nvrad, impl ); + _store.insert( srcpackage ); + } + else if ( kind == ResTraits<Product>::kind ) + { + ResImplTraits<cached::ProductImpl>::Ptr impl = new cached::ProductImpl(id, this); + Product::Ptr product = detail::makeResolvableFromImpl( nvrad, impl ); + _store.insert( product ); + } + else if ( kind == ResTraits<Pattern>::kind ) + { + ResImplTraits<cached::PatternImpl>::Ptr impl = new cached::PatternImpl(id, this); + Pattern::Ptr pattern = detail::makeResolvableFromImpl( nvrad, impl ); + _store.insert( pattern ); + } + else if ( kind == ResTraits<Patch>::kind ) + { + ResImplTraits<cached::PatchImpl>::Ptr impl = new cached::PatchImpl(id, this); + Patch::Ptr patch = detail::makeResolvableFromImpl( nvrad, impl ); + _store.insert( patch ); + } + else if ( kind == ResTraits<Message>::kind ) + { + ResImplTraits<cached::MessageImpl>::Ptr impl = new cached::MessageImpl(id, this); + Message::Ptr message = detail::makeResolvableFromImpl( nvrad, impl ); + _store.insert( message ); + } + else if ( kind == ResTraits<Script>::kind ) + { + ResImplTraits<cached::ScriptImpl>::Ptr impl = new cached::ScriptImpl(id, this); + Script::Ptr script = detail::makeResolvableFromImpl( nvrad, impl ); + _store.insert( script ); + } + else if ( kind == ResTraits<Atom>::kind ) + { + ResImplTraits<cached::AtomImpl>::Ptr impl = new cached::AtomImpl(id, this); + Atom::Ptr atom = detail::makeResolvableFromImpl( nvrad, impl ); + _store.insert( atom ); + } +} + +static inline Resolvable::Kind +kind_for_name (const char *name, Id arch) +{ + switch (name[0]) + { + case 'a': + if (!strncmp (name, "atom:", 5)) + return ResTraits<Atom>::kind; + break; + case 'l': + if (!strncmp (name, "language:", 9)) + return ResTraits<Language>::kind; + break; + case 'm': + if (!strncmp (name, "message:", 8)) + return ResTraits<Message>::kind; + break; + case 'p': + if (!strncmp (name, "patch:", 6)) + return ResTraits<Patch>::kind; + if (!strncmp (name, "pattern:", 8)) + return ResTraits<Pattern>::kind; + if (!strncmp (name, "product:", 8)) + return ResTraits<Product>::kind; + break; + case 's': + if (!strncmp (name, "selection:", 10)) + return ResTraits<Selection>::kind; + if (!strncmp (name, "script:", 7)) + return ResTraits<Script>::kind; + break; + } + if (arch == ARCH_SRC) + return ResTraits<SrcPackage>::kind; + return ResTraits<Package>::kind; +} + +map<Id, Capability> id2cap; +map<Id, Edition> id2ed; +static const Rel flag2rel[8] = { + Rel::NONE, + Rel::GT, + Rel::EQ, + Rel::GE, + Rel::LT, + Rel::NE, + Rel::LE, + Rel::ANY +}; +static inline void +add_deps (CapFactory &factory, Pool *pool, CapSet &caps, Id *ids) +{ + Id id; + char buf[1024]; + if (!ids) + return; + while((id = *ids++) != 0) + { + if (id2cap.find(id) != id2cap.end()) + { + caps.insert (id2cap[id]); + } + else + { + const char *name = id2str (pool, id); + capability::CapabilityImpl::Ptr capi; + if (name[0] == '/') + { + capi = new capability::FileCap (ResTraits<Package>::kind, name); + } + else if (!strncmp (name, "hal(", 4)) + { + char * modname = strdup (name + 4); + if (strlen (modname)) + modname[strlen (modname) - 1] = 0; + capi = new capability::HalCap (ResTraits<SystemResObject>::kind, modname); + free (modname); + } + else if (!strncmp (name, "modalias(", 9)) + { + char * modname = strdup (name + 9); + if (strlen (modname)) + modname[strlen (modname) - 1] = 0; + capi = new capability::ModaliasCap (ResTraits<SystemResObject>::kind, modname); + free (modname); + } + else if (!strncmp (name, "filesystem(", 11)) + { + char * modname = strdup (name + 11); + if (strlen (modname)) + modname[strlen (modname) - 1] = 0; + capi = new capability::FilesystemCap (ResTraits<SystemResObject>::kind, modname); + free (modname); + } + else if (!ISRELDEP (id)) + { + Resolvable::Kind refer = kind_for_name (name, 1); + capi = new capability::NamedCap (refer, name); + } + else + { + Reldep *rd = GETRELDEP (pool, id); + Resolvable::Kind refer = kind_for_name (name, 1); + Rel rel = flag2rel[rd->flags & 7]; + Edition ed; + if (id2ed.find (rd->evr) != id2ed.end()) + ed = id2ed[rd->evr]; + else + { + /* ??? catch exception here? */ + ed = Edition (id2str (pool, rd->evr)); + id2ed[rd->evr] = ed; + } + capi = new capability::VersionedCap( refer, name, rel, ed ); + } + + //snprintf(buf, sizeof(buf), "%s%s%s", name, id2rel(pool, id), id2evr(pool, id)); + /* XXX Bah, we need the kind of the thing referred _to_ :-( + Do something about the kind stuff, s->name contains it, + but zypp wants to have it as extra field. */ + //cerr << "doing cap " << id << ": " << buf << endl; + //capability::CapabilityImpl::Ptr capi = capability::parse (refer, buf); + Capability cap = factory.fromImpl(capi); + id2cap[id] = cap; + caps.insert (cap); + } + } +} + +void RepoImpl::createResFromSource() +{ + Source *source = _options.source; + Pool *pool = source->pool; + int i; + cerr << "SOLV: have " << source->nsolvables << " solvables" << endl; + CapFactory capfactory; + for (i = source->start; i < source->start + source->nsolvables; i++) + { + Solvable *s = pool->solvables + i; + data::RecordId id(i); + Resolvable::Kind kind; + Edition ed; + const char *name = id2str (pool, s->name); + if (id2ed.find (s->evr) != id2ed.end()) + ed = id2ed[s->evr]; + else + { + /* ??? catch exception here? */ + /* XXX Edition ctor does a regexp if given a string :( */ + ed = Edition (id2str (pool, s->evr)); + id2ed[s->evr] = ed; + } + NVRAD nvrad ( name, + ed, + /* XXX Arch needs to become Id based. */ + Arch(id2str (pool, s->arch))); + add_deps (capfactory, pool, nvrad[Dep::PROVIDES], s->provides); + add_deps (capfactory, pool, nvrad[Dep::REQUIRES], s->requires); + add_deps (capfactory, pool, nvrad[Dep::CONFLICTS], s->conflicts); + add_deps (capfactory, pool, nvrad[Dep::OBSOLETES], s->obsoletes); + add_deps (capfactory, pool, nvrad[Dep::RECOMMENDS], s->recommends); + add_deps (capfactory, pool, nvrad[Dep::SUGGESTS], s->suggests); + add_deps (capfactory, pool, nvrad[Dep::FRESHENS], s->freshens); + add_deps (capfactory, pool, nvrad[Dep::ENHANCES], s->enhances); + add_deps (capfactory, pool, nvrad[Dep::SUPPLEMENTS], s->supplements); + //add_deps (capfactory, nvrad[Dep::PREREQUIRES], s->prerequires); + /* XXX do something about the kind stuff, s->name contains it, + but zypp wants to have it as extra field. */ + kind = kind_for_name (name, s->arch); + insertNVRAD (id, kind, nvrad); + } + //id2cap.clear(); +} + void RepoImpl::createResolvables() { ProgressData ticks; @@ -74,6 +303,12 @@ void RepoImpl::createResolvables() CombinedProgressData subprogrcv(ticks); debug::Measure m("create resolvables"); + if (_options.source) + { + createResFromSource(); + return; + } + CapFactory capfactory; try { Index: repo/cached/RepoImpl.h =================================================================== --- repo/cached/RepoImpl.h (revision 7438) +++ repo/cached/RepoImpl.h (working copy) @@ -25,6 +25,8 @@ #include "zypp/cache/ResolvableQuery.h" #include "zypp/RepoInfo.h" +#include "source_solv.h" + /////////////////////////////////////////////////////////////////// namespace zypp { ///////////////////////////////////////////////////////////////// @@ -44,6 +46,7 @@ namespace zypp : repoinfo(repoinfo_) , dbdir(dbdir_) , repository_id(repository_id_) + , source(0) {} @@ -52,6 +55,7 @@ namespace zypp RepoInfo repoinfo; Pathname dbdir; data::RecordId repository_id; + Source *source; }; /** @@ -81,6 +85,8 @@ namespace zypp data::RecordId repo_id, std::map<data::RecordId, std::pair<Resolvable::Kind, NVRAD> > &nvras, ProgressData &ticks ); + void insertNVRAD (data::RecordId id, Resolvable::Kind kind, const NVRAD &nvrad); + void createResFromSource(); cache::CacheTypes _type_cache; cache::ResolvableQuery _rquery; RepoOptions _options; Index: CMakeLists.txt =================================================================== --- CMakeLists.txt (revision 7438) +++ CMakeLists.txt (working copy) @@ -1158,6 +1158,7 @@ TARGET_LINK_LIBRARIES(zypp ${CURL_LIBRAR TARGET_LINK_LIBRARIES(zypp ${LIBXML_LIBRARY} ) TARGET_LINK_LIBRARIES(zypp ${SQLITE_LIBRARY} ) TARGET_LINK_LIBRARIES(zypp ${ZLIB_LIBRARY} ) +TARGET_LINK_LIBRARIES(zypp /matz/projects/solver/klaus/dev/src/.libs/libsatsolver.a) INSTALL(TARGETS zypp LIBRARY DESTINATION ${LIB_INSTALL_DIR} ) Index: RepoManager.cc =================================================================== --- RepoManager.cc (revision 7438) +++ RepoManager.cc (working copy) @@ -12,6 +12,7 @@ #include <iostream> #include <fstream> +#include <sstream> #include <list> #include <algorithm> #include "zypp/base/InputStream.h" @@ -39,6 +40,9 @@ #include "zypp/ZYppCallbacks.h" +#include "pool.h" +#include "source_solv.h" + using namespace std; using namespace zypp; using namespace zypp::repo; @@ -249,13 +253,14 @@ namespace zypp static shared_ptr<Impl> _nullimpl( new Impl ); return _nullimpl; } - + private: friend Impl * rwcowClone<Impl>( const Impl * rhs ); /** clone for RWCOW_pointer */ Impl * clone() const { return new Impl( *this ); } }; + /////////////////////////////////////////////////////////////////// /** \relates RepoManager::Impl Stream output */ @@ -825,6 +830,9 @@ namespace zypp return cache_status; } + Pool *the_pool; + map<data::RecordId, Source *> repo2source; + Repository RepoManager::createFromCache( const RepoInfo &info, const ProgressData::ReceiverFnc & progressrcv ) { @@ -841,12 +849,41 @@ namespace zypp MIL << "Repository " << info.alias() << " is cached" << endl; + if (!the_pool) + { + the_pool = pool_create (); + the_pool->verbose = 1; + } + data::RecordId id = store.lookupRepository(info.alias()); + map<data::RecordId,Source*>::const_iterator it = repo2source.find(id); + Source *source; + if (it == repo2source.end()) + { + Pathname name = _pimpl->options.repoCachePath; + ostringstream os; + os << id.get(); + name += os.str() + ".solv"; + source = 0; + //cerr << "BLA: " << name.asString() << endl; + if (PathInfo(name.asString()).isExist()) + { + FILE *fp = fopen (name.c_str(), "r"); + if (fp) + { + source = pool_addsource_solv(the_pool, fp, name.c_str()); + fclose (fp); + } + } + repo2source[id] = source; + } else + source = it->second; CombinedProgressData subprogrcv(progress); repo::cached::RepoOptions opts( info, _pimpl->options.repoCachePath, id ); opts.readingResolvablesProgress = subprogrcv; + opts.source = source; repo::cached::RepoImpl::Ptr repoimpl = new repo::cached::RepoImpl( opts ); -- To unsubscribe, e-mail: zypp-devel+unsubscribe(a)opensuse.org For additional commands, e-mail: zypp-devel+help(a)opensuse.org

3 3

[zypp-devel] Speedup libzypp a bit
by Michael Matz 01 Oct '07

01 Oct '07

Hi, here's my work from the weekend. Basically just using callgrind and fixing the most egregious sources for slowness. My testcase was "zypper lu", i.e. just reading in caches. Result: # LD_LIBRARY_PATH=../../inst/lib/ time ../../zypper/build/src/zypper lu ... * Reading repository 'http://ftp.gwdg.de/pub/linux/misc/packman/suse/10.2/' cache * Reading repository 'openSUSE-10.3-Updates' cache * Reading repository 'openSUSE-10.3-retail 10.3' cache * Reading installed packages [100%] No updates found. 12.82user 0.96system 0:14.07elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+85700minor)pagefaults 0swaps vs. # LD_LIBRARY_PATH=./zypp time ../../zypper/build/src/zypper lu * Reading repository 'http://ftp.gwdg.de/pub/linux/misc/packman/suse/10.2/' cache * Reading repository 'openSUSE-10.3-Updates' cache * Reading repository 'openSUSE-10.3-retail 10.3' cache * Reading installed packages [100%] No updates found. 8.51user 0.80system 0:09.52elapsed 97%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+69055minor)pagefaults 0swaps 4 seconds shoved off, not bad. The patch is against current libzypp svn. It does several things: 1) ctor of Dep is slow, hence the hunk in cache/CacheTypes.{cc,h} 2) VersionedCap::encode is very slow. encode is used by the hashing and for equality testing for the uset in CapFactory, ergo that too is slow. Can be sped up quite some by implementing own hash() and same() methods for the Capabilities. So, new virtual methods CapabilityImpl::hash CapabilityImpl::same using encode(), but overrode for VersionCap to for explicit comparison for same() and quicker hash computation for hash(). Accompanying changes in CapFactory to use these new methods. 3) regexp machinery is very slow. It's actually worthwhile to do a string test if the regexp could match at all, before even trying the regexp. In the profile only the two touched regex_match calls show up, one must have a ':/' (the SplitSpec one), the other at least a ' ' in the string to match. So test that before. Now not many regex calls are left for me. 4) The SQL query for reading in the named capabilities is very slow. It's actually faster to not connect name_id with name in the query, but instead read in the named table separately, and doing the connection name_id -> name by hand. Actually the code I wrote also added an "order by name_id", and made use of that, to spare many capfactory.fromImpl calls. It saves quite some amount of time, or better, it would save much time, if sqlite3 wouldn't be so slow with the sorting in the query. But unfortunately sqlite3 is so much slower with sorting that the saved number of calls to fromImpl() (75% can be saved) can't make that up :-( Anyway reading in the names table by hand still improves the time spent in sqlite3 quite much. I also noticed one more thing which needs quite some time and memory. Namely target/rpm/RpmPackageImpl.cc happily reads in the rpm changelog and filelist when constructing the PackageImpl. Even when not needed at all. The changelogs are usually not needed at all (certainly not for zypper lu and most other commands). The filelists are not needed either most of the time, certainly not in their completeness (i.e. they should at least be filtered against isInterestingFile). I removed reading filelist and changelogs from the rpm database and it saved about 40 MB of RAM (!). I don't send this hunk, because for completeness this should of course be read in lazily from the changelog() and filenames() methods, but that would need some refactoring to still have a handle to the database at that time. (Actually also the rpm database should be read in from a cache and not from the rpm database directly :-/ ) So, here it is. Have fun. If you find this usefull, I'm even offering to clean the patch up a bit. Oh, btw. there are not many more low hanging fruits anymore if the changelog and filelist problem is solved. It's now much time in sqlite3, and very much time spent all over the place just allocating objects and strings, moving, copying and comparing them. Ciao, Michael. Index: zypp/capability/CapabilityImpl.cc =================================================================== --- zypp/capability/CapabilityImpl.cc (revision 7408) +++ zypp/capability/CapabilityImpl.cc (working copy) @@ -10,6 +10,7 @@ * */ #include <iostream> +#include <ext/hash_fun.h> #include "zypp/base/Logger.h" #include "zypp/base/Regex.h" @@ -46,6 +47,18 @@ namespace zypp return encode() < rhs->encode(); } + std::size_t CapabilityImpl::hash() const + { + std::size_t ret = __gnu_cxx::hash<const char*>()( encode().c_str() ); + return ret; + } + + bool CapabilityImpl::same (const constPtr &rhs) const + { + /* refers and kind are known to be the same */ + return encode() == rhs->encode(); + } + /////////////////////////////////////////////////////////////////// // // METHOD NAME : CapabilityImpl::capImplOrderLess @@ -167,11 +180,14 @@ namespace zypp } //split: name:/absolute/path - static const str::regex rx( "([^/]*):(/.*)" ); - str::smatch what; - if( str::regex_match( name_r, what, rx ) ) + if (isSplitSpec (name_r)) { - return new capability::SplitCap( refers_r, what[1], what[2] ); + static const str::regex rx( "([^/]*):(/.*)" ); + str::smatch what; + if( str::regex_match( name_r, what, rx ) ) + { + return new capability::SplitCap( refers_r, what[1], what[2] ); + } } //name: name @@ -292,7 +308,8 @@ namespace zypp // improve regex! static const str::regex rx( "(.*[^ \t])([ \t]+)([^ \t]+)([ \t]+)([^ \t]+)" ); str::smatch what; - if( str::regex_match( strval_r,what, rx ) ) + if( strval_r.find(' ') != std::string::npos + && str::regex_match( strval_r,what, rx ) ) { Rel op; Edition edition; Index: zypp/capability/CapabilityImpl.h =================================================================== --- zypp/capability/CapabilityImpl.h (revision 7408) +++ zypp/capability/CapabilityImpl.h (working copy) @@ -110,6 +110,9 @@ namespace zypp virtual std::string index() const { return encode(); } + virtual size_t hash() const; + virtual bool same (const constPtr &rhs) const; + public: /** Solver hack. */ struct SplitInfo @@ -318,6 +321,14 @@ namespace zypp typedef std::set<CapabilityImpl::Ptr> CapabilityImplPtrSet; + inline bool exactly_same_caps_p (const CapabilityImpl::constPtr & lhs, + const CapabilityImpl::constPtr & rhs ) + { + return ( lhs->refers() == rhs->refers() + && lhs->kind() == rhs->kind() + && lhs->same(rhs) ); + } + ///////////////////////////////////////////////////////////////// } // namespace capability /////////////////////////////////////////////////////////////////// Index: zypp/capability/VersionedCap.cc =================================================================== --- zypp/capability/VersionedCap.cc (revision 7408) +++ zypp/capability/VersionedCap.cc (working copy) @@ -9,6 +9,8 @@ /** \file zypp/capability/VersionedCap.cc * */ +#include <iostream> +#include <ext/hash_fun.h> #include "zypp/capability/VersionedCap.h" using namespace std; @@ -35,6 +37,38 @@ namespace zypp return ret; } + bool VersionedCap::same (const CapabilityImpl_constPtr &_rhs) const + { + intrusive_ptr<const Self> rhs( asKind<Self>(_rhs) ); + if (!rhs) { + /* Gnah! VersionCap::kind() is "NamedCap", hence we might come + here with a NamedCap rhs, which then is not the same as us. */ + intrusive_ptr<const NamedCap> rhs2( asKind<NamedCap>(_rhs) ); + if (!rhs2) { + std::cerr << "Argh! : " << this->kind() << " <-> " << _rhs->kind() << std::endl; + return false; + } + return name() == rhs2->name(); + } + if (name() != rhs->name()) + return false; + const Edition::MatchRange & other = rhs->_range; + if (_range.op != other.op) + return false; + const Edition &ed1 = _range.value; + const Edition &ed2 = other.value; + return (ed1.epoch() == ed2.epoch() + && ed1.version() == ed2.version() + && ed1.release() == ed2.release()); + } + + size_t VersionedCap::hash() const + { + size_t ret = __gnu_cxx::hash<const char*>()( name().c_str() ); + ret ^= __gnu_cxx::hash<const char*>()( _range.value.version().c_str() ); + return ret; + } + std::string VersionedCap::index() const { return name(); } Index: zypp/capability/VersionedCap.h =================================================================== --- zypp/capability/VersionedCap.h (revision 7408) +++ zypp/capability/VersionedCap.h (working copy) @@ -56,6 +56,9 @@ namespace zypp /** Edition. */ virtual Edition edition () const; + virtual size_t hash() const; + virtual bool same (const CapabilityImpl_constPtr &rhs) const; + protected: /** Implementation dependent value. */ virtual const Edition::MatchRange & range() const; Index: zypp/cache/CacheTypes.cc =================================================================== --- zypp/cache/CacheTypes.cc (revision 7408) +++ zypp/cache/CacheTypes.cc (working copy) @@ -67,7 +67,8 @@ namespace zypp if ( klass == "kind" ) _kind_cache[id] = Resolvable::Kind(name); if ( klass == "deptype" ) - _deptype_cache[id] = name; + /* As Dep has no default ctor we can't use the [] accessor. */ + _deptype_cache.insert (make_pair (id, Dep(name))); } MIL << "archs: " << _arch_cache.size() << endl; @@ -130,9 +131,9 @@ namespace zypp Dep CacheTypes::deptypeFor( const data::RecordId &id ) { - std::map<data::RecordId, string>::const_iterator it; + std::map<data::RecordId, Dep>::const_iterator it; if ( (it = _deptype_cache.find(id) ) != _deptype_cache.end() ) - return Dep(it->second); + return it->second; else { ERR << "deptype: " << id << endl; @@ -142,10 +143,10 @@ namespace zypp data::RecordId CacheTypes::idForDeptype( const Dep & dep ) { - std::map<data::RecordId, string>::const_iterator it; + std::map<data::RecordId, Dep>::const_iterator it; for ( it = _deptype_cache.begin(); it != _deptype_cache.end(); ++it ) { - if ( dep.asString() == it->second ) + if ( dep == it->second ) return it->first; } ZYPP_THROW(Exception("Inconsistent deptype")); Index: zypp/cache/CacheTypes.h =================================================================== --- zypp/cache/CacheTypes.h (revision 7408) +++ zypp/cache/CacheTypes.h (working copy) @@ -136,7 +136,7 @@ namespace zypp std::map<data::RecordId, Rel> _rel_cache; std::map<data::RecordId, Resolvable::Kind> _kind_cache; - std::map<data::RecordId, std::string> _deptype_cache; + std::map<data::RecordId, Dep> _deptype_cache; std::map<data::RecordId, Arch> _arch_cache; Pathname _dbdir; }; Index: zypp/repo/cached/RepoImpl.cc =================================================================== --- zypp/repo/cached/RepoImpl.cc (revision 7408) +++ zypp/repo/cached/RepoImpl.cc (working copy) @@ -333,7 +333,7 @@ void RepoImpl::read_capabilities( sqlite // // } // } - sqlite3_command select_named_cmd( con, "select v.refers_kind, n.name, v.version, v.release, v.epoch, v.relation, v.dependency_type, v.resolvable_id from named_capabilities v, names n, resolvables res where v.name_id=n.id and v.resolvable_id=res.id and res.repository_id=:repo_id;"); + sqlite3_command select_named_cmd( con, "select v.refers_kind, v.name_id, v.version, v.release, v.epoch, v.relation, v.dependency_type, v.resolvable_id from named_capabilities v, resolvables res where res.repository_id=:repo_id and v.resolvable_id=res.id;"); sqlite3_command select_file_cmd( con, "select fc.refers_kind, dn.name, fn.name, fc.dependency_type, fc.resolvable_id from file_capabilities fc, files f, dir_names dn, file_names fn, resolvables res where f.id=fc.file_id and f.dir_name_id=dn.id and f.file_name_id=fn.id and fc.resolvable_id=res.id and res.repository_id=:repo_id;"); @@ -348,6 +348,16 @@ void RepoImpl::read_capabilities( sqlite sqlite3_command select_other_cmd( con, "select oc.refers_kind, oc.value, oc.dependency_type, oc.resolvable_id from other_capabilities oc, resolvables res where oc.resolvable_id=res.id and res.repository_id=:repo_id;"); + std::map<int,std::string> namemap; + sqlite3_command get_names_cmd( con, "select id,name from names;" ); + { + sqlite3_reader reader = get_names_cmd.executereader(); + while ( reader.read() ) + { + namemap[reader.getint(0)] = reader.getstring(1); + } + } + { debug::Measure mnc("read named capabilities"); select_named_cmd.bind(":repo_id", repo_id); @@ -355,27 +365,59 @@ void RepoImpl::read_capabilities( sqlite // FIXME Move this logic to tick()? Date start(Date::now()); + Capability oldcap; + Resolvable::Kind oldrefer; + Rel oldrel; + //std::string oldname; + int oldname = -1; + std::string oldver, oldrelease; + int oldepoch = 0; + + while ( reader.read() ) { ticks.tick(); Resolvable::Kind refer = _type_cache.kindFor(reader.getint(0)); + int name = reader.getint(1); Rel rel = _type_cache.relationFor(reader.getint(5)); - data::RecordId rid = reader.getint64(7); - if ( rel == zypp::Rel::NONE ) { - capability::NamedCap *ncap = new capability::NamedCap( refer, reader.getstring(1) ); - zypp::Dep deptype = _type_cache.deptypeFor(reader.getint(6)); - nvras[rid].second[deptype].insert( capfactory.fromImpl( capability::CapabilityImpl::Ptr(ncap) ) ); + if (oldname != name || rel != oldrel || refer!=oldrefer) + { + oldrel = rel; + oldrefer = refer; + oldname = name; + capability::NamedCap *ncap = new capability::NamedCap( refer, namemap[name] ); + oldcap = capfactory.fromImpl ( capability::CapabilityImpl::Ptr(ncap) ); + } } else { - capability::VersionedCap *vcap = new capability::VersionedCap( refer, reader.getstring(1), /* rel */ rel, Edition( reader.getstring(2), reader.getstring(3), reader.getint(4) ) ); - zypp::Dep deptype = _type_cache.deptypeFor(reader.getint(6)); - nvras[rid].second[deptype].insert( capfactory.fromImpl( capability::CapabilityImpl::Ptr(vcap) ) ); + std::string ver = reader.getstring(2); + std::string release = reader.getstring(3); + int epoch = reader.getint(4); + if (oldname != name || rel != oldrel || refer!=oldrefer + || oldver != ver + || oldrelease != release + || oldepoch != epoch) + { + oldrel = rel; + oldrefer = refer; + oldname = name; + oldver = ver; + oldrelease = release; + oldepoch = epoch; + + capability::VersionedCap *vcap = new capability::VersionedCap( refer, namemap[name], /* rel */ rel, Edition( ver, release, epoch ) ); + oldcap = capfactory.fromImpl( capability::CapabilityImpl::Ptr(vcap) ); + } } + + zypp::Dep deptype = _type_cache.deptypeFor(reader.getint(6)); + data::RecordId rid = reader.getint64(7); + nvras[rid].second[deptype].insert(oldcap); } } Index: zypp/CapFactory.cc =================================================================== --- zypp/CapFactory.cc (revision 7408) +++ zypp/CapFactory.cc (working copy) @@ -38,7 +38,8 @@ namespace { size_t operator() ( const CapabilityImpl::Ptr & p ) const { - return __gnu_cxx::hash<const char*>()( p->encode().c_str() ); + //return __gnu_cxx::hash<const char*>()( p->encode().c_str() ); + return p->hash(); } }; @@ -46,9 +47,10 @@ namespace { bool operator() ( const CapabilityImpl::Ptr & lhs, const CapabilityImpl::Ptr & rhs ) const { - return ( lhs->encode() == rhs->encode() - && lhs->kind() == rhs->kind() - && lhs->refers() == rhs->refers() ); + return ::zypp::capability::exactly_same_caps_p (lhs, rhs); + /*return ( lhs->refers() == rhs->refers() + && lhs->kind() == rhs->kind() + && lhs->encode() == rhs->encode() );*/ } }; -- To unsubscribe, e-mail: zypp-devel+unsubscribe(a)opensuse.org For additional commands, e-mail: zypp-devel+help(a)opensuse.org

2 1