[zypp-devel] my plan: SOLV files for libzypp
Aloha,
for your viewing pleasure here are my weekend hacks (and this time it's
really really horrible, don't look at them) to make libzypp use SOLV files
instead of sqlite and rpm database as backends. In it's current form
quite some information about the resolvable details are missing (no
description, no summary, no changelog, nothing except what a NVRAD
contains). The connection between the resolvable and its SQLite ID is
lost too (hence zypper if bash will give a funny summary/description from
a different resolvable). The category of patches is lost (because that's
a simple text attribute), as are many other things.
And it doesn't automatically update the .solv files with the repositories,
you have to do that by hand (files are /var/cache/zypp/$ID.solv, where ID
is the repository ID from the sqlite repositories table). Except for the
rpmdatabase shadow, that's kept in sync (in /tmp/system.solv, I wanted to
develop as user, who can't write in /var/cache/zypp).
And it duplicates code. And it doesn't do error checking. And it kill
kittens, when someone reads it.
The nice thing is, that even without underlying changes in libzypp I get
my testcase of "zypper lu" down to 5 seconds, and in the profiles it's now
realitively obvious where to improve next.
If we go the route of .solv files (and we should) we can make much of the
underlying representation of Edition/Atom/Name/Caps and many other things
be just Id, i.e. an integer. CapSets will just be zero-terminated lists
of Ids. That could be implemented directly in the zypp classes, instead
of the current insane Impl indirection, saving memory and new calls
(currently we do about 6 millions (!)).
I'm personally planning to do more about this. In particular working with
mls on extending the .solv format a bit to store also some other
information about resolvables.
I haven't yet made up my mind if the SQLite database will stay or not.
It's quite nice to have for some information (as it's easily extensible),
but the necessity to then maintain a mapping from resolvable-ID (in
zypp.db) to source/resolvable-id (in SOLV) makes this less attractive.
Anyway, so you know about my plans, here's my current
patch^Whack^Wunbelievable bloody crap.
Ciao,
Michael.
--
Index: target/rpm/RpmPackageImpl.cc
===================================================================
--- target/rpm/RpmPackageImpl.cc (revision 7438)
+++ target/rpm/RpmPackageImpl.cc (working copy)
@@ -48,17 +48,45 @@ RPMPackageImpl::RPMPackageImpl(
_license(data->tag_license()),
_packager(data->tag_packager()),
_group(data->tag_group()),
- _changelog(data->tag_changelog()),
+ //_changelog(data->tag_changelog()),
+ _changelog(Changelog()),
_type("rpm"), // FIXME in the future
- _filenames(data->tag_filenames()),
+ //_filenames(data->tag_filenames()),
+ _filenames(std::liststd::string()),
_size(data->tag_size())
{
// we know we are reading english.
- _description.setText(data->tag_description(), Locale("en"));
+ //_description.setText(data->tag_description(), Locale("en"));
+ _description.setText(std::string(), Locale("en"));
data->tag_du(_disk_usage);
_location.setDownloadSize(data->tag_archivesize());
}
+RPMPackageImpl::RPMPackageImpl()
+ : _summary("XXXXXXXXXXX", Locale("en")),
+ _description("XXX"),
+ _buildtime(Date()),
+ _installtime(Date()),
+ _buildhost(""),
+ _url(""),
+ _vendor(""),
+ _license(""),
+ _packager(""),
+ _group(PackageGroup()),
+ //_changelog(data->tag_changelog()),
+ _changelog(Changelog()),
+ _type("rpm"), // FIXME in the future
+ //_filenames(data->tag_filenames()),
+ _filenames(std::liststd::string()),
+ _size(0)
+{
+ // we know we are reading english.
+ //_description.setText(data->tag_description(), Locale("en"));
+ _description.setText(std::string(), Locale("en"));
+ //data->tag_du(_disk_usage);
+ //_location.setDownloadSize(data->tag_archivesize());
+}
+
/** Package summary */
TranslatedText RPMPackageImpl::summary() const
{
@@ -128,6 +156,7 @@ PackageGroup RPMPackageImpl::group() con
/** */
Changelog RPMPackageImpl::changelog() const
{
+std::cerr << "Argh" << std::endl;
return _changelog;
}
@@ -189,6 +218,7 @@ ByteCount RPMPackageImpl::sourcesize() c
/** */
std::liststd::string RPMPackageImpl::filenames() const
{
+//std::cerr << "Argh2" << std::endl;
return _filenames;
}
Index: target/rpm/RpmPackageImpl.h
===================================================================
--- target/rpm/RpmPackageImpl.h (revision 7438)
+++ target/rpm/RpmPackageImpl.h (working copy)
@@ -40,6 +40,8 @@ public:
const RpmHeader::constPtr data
);
+ RPMPackageImpl();
+
/** Package summary */
virtual TranslatedText summary() const;
/** Package description */
Index: target/rpm/RpmDb.cc
===================================================================
--- target/rpm/RpmDb.cc (revision 7438)
+++ target/rpm/RpmDb.cc (working copy)
@@ -28,11 +28,15 @@
#include "zypp/base/String.h"
#include "zypp/base/Regex.h"
+#include "zypp/data/RecordId.h"
+
#include "zypp/Date.h"
#include "zypp/Pathname.h"
#include "zypp/PathInfo.h"
#include "zypp/PublicKey.h"
+#include "zypp/capability/Capabilities.h"
+
#include "zypp/target/rpm/RpmDb.h"
#include "zypp/target/rpm/RpmCallbacks.h"
@@ -55,6 +59,7 @@ using namespace zypp::filesystem;
namespace zypp
{
+extern Pool * the_pool;
namespace target
{
namespace rpm
@@ -277,6 +282,7 @@ public:
RpmDb::RpmDb()
: _dbStateInfo( DbSI_NO_INIT )
, _packages( * new Packages ) // delete in destructor
+ , source(0)
#warning Check for obsolete memebers
, _backuppath ("/var/adm/backup")
, _packagebackups(false)
@@ -1254,6 +1260,117 @@ Package::Ptr RpmDb::makePackageFromHeade
return pptr;
}
+static bool
+update_solv (string dbpath)
+{
+ cerr << "trying " << dbpath << endl;
+ if (dbpath != "//var/lib/rpm" && dbpath != "/var/lib/rpm")
+ return false;
+ cerr << "Building system.solv" << endl;
+ system ("if test -f /tmp/system.solv; then /matz/projects/solver/klaus/dev/tools/rpmdb2solv /tmp/system.solv > /tmp/system.solv.new; else /matz/projects/solver/klaus/dev/tools/rpmdb2solv > /tmp/system.solv.new; fi; mv /tmp/system.solv.new /tmp/system.solv");
+ return true;
+}
+
+static inline Resolvable::Kind
+kind_for_name (const char *name, Id arch)
+{
+ switch (name[0])
+ {
+ case 'a':
+ if (!strncmp (name, "atom:", 5))
+ return ResTraits<Atom>::kind;
+ break;
+ case 'l':
+ if (!strncmp (name, "language:", 9))
+ return ResTraits<Language>::kind;
+ break;
+ case 'm':
+ if (!strncmp (name, "message:", 8))
+ return ResTraits<Message>::kind;
+ break;
+ case 'p':
+ if (!strncmp (name, "patch:", 6))
+ return ResTraits<Patch>::kind;
+ if (!strncmp (name, "pattern:", 8))
+ return ResTraits<Pattern>::kind;
+ if (!strncmp (name, "product:", 8))
+ return ResTraits<Product>::kind;
+ break;
+ case 's':
+ if (!strncmp (name, "selection:", 10))
+ return ResTraits<Selection>::kind;
+ if (!strncmp (name, "script:", 7))
+ return ResTraits<Script>::kind;
+ break;
+ }
+ if (arch == ARCH_SRC)
+ return ResTraits<SrcPackage>::kind;
+ return ResTraits<Package>::kind;
+}
+
+map
* Michael Matz
Aloha,
for your viewing pleasure here are my weekend hacks (and this time it's really really horrible, don't look at them) to make libzypp use SOLV files instead of sqlite and rpm database as backends.
Hey, thanks a lot and welcome to the zypp coders team ! :-)
And it duplicates code. And it doesn't do error checking. And it kill kittens, when someone reads it.
Too bad you included the patch in the mail to zypp-devel where we have approx. 20 subscribers and hence 20 dead kittens now. [...]
I'm personally planning to do more about this. In particular working with mls on extending the .solv format a bit to store also some other information about resolvables.
Participation and help is very welcome and you're invited to share your plans with us so we can plan and organize future work.
I haven't yet made up my mind if the SQLite database will stay or not. It's quite nice to have for some information (as it's easily extensible), but the necessity to then maintain a mapping from resolvable-ID (in zypp.db) to source/resolvable-id (in SOLV) makes this less attractive.
The current thinking on this topic is to keep the database for on-demand retrieval of UI information (summary, description, license, ...) and for extended search.
Anyway, so you know about my plans, here's my current patch^Whack^Wunbelievable bloody crap.
Does it compile ? Does it run ? Lets ship it ! ;-) Klaus --- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Hi, On Mon, 8 Oct 2007, Klaus Kaempf wrote:
Too bad you included the patch in the mail to zypp-devel where we have approx. 20 subscribers and hence 20 dead kittens now.
Yeah, too bad. I don't like kittens anyway.
I haven't yet made up my mind if the SQLite database will stay or not. It's quite nice to have for some information (as it's easily extensible), but the necessity to then maintain a mapping from resolvable-ID (in zypp.db) to source/resolvable-id (in SOLV) makes this less attractive.
The current thinking on this topic is to keep the database for on-demand retrieval of UI information (summary, description, license, ...) and for extended search.
Yeah, along those lines I also think. Though I'd like to experiment with some more things put into .solv files, e.g. license and summary (certainly not description or, heaven forbid, changelog, though even for them I'd like to have some better on-disk form, at least chunk-wise compressed). At least all info "important" for solving should be in there, which also includes IMHO the category of patches and the like.
Anyway, so you know about my plans, here's my current patch^Whack^Wunbelievable bloody crap.
Does it compile ? Does it run ? Lets ship it ! ;-)
;-) It actually compiles (against a libzypp svn from, hmm, friday I think), but you need changes to the CMakeLists.txt files for INCLUDE_DIRECTORIES and TARGET_LINK_LIBRARIES. And then you need to create the .solv files by hand. As root, do: % cd /var/cache/zypp % sqlite3 zypp.db 'select id,alias from repositories' 1|openSUSE-10.3-retail 10.3 28|openSUSE-10.3-Updates 29|http://ftp.gwdg.de/pub/linux/misc/packman/suse/10.2/ Now create 1.solv, 28.solv and 29.solv (for me) from the respective raw repodata in the raw/ subdirs: % zcat /var/cache/zypp/raw/openSUSE-10.3-retail\ 10.3/suse/setup/descr/packages.gz | susetags2solv > 1.solv % ( echo "<patches>" ; cat raw/openSUSE-10.3-Updates/repodata/patch-*.xml; echo "</patches>" ) | grep -v ' 28.solv % zcat raw/http\:/ftp.gwdg.de/pub/linux/misc/packman/suse/10.2/repodata/primary.xml.gz | rpmmd2solv > 29.solv The patchxml2solv needs a patch from me, and it's broken as the primary.xml.gz in that repo is not used. Now zypper with libzypp on that patch will take up these .solv files instead of zypp.db. And hence will probably break all over, but at least: % time LD_LIBRARY_PATH=./zypp ../../zypper/build/src/zypper lu Repository 'http://ftp.gwdg.de/pub/linux/misc/packman/suse/10.2/' is out-of-date. You can run 'zypper refresh' as root to update it. Repository 'openSUSE-10.3-retail 10.3' is out-of-date. You can run 'zypper refresh' as root to update it. SOLV: have 6813 solvables SOLV: have 129 solvables SOLV: have 15284 solvables * Reading installed packages [0%]trying /var/lib/rpm Building system.solv SOLV: have 1449 solvables Repository: | Name | Version | Category | Status ----------------------+-------------------------+---------+----------+------- openSUSE-10.3-Updates | patch:amarok | 4492-0 | | Needed openSUSE-10.3-Updates | patch:cpio | 4474-0 | | Needed openSUSE-10.3-Updates | patch:evince | 4465-0 | | Needed openSUSE-10.3-Updates | patch:fetchmsttfonts.sh | 4347-0 | | Needed openSUSE-10.3-Updates | patch:fvwm2 | 4475-0 | | Needed openSUSE-10.3-Updates | patch:glibc | 4467-0 | | Needed openSUSE-10.3-Updates | patch:gtk2 | 4466-0 | | Needed openSUSE-10.3-Updates | patch:ksh | 4489-0 | | Needed openSUSE-10.3-Updates | patch:release-notes | 4464-0 | | Needed openSUSE-10.3-Updates | patch:wvdial | 4461-0 | | Needed real 0m6.835s user 0m5.680s sys 0m0.896s Ciao, Michael. -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Mon, Oct 08, 2007 at 11:40:21AM +0200, Michael Matz wrote:
for your viewing pleasure here are my weekend hacks (and this time it's really really horrible, don't look at them) to make libzypp use SOLV files instead of sqlite and rpm database as backends. In it's current form
What are SOLV files anyway?
And it doesn't automatically update the .solv files with the repositories, you have to do that by hand (files are /var/cache/zypp/$ID.solv, where ID is the repository ID from the sqlite repositories table). Except for the rpmdatabase shadow, that's kept in sync (in /tmp/system.solv, I wanted to develop as user, who can't write in /var/cache/zypp).
BTW in /etc/zypp/zypp.conf you can specify different paths for everything. We may be missing ~/.zypp.conf though.
Anyway, so you know about my plans, here's my current patch^Whack^Wunbelievable bloody crap.
Hey great, thanks! -- Martin Vidner, YaST developer http://en.opensuse.org/User:Mvidner Kuracke oddeleni v restauraci je jako fekalni oddeleni v bazenu -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
participants (3)
-
Klaus Kaempf
-
Martin Vidner
-
Michael Matz