[yast-devel] [RFC] QueryBuilder and (Resolvable)Query
Hi! Here is my proposal of a new libzypp API for various metadata queries. It should be useful for YaST package managers, zypper, YaST online update and maybe others. Pease see attached QueryBuilder.h file and comment on the design and usability. QueryBuilder is meant to build an sqlite query for libzypp's metadata cache database and to be used by ResolvableQuery::query(QueryBuilder,ProcessingFunctor) function (or other Query class) to do complex resolvable data queries. Result of these queries will be processed by ProcessingFunctor (read on). The reason to create such an API is to have a convenient API for coding various kinds of user queries, so that apps using libzypp don't have to code the functionality by themselves by traversing the resolvable pool. And since we (will hopefully) have the new metadata cache database, there is another reason: to make use of the database to speed up the search and save memory requirements (don't load the pool, just query the database). I would have to code it once in zypper anyway, so this way it can be reused by others as well. CODE EXAMPLE: //////// code START ///////////////////////////////////////// bool is_regexp; // treat the search string as regexp? bool search_summaries; // search also in summaries? string search_string; // user given search string Resolvable::Kind kind; // the result will be limited to this kind <snip> zypp::cache::QueryBuilder builder; builder.addName(search_string, is_regexp); if (search_summaries) builder.addAttribute(zypp::cache::attrResObjectSummary(), search_string, is_regexp); builder.addKind(kind); zypp::cache::ResolvableQuery resquery; resquery.query(builder, processing_functor); //////// code END ///////////////////////////////////////// (the functor is function<bool( const data::RecordId &, data::ResObject_Ptr )> we can provide one which would use Selectables (or NameKindProxy) for yast as well) FEATURES - search by name, any cache attribute, kind, (future: installed status), version - uses simple strings or regexes - search by dependencies (select resolvables which are required by e.g. pattern X) - use multiple search conditions and choose to require all or any of them satisfied. IMPLEMENTATION - based on given arguments and options, an sqlite command will be built This command will be executed by a ResolvableQuery::query() or another Query class (see attachment - just a stub). - i decided to write field values into the sqlite command string itself. Using variable names and bind values to them would be too complex. - regexes - sqlite's REGEXP keyword will be used, bound to user (libzypp) defined regexp matching function of our liking. ISSUES - what regexp syntax do we want to support? - something missing/to be changed in the API? Cheers, Jano
On Friday 29 June 2007 15:14, Jan Kupec wrote:
Here is my proposal of a new libzypp API for various metadata queries. It should be useful for YaST package managers, zypper, YaST online update and maybe others. Pease see attached QueryBuilder.h file and comment on the design and usability.
QueryBuilder is meant to build an sqlite query for libzypp's metadata cache database and to be used by ResolvableQuery::query(QueryBuilder,ProcessingFunctor) function (or other Query class) to do complex resolvable data queries. Result of these queries will be processed by ProcessingFunctor (read on).
The reason to create such an API is to have a convenient API for coding various kinds of user queries, so that apps using libzypp don't have to code the functionality by themselves by traversing the resolvable pool.
That's a nice API for querying resolvables. Will there be a counterpart for querying selectables (which is what the UI needs), too? CU -- Stefan Hundhammer <sh@suse.de> Penguin by conviction. YaST2 Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Nürnberg, Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org For additional commands, e-mail: yast-devel+help@opensuse.org
Stefan Hundhammer wrote:
On Friday 29 June 2007 15:14, Jan Kupec wrote:
Here is my proposal of a new libzypp API for various metadata queries. It should be useful for YaST package managers, zypper, YaST online update and maybe others. Pease see attached QueryBuilder.h file and comment on the design and usability.
QueryBuilder is meant to build an sqlite query for libzypp's metadata cache database and to be used by ResolvableQuery::query(QueryBuilder,ProcessingFunctor) function (or other Query class) to do complex resolvable data queries. Result of these queries will be processed by ProcessingFunctor (read on).
The reason to create such an API is to have a convenient API for coding various kinds of user queries, so that apps using libzypp don't have to code the functionality by themselves by traversing the resolvable pool.
That's a nice API for querying resolvables. Will there be a counterpart for querying selectables (which is what the UI needs), too?
Sure! Note the comment in parenthesis after the code example: (the functor is function<bool( const data::RecordId &, data::ResObject_Ptr )> we can provide one which would use Selectables (or NameKindProxy) for yast as well) I hope it will be doable in a resource-sparing way. Maybe by adding name and kind to the 'order by' clause of the query so that the resolvables belonging to one selectable can be collected as the result is read. Anyway, the data::RecordId should probably better be hidden, so the functions might look like: function<bool(const zypp::Resolvable::Ptr &)> and function<bool(const zypp::ui::Selectable::Ptr &)> for YaST j. -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org For additional commands, e-mail: yast-devel+help@opensuse.org
On Mon, Jul 02, Jan Kupec wrote:
Stefan Hundhammer wrote:
On Friday 29 June 2007 15:14, Jan Kupec wrote:
Here is my proposal of a new libzypp API for various metadata queries. It should be useful for YaST package managers, zypper, YaST online update and maybe others. Pease see attached QueryBuilder.h file and comment on the design and usability.
QueryBuilder is meant to build an sqlite query for libzypp's metadata cache database and to be used by ResolvableQuery::query(QueryBuilder,ProcessingFunctor) function (or other Query class) to do complex resolvable data queries. Result of these queries will be processed by ProcessingFunctor (read on).
The reason to create such an API is to have a convenient API for coding various kinds of user queries, so that apps using libzypp don't have to code the functionality by themselves by traversing the resolvable pool.
That's a nice API for querying resolvables. Will there be a counterpart for querying selectables (which is what the UI needs), too?
Sure! Note the comment in parenthesis after the code example:
IMO Huha is looking for something, the 'UI' will not find here. Neither with the existing query API, nor with Janos proposal. A better integration of Selectables into the Pool. (I tried to describe it as well in an earlier mail: 29 Jun; Re: [yast-devel] Adding missing API to libzypp (or between libzypp andapps) ) A Selectable (and NameKindProxy too) is a container of PoolItems, all of the same kind and name. E.g: ALL known Packages named glibc will be available in the 'NameKindProxy(Package,glibc)'. A 'Selectable(Package,glibc)' will contain at most ONE installed, and all available glibc packages. ;( There is no difference, unless more than one version of a package is installed. In that case there are as many Selectables as there are versions installed. Each Selectable links to only ONE installed but to ALL available versions. Unfortunately we support this (e.g. multiple kernel versions being installed at the same time). If we query for some property (like 'provides glibc = 2.5'), the query usually leads to individual Packages (or PoolItems). Each PoolItem maps one-to-one to a NameKindProxy, but not to a Selectable. That's why we could easily implement a backlink from each PoolItem to it's corresponding NameKindProxy. But not to a Selectable. If the UI would move, and use a Selectable that actually links to ALL known instances of a certain (name,kind), we could start to implement a backlink from each PoolItem to it's coresponding Selectable. That way the UI could immediately benefit from each query zypp provides, because the PoolItem maps to a UI Selectable. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org For additional commands, e-mail: yast-devel+help@opensuse.org
Hola!
A 'Selectable(Package,glibc)' will contain at most ONE installed, and all available glibc packages. ;(
There is no difference, unless more than one version of a package is installed. In that case there are as many Selectables as there are versions installed. Each Selectable links to only ONE installed but to ALL available versions.
Unfortunately we support this (e.g. multiple kernel versions being installed at the same time).
My apologies for the dumb question *), but if the concept of selectables has so many drawbacks, why did we decided to use it? Why didn't we switch to PoolItems instead? Can anyone (Michael, HuHa,...) comment on that in some human-readable, easy-to-understand way? ;-) Thanks B. (*) read: I've joined YaST team only a year ago, already after package management rewrite -- \\\\\ Katarina Machalkova \\\\\\\__o YaST developer __\\\\\\\'/_ & hedgehog painter
On Tue, Jul 03, Katarina Machalkova wrote:
Hola!
A 'Selectable(Package,glibc)' will contain at most ONE installed, and all available glibc packages. ;(
There is no difference, unless more than one version of a package is installed. In that case there are as many Selectables as there are versions installed. Each Selectable links to only ONE installed but to ALL available versions.
Unfortunately we support this (e.g. multiple kernel versions being installed at the same time).
My apologies for the dumb question *), but if the concept of selectables has so many drawbacks, why did we decided to use it? Why didn't we switch to PoolItems instead? Can anyone (Michael, HuHa,...) comment on that in some human-readable, easy-to-understand way?
Historical. You may remember the first libzypp was a bit late, so there was little time to migrate the UI from using the old y2pm to zypp. In the old y2pm the status information was kept in a 'Selectable'. This was possible, because the old y2pm neither supported multiple installed nor multiple available versions. There was at most one installed and one available item and the status. Now we may have multiple installed and multiple available versions. And each item has it's individual status. We tried to find some 'Selectable' that allowed to reuse as much of the existing UI code as possible. An Item that mangles the individual stati into one UI-status. And we found one, but it has some drawbacks. The concept of a "Selectable" is right. An easy view and access to the pool. Pkg-bindings and zypper would like to use something like this as well. Changing (fixing) the ui::Selectable will lead to changes at the UI. But it would allow us to bring PoolItems and Selectables closer together. Or we had to invent something in parallel to the ui::Selectable for Pkg-bindings and zypper. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org For additional commands, e-mail: yast-devel+help@opensuse.org
On Tuesday 03 July 2007 11:58, Michael Andres wrote:
Historical. You may remember the first libzypp was a bit late, so there was little time to migrate the UI from using the old y2pm to zypp.
No. Not at all historical. The concept of "selectables" was the foundation of the package selection user interface. As a user, you don't want to deal with a dozen list items with the same name (which is becoming a more and more realistic scenario with the ever-increasing number of active installation sources). You don't want to see something like [ ] emacs 21.3-263.1 [ ] emacs 21.3-263.1 [x] emacs 21.3-263.2 [ ] emacs 21.3-263.3 [ ] emacs 21.3-263.3 [ ] emacs 21.3-263.4 just because you happen to have 5 active installation sources that each provide an emacs package, plus your installed system that also has an emacs package installed. The selectable is what groups those resolvables of the same kind so we can display ONE list items for "emacs". Yes, of course, we could move that to the UI, too. We could let the UI iterate over all packages and group those together that have the same name. Thus we could move the handling of selectables to the UI level. That's basicaly what the current zypp::ui::selectable implementation does. And that's also the reason why there is no back-pointer, i.e. no way to move to a resolvable's selectable parent. That's why we had to change EVERY SINGLE UI function's API so they now accept both a zypp::ui::SelectablePtr and a zypp::ui::ResolvablePtr. And of course it can only be hoped that they never get out of sync. The old package manager OTOH had that data structure in its core. A PMSelectable had a list of PMResolvables, and each PMResolvable had a "parent" pointer that pointed to its PMSelectable. Thus, one pointer was enough to handle everything. CU -- Stefan Hundhammer <sh@suse.de> Penguin by conviction. YaST2 Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Nürnberg, Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org For additional commands, e-mail: yast-devel+help@opensuse.org
On Tue, Jul 03, Stefan Hundhammer wrote:
The selectable is what groups those resolvables of the same kind so we can display ONE list items for "emacs".
Yes, of course, we could move that to the UI, too. We could let the UI iterate over all packages and group those together that have the same name. Thus we could move the handling of selectables to the UI level.
That's basicaly what the current zypp::ui::selectable implementation does. And that's also the reason why there is no back-pointer, i.e. no way to move to a
IMO not. If we'd group "ALL resolvables of the same kind and name" into ONE selectable, we could easily implement the back-pointer: 1) Pool Selectable i:kernel-1.0 ---+ i:kernel-1.2 ---+ +---- Selectable(package,kernel) a:kernel-1.2 ---+ a:kernel-2.0 ---+ But it is done that way: Pool Selectables i:kernel-1.0 ---+---- S1(package,kernel) ----+ | i:kernel-1.2 ---+---- S2(package,kernel) --+ | | | a:kernel-1.2 ---+--------------------------+-+ a:kernel-2.0 ---+ One selectable for each installed version, ignoring the presence of other installed versions. Here we had to implement back-pointer lists. Here the selectable does not neccessarily contain all relevant information about the resolvables (S1 is not aware of kernel-1.2). This kind of grouping is IMO not helpfull outside the UI. If the UI could use a structure like in 1) we could advance. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org For additional commands, e-mail: yast-devel+help@opensuse.org
If multiple installed resolvables in one selectable is the major problem, I think we can go ahead and redefine the selectables like that. So far, it is a very rare case that multiple instances of the same package are installed at the same time. As a matter of fact, the kernel packages seem to be the only ones that are (currently) even capable of that; all other packages would get a zillion file conflicts right away. So, let's not make that pathological case the norm that prevents all further improvements. Let's have a selectable multiple installed instances. BUT... (you had that coming, didn't you? ;-) ) We should change the API gently. There should still be a call that returns the first installed instance. Currently, zypp::ui::Selectable::installedObj() returns the one and only installed instance. Let's change this to zypp::ui::Selectable::firstInstalledObj() Let's also add zypp::ui::Selectable::installedObjCount() that returns the number (a plain int, not weird typedef'ed-to-death type) of installed instances of that selectable. The UI will have to handle the case of installedObjCount() > 1 differently. But since it will affect only a couple of packages, this will be tolerable IMHO. I could imagine several scenarios: (1) You can't simply change the status of such a selectable as with a simple package where you have one (or none) installed and a number of instances available. Rather, you'd (1a) either have to use the "Versions" tab or (1b) (better? worse? dunno yet) a popup opens when you click on that status icon where you can do more sophisticated things with all the available and installed instances. Of course, we'd need another special icon (and internally a status "multi-installed") to tell the user. (2) The UI does the special handling for such selectables in a similar way like today: Create multiple list entries, one for each installed instance. CU -- Stefan Hundhammer <sh@suse.de> Penguin by conviction. YaST2 Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Nürnberg, Germany -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org For additional commands, e-mail: yast-devel+help@opensuse.org
* Michael Andres <ma@suse.de> [Jul 03. 2007 11:54]: [...]
The concept of a "Selectable" is right. An easy view and access to the pool. Pkg-bindings and zypper would like to use something like this as well.
And also the dependency solver. Klaus -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org For additional commands, e-mail: yast-devel+help@opensuse.org
participants (5)
-
Jan Kupec -
Katarina Machalkova -
Klaus Kaempf -
Michael Andres -
Stefan Hundhammer