[zypp-devel] Depsolver and C API for package management
Hi! We at Red Hat are currently looking for a C based depsolver and C API for package management. In addition we are considering a deeper integration of the depsolver with rpm - both on API and code level. While depsolvers and rpm do different things they are much more similar than the current code bases would suggest. We actually did a prototype implementation in Python of such an integrate design a few years ago [1] (based on a yum-like depsolver). For package management we need a simple API on top of the whole stack that is easy to use. Target audience are tools dealing with packages but not changing the way we deal with packages - like CLI or GUI installers/updaters, system management tools, ... This API needs to hide a lot of complexity and should only offer access to the installed and available packages, allow running simple commands and doing simple adjustments to the configuration (e.g. enable/disable repositories). For plugins and more complicated tools we will probably need another - more complicated - API that is closer to the metal. It would probably allow creating rules for the depsolver and would be dependent of the actual depsolver implementation. This API could be separate and come with lower expectations on stability. So one obvious thing for us to do was looking at libzypp and sat-solver / libsolv. While we have not yet settled to specific solution we already have some ideas and opinions. As there is a chance that you might do something similar and there is the chance to come to a more unified code base in the package management stack we want you to know what we have in mind early. Our conclusions so far: Libsolv is an interesting implementation for a depsolver but does not offer the API we are looking for. It needs some kind of abstraction layer to hide all the moving parts. This is one of the things libzypp does. The libzypp API looks pretty complicated. The "next-gen API for package management?" thread on the zypp-devel (September 2011) did not exactly dispel our concerns about that. As we need a C API using C++ creates a lot of overhead. If libzypp would provide the perfect API just putting a C layer in front would be a good way to go. But doing a lot of simplification within this step (as zpm [2]) will likely require tools needing access to "more internal" parts of the API to either be in C++ or wait for a new C layer being written. This is not a situation we are looking forward to. As we are not as attached to the functionality implemented in libzypp as you might be (we are not using Patches or Patterns) implementing the package management API in pure C is an option we are considering. This layer should also hide libsolv's implementation details from plugins or tools that interfere more deeply with the depsolver than "simple GUI applications". For tools implementing new features it is probably easier to do the implementation within libsolv and then export the new functionality than exporting an API allowing to do so. This kinda breaks the idea of different levels of abstraction but allows keeping the important code paths highly optimized - which is what libsolv obviously aims for. We are also thinking about using libsolv - or at least parts of libsolv for implementing functionality of rpm itself. First tests show that small but notable improvements can be achieved by swapping out data structures - even without replacing the whole mode of operation. We are also considering different approaches of integrating libsolv with the rpm dependency checker but even the basic steps are still undetermined. One possible model would be dropping rpmlib's dependency checker in favor of the sat solver and write some interfacing code. As a result of these considerations we are interested if your discussion about the "next-gen API" has led to further results since September and if you are already in favor of one of the suggested solutions. As the requirements are not fundamentally different for the different distributions there might be a chance of increasing the shared code base and may be even integrating *the* dep solver into RPM if this should prove to be beneficial. Florian [1] http://pyrpm.sourceforge.net/ [2] https://github.com/openSUSE/zpm -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
Hi Florian! On Tue, Jan 03, 2012 at 01:08:42PM +0100, Florian Festi wrote:
[...] Libsolv is an interesting implementation for a depsolver but does not offer the API we are looking for. It needs some kind of abstraction layer to hide all the moving parts. This is one of the things libzypp does.
Yes, the "bare bone" C part of libsolv is probably not what you're looking for. The usage of Ids instead of pointers is a tad confusing, e.g. a package is described by (pool,id). A pointer to a package can become invalid when another package is added and so on. The interface itself isn't moving that rapidly, I just used the sat-solver -> libsolv rename to get rid of some old cruft and do some function name cleanup. But did you look at the perl/python/ruby bindings? They provide a saner high level interface (at the cost of slowing things down, obviously). (Klaus Kaempf also wrote some glue code called "applayer", which offers a similar interface in C for the old sat-solver library. He used it for his bindings, I currently have the layer in the swig code.) It seems like you also want to provide repository management and download functions in your new API, that's currently not in scope of libsolv (and probably never will be). Are you already in discussions with the yum folks about their needs? Maybe they have some suggestions about how the interface could look like. (You do want yum to use your new API, right?) Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
Hi Florian!
On Tue, Jan 03, 2012 at 01:08:42PM +0100, Florian Festi wrote:
[...] Libsolv is an interesting implementation for a depsolver but does not offer the API we are looking for. It needs some kind of abstraction layer to hide all the moving parts. This is one of the things libzypp does. Yes, the "bare bone" C part of libsolv is probably not what you're looking for. The usage of Ids instead of pointers is a tad confusing, e.g. a package is described by (pool,id). A pointer to a package can become invalid when another package is added and so on. The interface itself isn't moving that rapidly, I just used the sat-solver -> libsolv rename to get rid of some old cruft and do some function name cleanup. Yes, we have noticed the sat-solver -> libsolv transition. Do you already have an idea when this step is going to be completed? It looks
But did you look at the perl/python/ruby bindings? They provide a saner high level interface (at the cost of slowing things down, obviously). (Klaus Kaempf also wrote some glue code called "applayer", which offers a similar interface in C for the old sat-solver library. He used it for his bindings, I currently have the layer in the swig code.) Yes, we had a look at the bindings (and examples). They are better than
It seems like you also want to provide repository management and download functions in your new API, that's currently not in scope of libsolv (and probably never will be). We are still trying to figure out how the different pieces need to be
On 01/03/2012 02:04 PM, Michael Schroeder wrote: like people work on both trees right now. the pure C API but still expose many implementation details. We'd rather not even expose the ids in the API but just the strings (or other tag data). As long as the API is located just below the cli the overhead should not matter much - even if the API is an order of magnitude more expensive than the direct lookup by id. put together. For now we'd try to hide libsolv as an implementation detail similar as libzypp does. On the long run I can imagine moving the libsolv code into librpm - replacing rpm's dependency check. But there is a lot of other stuff to do before even thinking about the details of such a step (e.g. adding support for 64 bit integers to libsolv).
Are you already in discussions with the yum folks about their needs? Maybe they have some suggestions about how the interface could look like. (You do want yum to use your new API, right?) Yes, we are in discussion with the yum folks. But a new depsolver will require a new data format (aka pool). Replacing both will not keep a lot of stones on one another within yum. So we will decide later whether and how to integrate the depsolver into yum. For now we are rather looking for getting things right than staying compatible.
Florian -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
* Florian Festi
On 01/03/2012 02:04 PM, Michael Schroeder wrote:
But did you look at the perl/python/ruby bindings? They provide a saner high level interface (at the cost of slowing things down, obviously). (Klaus Kaempf also wrote some glue code called "applayer", which offers a similar interface in C for the old sat-solver library. He used it for his bindings, I currently have the layer in the swig code.)
Yes, we had a look at the bindings (and examples). They are better than the pure C API but still expose many implementation details.
Well, it gives you all the capabilities and the flexibility of satsolver on the scripting language level. And there are quite some packages in openSUSE making use of it.
We'd rather not even expose the ids in the API but just the strings (or other tag data).
You can use the bindings in this way, while losing functionality. And a string is not sufficient to identify a package. So I wonder what level of detail you envision for the API ? Regards, Klaus --- SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
On 01/03/2012 01:08 PM, Florian Festi wrote:
The libzypp API looks pretty complicated. The "next-gen API for package management?" thread on the zypp-devel (September 2011) did not exactly dispel our concerns about that. As we need a C API using C++ creates a lot of overhead. If libzypp would provide the perfect API just putting a C layer in front would be a good way to go. But doing a lot of simplification within this step (as zpm [2]) will likely require tools needing access to "more internal" parts of the API to either be in C++ or wait for a new C layer being written. This is not a situation we are looking forward to.
Yes. Our plan is to write a simplified version of ZYpp API more suitable for writing clients and tools and then wrapping it in C. I know that this means adding new APIs everytime one needs to expose something advanced, but if the alternative is to write the complete core in C, then I think there is still an advantage in our approach. I would still be interested in more concrete examples about the kind API you need as you develop it, as it may help us shaping up the ZYpp API. -- Duncan Mac-Vicar P. - http://www.suse.com/ SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
Hi Florian,
* Florian Festi
Libsolv is an interesting implementation for a depsolver but does not offer the API we are looking for. It needs some kind of abstraction layer to hide all the moving parts.
can you come up with some (abstract) examples on how a C API should look like ?
This is one of the things libzypp does.
The problem with libzypp is, that is does (too?) many things. A collection of libraries would be more useful, e.g. - repository management - download & cache management (repo metadata, packages) - dependency solving [...]
As we are not as attached to the functionality implemented in libzypp as you might be (we are not using Patches or Patterns) implementing the package management API in pure C is an option we are considering.
Using C for time-critical parts like dependency solving is a must. What's the feedback from YUM developers on using a scripting language for the none-timecritical parts (i.e. package download) ? Regards, Klaus --- SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
participants (4)
-
Duncan Mac-Vicar P. -
Florian Festi -
Klaus Kaempf -
Michael Schroeder