[yast-devel] Improving the pkg-bindings API
Hi all,
I'd like to discuss how to improve the pkg-bindings API in YaST, more specifically
the Pkg.ResolvableProperties(name, kind, version) call.
But if you find some other pkg-bindings API calls difficult to use just ping me,
we can improve more calls.
There are basically two problems with Pkg.ResolvableProperties() call:
- You cannot use any input filters, you can only filter by name, kind or version.
That's quite limited, for the other cases you need to get *all* objects and filter
the returned list.
- The result contains all possible attributes for each object, even those you do not
need in most cases (like the package description and summary, installed size, RPM
size, full RPM name, vendor, ...). That is wasting the memory space.
If you call Pkg.ResolvableProperties("", :package, "") in Leap with the online
repositories enabled you will get a huge detailed list with about 40.000 (!)
packages. And that requires quite a lot of memory and might freeze the installer as
reported in [1].
A GraphQL Side Note
-------------------
I was interested in some existing generic solution for this problem. And I found a
quite nice GraphQL project. It can not only describe the search query but also
describe the result (which objects or attributes you need in the response).
Although there is a C++ library [2] or a Ruby library [3] in the end it looks like an
overkill for the internal API. But if you would need to design some big and
complicated REST API then you could consider it. See more details in [4] or in the
specification [5].
The Proposal
============
To not break the backward compatibility we cannot change the ResolvableProperties()
call so let's create a new one:
#
V Tue, 14 May 2019 15:04:43 +0200
Ladislav Slezak
Hi all,
I'd like to discuss how to improve the pkg-bindings API in YaST, more specifically the Pkg.ResolvableProperties(name, kind, version) call.
But if you find some other pkg-bindings API calls difficult to use just ping me, we can improve more calls.
There are basically two problems with Pkg.ResolvableProperties() call:
- You cannot use any input filters, you can only filter by name, kind or version. That's quite limited, for the other cases you need to get *all* objects and filter the returned list.
- The result contains all possible attributes for each object, even those you do not need in most cases (like the package description and summary, installed size, RPM size, full RPM name, vendor, ...). That is wasting the memory space.
If you call Pkg.ResolvableProperties("", :package, "") in Leap with the online repositories enabled you will get a huge detailed list with about 40.000 (!) packages. And that requires quite a lot of memory and might freeze the installer as reported in [1].
A GraphQL Side Note -------------------
I was interested in some existing generic solution for this problem. And I found a quite nice GraphQL project. It can not only describe the search query but also describe the result (which objects or attributes you need in the response).
Although there is a C++ library [2] or a Ruby library [3] in the end it looks like an overkill for the internal API. But if you would need to design some big and complicated REST API then you could consider it. See more details in [4] or in the specification [5].
The Proposal ============
To not break the backward compatibility we cannot change the ResolvableProperties() call so let's create a new one:
#
= required matching values # = returned attributes, empty list => return all Yast::Pkg.Resolvables( , ) # a ResolvableProperties() compatible version: Yast::Pkg.Resolvables({kind: :package, name: "yast2"}, []) => Array<Hash>
[{ "arch"=>"x86_64", "description"=> "This package contains scripts and data needed for SUSE Linux\n" + "installation with YaST2", "download_size"=>518308, "inst_size"=>2388454, "location"=>"yast2-4.0.87-lp150.2.9.1.x86_64.rpm", "locked"=>false, "medium_nr"=>1, "name"=>"yast2", "on_system_by_user"=>false, "path"=>"./rpms/x86_64/yast2-4.0.87-lp150.2.9.1.x86_64.rpm", "source"=>7, "status"=>:available, "summary"=>"YaST2 - Main Package", "transact_by"=>:solver, "vendor"=>"openSUSE", "version"=>"4.0.87-lp150.2.9.1", "version_epoch"=>nil, "version_release"=>"lp150.2.9.1", "version_version"=>"4.0.87" }]
# new functionality: more search criteria and limiting the result: Yast::Pkg.Resolvables({kind: :package, status: :available, name: "yast2"}, [:name, :version]) => Array<Hash>
[{ "name"=>"yast2", "version"=>"4.0.87-lp150.2.9.1", }]
Quite often we just need to know whether an object with some attributes exists without need to know the specific details, like a package is installed:
Yast::Pkg.AnyResolvable(name: "foo", kind: :package, status: :installed) => Boolean
That would be basically the same as Pkg.Resolvables, just returning a simple Boolean as the result instead of the details. (And it could be faster, it can return after finding the first matching item, it does not need to iterate over all objects.)
Ruby Wrapper ------------
Unfortunately the C++ Pkg-bindings API does not allow specifying optional parameters (that's a limitation of the YaST component system)
In fact there is a way, just not so easy. see e.g. https://github.com/yast/yast-core/blob/master/libycp/src/YCPBuiltinPath.cc#L... so you need to define it multiple times there. ( just note that I do not test it enough with ruby, as it is used now only in builtins which is already defined in ruby itself. I do not test it, but maybe also in ruby you can have something like: publish function: :test, signature: "string (string)" publish function: :test, signature: "string (string, string)" You can try it and maybe it will work. If not, I can check a bit more code as in general component system should allow it, just ycp does not allow it. Josef
and cannot provide an object-oriented API so probably having a small Ruby wrapper would be nice:
Yast::Resolvable.find(
) => Array<Resolvable>) Where the Resolvable would be an object wrapper for the pkg-bindings Hash.
Examples:
Yast::Resolvable.find(kind: :package, status: :selected) Yast::Resolvable.find(name: "package", kind: :package, status: :installed)
Similarly for the existence check (just for completeness):
Yast::Resolvable.any?(kind: :package, status: :selected)
I'll propose some code snippets at GitHub, I think it will be better to discuss some details in a draft pull request with some real code...
[1] https://bugzilla.suse.com/show_bug.cgi?id=1132650 [2] https://github.com/graphql/libgraphqlparser [3] https://graphql-ruby.org/ [4] https://graphql.org/ [5] https://graphql.github.io/graphql-spec/
-- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
Dne 14. 05. 19 v 15:04 Ladislav Slezak napsal(a):
The Proposal
To not break the backward compatibility we cannot change the ResolvableProperties() call so let's create a new one:
#
= required matching values # = returned attributes, empty list => return all Yast::Pkg.Resolvables( , )
[...]
Yast::Pkg.AnyResolvable(name: "foo", kind: :package, status: :installed) => Boolean
That would be basically the same as Pkg.Resolvables, just returning a simple Boolean as the result instead of the details. (And it could be faster, it can return after finding the first matching item, it does not need to iterate over all objects.)
This proposal has been fully implemented in yast2-pkg-bindings-4.2.0 (https://github.com/yast/yast-pkg-bindings/pull/114).
Ruby Wrapper
Unfortunately the C++ Pkg-bindings API does not allow specifying optional parameters (that's a limitation of the YaST component system) and cannot provide an object-oriented API so probably having a small Ruby wrapper would be nice:
In the end I used the Y2Packager namespace as we already have some similar classes there. The new Y2Packager::Resolvable instance methods maps to the old Pkg.ResolvableProperties() keys, e.g. Pkg.ResolvableProperties() returns "vendor" in the has so use the #vendor method to read it. Note: there is no direct link between the Ruby part and libzypp, if you change the libzypp state (e.g. add/remove repositories, install packages etc.) then the previously returned objects might be invalid or obsolete, you should always (re)load the latest status and do not store them for later. Implemented in yast2-packager-4.2.8 (https://github.com/yast/yast-packager/pull/442). Some examples: --------------------------------------------------------------------------------------- require "y2packager/resolvable" # returns ArrayY2Packager::Resolvable (or empty list if nothing found) res = Y2Packager::Resolvable.find(kind: :package, name: "yast2") r = res.first # => "yast2" r.name # => "4.0.77-lp150.2.3.1" r.version # the Y2Packager::Resolvable object loads only the basic values from libzypp # to save the memory, the other attributes are lazy loaded when needed (via # method_missing) # this will query libzypp to find the resolvable and fetch the description # => "This package contains scripts and data needed for SUSE Linux\n" \ # "installation with YaST2" r.description # the result is cached so the next call does not need libzypp and is much faster # => "This package ...." r.description # but if you need to process a large collection then the lazy loading might be slow, # in that case you can preload additional attributes in the initial call pkgs = Y2Packager::Resolvable.find({kind: :package}, [:description]) # this will NOT call libzypp for each object to get the description pkgs.each {|p| puts "#{p.name}: #{p.description}"} --------------------------------------------------------------------------------------- Enjoy! BTW there are more than one hundred Pkg.ResolvableProperties() calls in YaST, we should gradually replace them by the new API... -- Ladislav Slezák YaST Developer SUSE LINUX, s.r.o. Corso IIa Křižíkova 148/34 18600 Praha 8 -- To unsubscribe, e-mail: yast-devel+unsubscribe@opensuse.org To contact the owner, e-mail: yast-devel+owner@opensuse.org
participants (2)
-
Josef Reidinger -
Ladislav Slezak