[zypp-devel] RFC: Adding a cache of service metadata (bnc#765308)
Hi, (proposed) short version: in the future (not including SLE11-SP2) repo services will be cached, so if you expect a service to change (SP migration?), call "zypper refs". Discussion, please comment (Wagon, SUSE Manager, hello): As you know, zypp takes packages from *repositories*, and the package/repo metadata is cached: without autorefresh, the cache is refreshed only explicitly by zypper refresh (ref). With autorefresh (and unless overriden by --no-refresh) , we contact the server to update one or more metadata files BUT only if some time has elapsed since we last contacted the server (5 minutes by default). Zypp also knows *services*, which essentially are server-defined mutable lists of repositories. They too have an autorefresh attribute and respect --no-refresh. However, they do not have the timed cache like above. For interactive usage it is only a minor annoyance since you usually have only one service (nu.novell.com) so there is only one round trip. But as soon as you use Chef which checks many packages individualy with a "zypper info" call, the latencies add up. Therefore I've proposed to add a service cache. Michael has pointed out that some use cases rely on the service not being cached, namely the migration to a next service pack, which we implement by installing a product and relying on nu.novell.com to give us a new set of repos. With the service cache, those use cases would need an explicit service cache invalidation AKA zypper service-refresh (refs). I think that should be done in the long term, and in the short term (SLE11 SP2) the service cache would be disabled by default. Regarding how to control the service cache: a) zypp.conf has an option repo.refresh.delay, so we could add service.refresh.delay b) a CLI option '--allow-service-cache <sec>' I prefer (a) for symmetry and it seems that (b) per-run config is not needed, otherwise users would request --allow-repo-cache already. -- Martin Vidner, YaST developer http://en.opensuse.org/User:Mvidner Kuracke oddeleni v restauraci je jako fekalni oddeleni v bazenu
* Martin Vidner <mvidner@suse.cz> [Jun 19. 2012 15:55]:
But as soon as you use Chef which checks many packages individualy with a "zypper info" call, the latencies add up.
What kind of information does Chef need ? Are there alternatives to a 'zypper' call ? Klaus -- SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
On Tue, Jun 19, 2012 at 03:58:17PM +0200, Klaus Kaempf wrote:
* Martin Vidner <mvidner@suse.cz> [Jun 19. 2012 15:55]:
But as soon as you use Chef which checks many packages individualy with a "zypper info" call, the latencies add up.
What kind of information does Chef need ?
Are there alternatives to a 'zypper' call ?
We are talking about https://github.com/opscode/chef/blob/master/chef/lib/chef/provider/package/z... One way of improving this, would be to use libzypp ruby bindings instead of calling to zypper. Not sure if this would solve the service refresh issues. Best Christoph -- Christoph Thiel, Project Manager, SUSE Cloud SUSE LINUX Products GmbH, Maxfeldstraße 5, 90409 Nürnberg, Germany GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
* Christoph Thiel <cthiel@suse.com> [Jun 19. 2012 16:02]:
On Tue, Jun 19, 2012 at 03:58:17PM +0200, Klaus Kaempf wrote:
* Martin Vidner <mvidner@suse.cz> [Jun 19. 2012 15:55]:
But as soon as you use Chef which checks many packages individualy with a "zypper info" call, the latencies add up.
What kind of information does Chef need ?
Are there alternatives to a 'zypper' call ?
We are talking about https://github.com/opscode/chef/blob/master/chef/lib/chef/provider/package/z...
One way of improving this, would be to use libzypp ruby bindings instead of calling to zypper. Not sure if this would solve the service refresh issues.
Probably not since zypper is 'just' a cli frontend to libzypp. However, if we assume that all package operations are performed via zypper, operating directly on /var/cache/solve + libsatsolver (resp. libsolv) could be an alternative. Klaus -- SUSE LINUX Products GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 16746 (AG Nürnberg) Maxfeldstraße 5, 90409 Nürnberg, Germany -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
On Wednesday 20 June 2012 08:43:00 Klaus Kaempf wrote:
* Christoph Thiel <cthiel@suse.com> [Jun 19. 2012 16:02]:
On Tue, Jun 19, 2012 at 03:58:17PM +0200, Klaus Kaempf wrote:
Are there alternatives to a 'zypper' call ?
We are talking about https://github.com/opscode/chef/blob/master/chef/lib/chef/provider/packag e/zypper.rb
One way of improving this, would be to use libzypp ruby bindings instead of calling to zypper. Not sure if this would solve the service refresh issues.
Probably not since zypper is 'just' a cli frontend to libzypp.
Well, it's not a bug, it's designed that way. If zypper would not call ServiceRefresh, it would not happen. But zypper intentionally does a service refresh before refreshing the repos. It is not enforced but libzypp, so it would be possible to solve this 'zypper-only' (via CLI or ENV).
However, if we assume that all package operations are performed via zypper, operating directly on /var/cache/solve + libsatsolver (resp. libsolv) could be an alternative.
That's basically the same as using 'zypper --no-refresh info'. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres SUSE LINUX Products GmbH, Development, ma@suse.de GF:Jeff Hawn,Jennifer Guild,Felix Imendörffer, HRB16746(AG Nürnberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
Dne 19.6.2012 15:55, Martin Vidner napsal(a):
Hi,
(proposed) short version: in the future (not including SLE11-SP2) repo services will be cached, so if you expect a service to change (SP migration?), call "zypper refs".
Discussion, please comment (Wagon, SUSE Manager, hello):
As you know, zypp takes packages from *repositories*, and the package/repo metadata is cached: without autorefresh, the cache is refreshed only explicitly by zypper refresh (ref). With autorefresh (and unless overriden by --no-refresh) , we contact the server to update one or more metadata files BUT only if some time has elapsed since we last contacted the server (5 minutes by default).
Zypp also knows *services*, which essentially are server-defined mutable lists of repositories. They too have an autorefresh attribute and respect --no-refresh. However, they do not have the timed cache like above. For interactive usage it is only a minor annoyance since you usually have only one service (nu.novell.com) so there is only one round trip. But as soon as you use Chef which checks many packages individualy with a "zypper info" call, the latencies add up.
Therefore I've proposed to add a service cache. Michael has pointed out that some use cases rely on the service not being cached, namely the migration to a next service pack, which we implement by installing a product and relying on nu.novell.com to give us a new set of repos.
Can a change of the installed products just invalidate the service cache automatically? Jiri
With the service cache, those use cases would need an explicit service cache invalidation AKA zypper service-refresh (refs). I think that should be done in the long term, and in the short term (SLE11 SP2) the service cache would be disabled by default.
Regarding how to control the service cache: a) zypp.conf has an option repo.refresh.delay, so we could add service.refresh.delay b) a CLI option '--allow-service-cache<sec>' I prefer (a) for symmetry and it seems that (b) per-run config is not needed, otherwise users would request --allow-repo-cache already.
-- Regards, Jiri Srain Project Manager --------------------------------------------------------------------- SUSE LINUX, s.r.o. e-mail: jsrain@suse.com Lihovarska 1060/12 tel: +420 284 084 659 190 00 Praha 9 fax: +420 284 084 001 Czech Republic http://www.suse.com -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
On Tuesday 19 June 2012 15:55:31 Martin Vidner wrote:
Hi,
(proposed) short version: in the future (not including SLE11-SP2) repo services will be cached, so if you expect a service to change (SP migration?), call "zypper refs".
Discussion, please comment (Wagon, SUSE Manager, hello):
SP migration is quite an easy case as it is our code and we know when the service will change. Not a big deal to add the 'zypper refs'. More interesting are the plugin services (Spacewalk / SUSE Manager) where the client does not know when the channels at the server side are reassigned. Scripts we provide can be adapted (action scripts, bootstrap).
With the service cache, those use cases would need an explicit service cache invalidation AKA zypper service-refresh (refs). I think that should be done in the long term, and in the short term (SLE11 SP2) the service cache would be disabled by default.
In the first place it would not be implemented for SLE11-SP2. It is targeting SLE12, isn't it?
Regarding how to control the service cache: a) zypp.conf has an option repo.refresh.delay, so we could add service.refresh.delay b) a CLI option '--allow-service-cache <sec>' I prefer (a) for symmetry and it seems that (b) per-run config is not needed, otherwise users would request --allow-repo-cache already.
Well --allow-repo-cache isn't needed because we already have repo.refresh.delay. A CLI option would make sense if we are looking for a quick zypper-only solution which may even be portable to SP2. While a) is a real feature. -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres SUSE LINUX Products GmbH, Development, ma@suse.de GF:Jeff Hawn,Jennifer Guild,Felix Imendörffer, HRB16746(AG Nürnberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org To contact the owner, e-mail: zypp-devel+owner@opensuse.org
participants (5)
-
Christoph Thiel -
Jiri Srain -
Klaus Kaempf -
Martin Vidner -
Michael Andres