[opensuse] In seek of a working proxying/caching scheme for zypper updates
Hi, anybody out there with a working caching scheme for zypper updates/upgrades? The problem is two-fold: for one, it needs to intercept https with all kinds of side effects, security implications, etc, and the other is the CDN (mirrorbrain). What I'm looking for is *not* a complete mirror, because this is overkill for small sites. It would be more in a fashion of a proxy, but should provide the cached packages in a human readable form as well. It would proxy zypper fetch requests in a CDN-server agnostic form, fetch the files, as zypper does, and deliver them the "last mile". That way, any (update) package would only be fetched once from upstream, and subsequent accesses are dealt with locally, much improving speed and reducing bandwidth needs. The human readable caching form would allow for some manual downgrades, if needed. As a bonus, some retention processing may limit storage needs. I've experimented with a squid helper already, https://github.com/frispete/ squid_dedup, but intercepting https isn't fun... Have you guys something up, that (at least partly) works that way? Per? Cheers, Pete -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 4/20/19 6:33 AM, Hans-Peter Jansen wrote:
anybody out there with a working caching scheme for zypper updates/upgrades?
I am using what I call "the Carlos method", because it is based on an email message from Carlos in May 2015. Check the archives, where there is a longish discussion on the topic. I blogged about how I set this up: https://nwrickert2.wordpress.com/2015/09/04/sharing-updates-with-opensuse/ There's a link to the message by Carlos in that blog post. I'm still using that method, and it is working pretty well. Ignore the comments about "NetworkManager" in that post. The problem that they describe has long since been fixed. And note that I am using this over wired connections. It generates a lot of NFS network traffic (home network only), which I prefer to keep off wifi. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 20/04/2019 15.09, Neil Rickert wrote:
On 4/20/19 6:33 AM, Hans-Peter Jansen wrote:
anybody out there with a working caching scheme for zypper updates/upgrades?
I am using what I call "the Carlos method", because it is based on an email message from Carlos in May 2015. Check the archives, where there is a longish discussion on the topic.
:-D
I blogged about how I set this up:
https://nwrickert2.wordpress.com/2015/09/04/sharing-updates-with-opensuse/
:-)
There's a link to the message by Carlos in that blog post. I'm still using that method, and it is working pretty well. Ignore the comments about "NetworkManager" in that post. The problem that they describe has long since been fixed. And note that I am using this over wired connections. It generates a lot of NFS network traffic (home network only), which I prefer to keep off wifi.
One note for your blog: the directory name is also what zypper calls the "alias" of the repo, not its name. "zypper lr --details" prints both. And there is a command that changes it, beside editing the files directly, but see the comment below. Also, you can not do "zypper clean". I think it was that one. I have emptied the cache accidentally more than once one or another way. Like accidentally having one computer without "keep packages" activated. Unfortunately "zypper lr --details" does not print it. Instead: grep keeppackages /etc/zypp/repos.d/*repo or grep "keeppackages=0" /etc/zypp/repos.d/*repo The command "zypper renamerepo" deletes the repo contents, too (Bug 1054328). However, as Per says, why use https? Mirrorbrain doesn't support it. -- Cheers / Saludos, Carlos E. R. (from openSUSE, Leap 15.1 x86_64 (ssd-test)) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hans-Peter Jansen writes:
anybody out there with a working caching scheme for zypper updates/upgrades?
What I'm looking for is *not* a complete mirror, because this is overkill for small sites. It would be more in a fashion of a proxy, but should provide the cached packages in a human readable form as well. (...) Have you guys something up, that (at least partly) works that way? Per?
apt-cacher-ng has worked well for me with Debian systems and supports openSUSE with some limitations: https://www.unix-ag.uni-kl.de/~bloch/acng/html/distinstructions.html#hint-su... https://build.opensuse.org/package/show/server:proxy/apt-cacher-ng Hope this helps. Jeff -- Jeff Kowalczyk Software Engineer, Go Developer Experience SUSE Linux -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Hans-Peter Jansen wrote:
Hi,
anybody out there with a working caching scheme for zypper updates/upgrades?
Yes, I actually wrote this up 6-7 years ago, it works very well. The core is a plain squid cache, which a few bits bolted on. http://wiki.jessen.ch/index/How_to_cache_openSUSE_repositories_with_Squid
The problem is two-fold: for one, it needs to intercept https with all kinds of side effects, security implications, etc, and the other is the CDN (mirrorbrain).
Uh, why https:// ?? The mirror system does not support https:// The key issue in caching with squid is that zypper, to speed things up, does segmented downloading. A package will be put together by chunks from multiple mirrors.
What I'm looking for is *not* a complete mirror, because this is overkill for small sites. It would be more in a fashion of a proxy, but should provide the cached packages in a human readable form as well.
What is "a human readable form" to you?
It would proxy zypper fetch requests in a CDN-server agnostic form, fetch the files, as zypper does, and deliver them the "last mile". That way, any (update) package would only be fetched once from upstream, and subsequent accesses are dealt with locally, much improving speed and reducing bandwidth needs.
Yes, that is precisely what I needed too. At the time, I did not have sufficient bandwidth/space to run a mirror, but I wanted a 2nd install or upgrade to run at wire speed. I also did not want the added effort in maintaning a mirror.
Have you guys something up, that (at least partly) works that way? Per?
See above :-) -- Per Jessen, Zürich (13.0°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (5)
-
Carlos E. R. -
Hans-Peter Jansen -
Jeff Kowalczyk -
Neil Rickert -
Per Jessen