Zypper vs. package caching : solution needed
Hi, Our local school has a painfully slow Internet connection, and there's no way to get a faster one. We have about two dozen desktop clients, and before starting the dist-upgrade from Leap 15.3 to Leap 15.4, I'm looking for a simple solution to avoid repeated downloads which would take ages. In the past I've fiddled with Apt-Cacher-NG, which sometimes works (sort of) for some repositories, but in daily practice it's unfortunately too buggy to be really usable. I've taken some notes about Apt-Cacher-NG and published them on my blog: https://blog.microlinux.fr/apt-cacher-ng-centos-7-opensuse/ Next, I went for a bone-headed solution consisting in sharing /var/cache/zypp/packages over NFS. On my local server (running Rocky Linux 8) I exported /srv/packages to the local network, and then all OpenSUSE clients are mounting this export on /var/cache/zypp/packages. Repositories are configured with the --keep-packages option, so the cache is not erased after each download/installation. Well, in theory. In practice, here's how it works (or not). All my desktop clients have a shared /var/cache/zypp/packages mounted over NFS. I perform my first upgrade on the first client, and things go smoothly. The cache is filled with almost 1 GB of downloaded RPMS and the corresponding metadata. So far so good. Now when I move on to the second desktop client, first thing I do is setup the official repos for Leap 15.4 before doing the upgrade with 'zypper addrepo --keep-packages etc.' And here's the thing: this operation WIPES THE CACHE. Which, as far as I can tell, is Zypper being particularly stupid. "Hey Zypper, please add these repositories but also please keep all the packages in the cache." "OK I'm setting up the repositories but I'm wiping the cache NOW. Duh." I spent some time reading and re-reading 'man zypper' but apparently there seem to be no more options to tell zypper to *never* *ever* wipe the cache in /var/cache/zypp/packages. Any suggestions ? Cheers from the sunny South of France, Niki -- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
Why not just setup a local mirror of the repos? https://en.opensuse.org/openSUSE:Mirror_howto -- Later, Darin -- Later, Darin On Fri, Jul 1, 2022 at 8:25 AM Nicolas Kovacs <info@microlinux.fr> wrote:
Hi,
Our local school has a painfully slow Internet connection, and there's no way to get a faster one.
We have about two dozen desktop clients, and before starting the dist-upgrade from Leap 15.3 to Leap 15.4, I'm looking for a simple solution to avoid repeated downloads which would take ages.
In the past I've fiddled with Apt-Cacher-NG, which sometimes works (sort of) for some repositories, but in daily practice it's unfortunately too buggy to be really usable. I've taken some notes about Apt-Cacher-NG and published them on my blog:
https://blog.microlinux.fr/apt-cacher-ng-centos-7-opensuse/
Next, I went for a bone-headed solution consisting in sharing /var/cache/zypp/packages over NFS. On my local server (running Rocky Linux 8) I exported /srv/packages to the local network, and then all OpenSUSE clients are mounting this export on /var/cache/zypp/packages.
Repositories are configured with the --keep-packages option, so the cache is not erased after each download/installation.
Well, in theory.
In practice, here's how it works (or not).
All my desktop clients have a shared /var/cache/zypp/packages mounted over NFS.
I perform my first upgrade on the first client, and things go smoothly. The cache is filled with almost 1 GB of downloaded RPMS and the corresponding metadata.
So far so good.
Now when I move on to the second desktop client, first thing I do is setup the official repos for Leap 15.4 before doing the upgrade with 'zypper addrepo --keep-packages etc.'
And here's the thing: this operation WIPES THE CACHE. Which, as far as I can tell, is Zypper being particularly stupid.
"Hey Zypper, please add these repositories but also please keep all the packages in the cache."
"OK I'm setting up the repositories but I'm wiping the cache NOW. Duh."
I spent some time reading and re-reading 'man zypper' but apparently there seem to be no more options to tell zypper to *never* *ever* wipe the cache in /var/cache/zypp/packages.
Any suggestions ?
Cheers from the sunny South of France,
Niki
-- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
Le 01/07/2022 à 14:44, Darin Perusich a écrit :
Why not just setup a local mirror of the repos?
https://en.opensuse.org/openSUSE:Mirror_howto <https://en.opensuse.org/openSUSE:Mirror_howto>
Because we have very limited bandwidth. I just want to keep already downloaded packages, not fetch the whole download.opensuse.org site. Cheers, Niki -- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
On Fri, Jul 1, 2022 at 9:19 AM Nicolas Kovacs <info@microlinux.fr> wrote:
Le 01/07/2022 à 14:44, Darin Perusich a écrit :
Why not just setup a local mirror of the repos?
https://en.opensuse.org/openSUSE:Mirror_howto <https://en.opensuse.org/openSUSE:Mirror_howto>
Because we have very limited bandwidth.
I just want to keep already downloaded packages, not fetch the whole download.opensuse.org site.
I understand you have limited bandwidth, this is the reason for having a local mirror. If the bandwidth is that poor then mirror the repos that you need from another site with a faster internet circuit, then sync that data to your onsite mirror. Then you only need to sync the updates repositories which should be a small subset of packages on a nightly basis.I would think this to be a simpler solution then trying to figure out some caching thing across N+1 computers. Good luck
Le 01/07/2022 à 15:54, Darin Perusich a écrit :
I understand you have limited bandwidth, this is the reason for having a local mirror. If the bandwidth is that poor then mirror the repos that you need from another site with a faster internet circuit, then sync that data to your onsite mirror. Then you only need to sync the updates repositories which should be a small subset of packages on a nightly basis.I would think this to be a simpler solution then trying to figure out some caching thing across N+1 computers.
Here's how you do it on Debian-based systems: 1. Install apt-cacher on a local machine and run it. Works out of the box. 2. Point all clients to that local machine using dpkg-reconfigure and typing the cache machine's host name. Takes 30 seconds to setup. No jumping through burning loops. No downloading/mirroring tons of RPMs. The OpenSUSE galaxy is badly lacking such a tool. https://en.wikipedia.org/wiki/KISS_principle -- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
Nicolas Kovacs wrote:
The OpenSUSE galaxy is badly lacking such a tool.
I wrote this just about 10 years ago: https://wiki.jessen.ch/index/How_to_cache_openSUSE_repositories_with_Squid It'll take you more than 30 seconds to set up though :-) -- Per Jessen, Zürich (21.9°C)
On 01.07.2022 18:56, Per Jessen wrote:
Nicolas Kovacs wrote:
The OpenSUSE galaxy is badly lacking such a tool.
I wrote this just about 10 years ago: https://wiki.jessen.ch/index/How_to_cache_openSUSE_repositories_with_Squid
ERR_SSL_VERSION_OR_CIPHER_MISMATCH bor@bor-Latitude-E5450:~$ sslscan wiki.jessen.ch Version: 2.0.7 OpenSSL 3.0.2 15 Mar 2022 Connected to 2a03:7520:4c68::2133 Testing SSL server wiki.jessen.ch on port 443 using SNI name wiki.jessen.ch SSL/TLS Protocols: SSLv2 enabled SSLv3 enabled TLSv1.0 enabled TLSv1.1 disabled TLSv1.2 disabled TLSv1.3 disabled
It'll take you more than 30 seconds to set up though :-)
Andrei Borzenkov wrote:
On 01.07.2022 18:56, Per Jessen wrote:
Nicolas Kovacs wrote:
The OpenSUSE galaxy is badly lacking such a tool.
I wrote this just about 10 years ago:
https://wiki.jessen.ch/index/How_to_cache_openSUSE_repositories_with_Squid
ERR_SSL_VERSION_OR_CIPHER_MISMATCH
Yeah, sorry, it's a very old webserver :-( Maybe I'll move that site - wikimedia also needs updating, to php7. In the meantime, for those truly interested, firefox permits downgrading tls for specific hosts: about:config security.tls.insecure_fallback_hosts -- Per Jessen, Zürich (21.4°C)
Per Jessen wrote:
Andrei Borzenkov wrote:
On 01.07.2022 18:56, Per Jessen wrote:
Nicolas Kovacs wrote:
The OpenSUSE galaxy is badly lacking such a tool.
I wrote this just about 10 years ago:
https://wiki.jessen.ch/index/How_to_cache_openSUSE_repositories_with_Squid
ERR_SSL_VERSION_OR_CIPHER_MISMATCH
Yeah, sorry, it's a very old webserver :-(
If anyone is interested, I've moved the site to a server with modern TLS :-) -- Per Jessen, Zürich (18.8°C)
On 2022-07-01 16:11, Nicolas Kovacs wrote:
Le 01/07/2022 à 15:54, Darin Perusich a écrit :
I understand you have limited bandwidth, this is the reason for having a local mirror. If the bandwidth is that poor then mirror the repos that you need from another site with a faster internet circuit, then sync that data to your onsite mirror. Then you only need to sync the updates repositories which should be a small subset of packages on a nightly basis.I would think this to be a simpler solution then trying to figure out some caching thing across N+1 computers.
Here's how you do it on Debian-based systems:
1. Install apt-cacher on a local machine and run it. Works out of the box.
2. Point all clients to that local machine using dpkg-reconfigure and typing the cache machine's host name.
Takes 30 seconds to setup.
No jumping through burning loops.
No downloading/mirroring tons of RPMs.
The OpenSUSE galaxy is badly lacking such a tool.
Yes. I have said so for ages.
-- Cheers / Saludos, Carlos E. R. (from Elesar, using openSUSE Leap 15.3)
On Fri, 1 Jul 2022 14:25:39 +0200 Nicolas Kovacs <info@microlinux.fr> wrote:
Hi,
Our local school has a painfully slow Internet connection, and there's no way to get a faster one.
We have about two dozen desktop clients, and before starting the dist-upgrade from Leap 15.3 to Leap 15.4, I'm looking for a simple solution to avoid repeated downloads which would take ages.
In the past I've fiddled with Apt-Cacher-NG, which sometimes works (sort of) for some repositories, but in daily practice it's unfortunately too buggy to be really usable. I've taken some notes about Apt-Cacher-NG and published them on my blog:
https://blog.microlinux.fr/apt-cacher-ng-centos-7-opensuse/
Next, I went for a bone-headed solution consisting in sharing /var/cache/zypp/packages over NFS. On my local server (running Rocky Linux 8) I exported /srv/packages to the local network, and then all OpenSUSE clients are mounting this export on /var/cache/zypp/packages.
Repositories are configured with the --keep-packages option, so the cache is not erased after each download/installation.
Well, in theory.
In practice, here's how it works (or not).
All my desktop clients have a shared /var/cache/zypp/packages mounted over NFS.
I perform my first upgrade on the first client, and things go smoothly. The cache is filled with almost 1 GB of downloaded RPMS and the corresponding metadata.
So far so good.
Now when I move on to the second desktop client, first thing I do is setup the official repos for Leap 15.4 before doing the upgrade with 'zypper addrepo --keep-packages etc.'
And here's the thing: this operation WIPES THE CACHE. Which, as far as I can tell, is Zypper being particularly stupid.
"Hey Zypper, please add these repositories but also please keep all the packages in the cache."
"OK I'm setting up the repositories but I'm wiping the cache NOW. Duh."
I spent some time reading and re-reading 'man zypper' but apparently there seem to be no more options to tell zypper to *never* *ever* wipe the cache in /var/cache/zypp/packages.
Any suggestions ?
<bodge> Set all the files so zypper doesn't have enough privilage to erase them? </bodge>
Cheers from the sunny South of France,
Niki
Le 01/07/2022 à 15:29, Dave Howorth a écrit :
Set all the files so zypper doesn't have enough privilage to erase them?
All these files are created by Zypper in the first place. Let me state this otherwise: How can I potty-train Zypper NEVER TO WIPE THE CACHE ? :o) -- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
On Fri, 2022-07-01 at 15:33 +0200, Nicolas Kovacs wrote:
Le 01/07/2022 à 15:29, Dave Howorth a écrit :
Set all the files so zypper doesn't have enough privilage to erase them?
All these files are created by Zypper in the first place.
Let me state this otherwise:
How can I potty-train Zypper NEVER TO WIPE THE CACHE ?
:o)
-- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
How about setting the immutable flag for any RPMs found in /var/cache/zypp (ahead of any host being upgraded w/ that cache in place)? # find /var/cache/zypp -type f -iname '*.rpm' -exec chattr +i {} \; # find /var/cache/zypp -type f -iname '*.rpm' -exec lsattr {} \; # find /var/cache/zypp -type f -iname '*.rpm' -exec chattr +i {} \; # find /var/cache/zypp -type f -iname '*.rpm' -exec lsattr {} \; ----i----------C------ /var/cache/zypp/packages/home_gmbr3_Chromium/x86_64/chromium- 96.0.4664.110-1626.1.x86_64.rpm # find /var/cache/zypp -type f -iname '*.rpm' -exec rm -v {} \; rm: cannot remove '/var/cache/zypp/packages/home_gmbr3_Chromium/x86_64/chromium- 96.0.4664.110-1626.1.x86_64.rpm': Operation not permitted -- ~ Scott Bradnick |- Windows Subsystem for Linux (WSL) Developer |-- Tumbleweed: |--- Dell Precision 5540 [NVIDIA Quadro T1000] (x86_64) |--- O-DROID H2+ [UHD Graphics 600] (x86_64) |--- 2x Raspberry Pi 4 Model B Rev 1.2 (aarch64) |--- 1x Raspberry Pi 3 Model B Rev 1.2 (aarch64) |--- WinBook TW100 (x86_64) https://keys.openpgp.org/ :: DBC5AA9A2D2BAEBC
On 2022-07-01 15:33, Nicolas Kovacs wrote:
Le 01/07/2022 à 15:29, Dave Howorth a écrit :
Set all the files so zypper doesn't have enough privilage to erase them?
All these files are created by Zypper in the first place.
Let me state this otherwise:
How can I potty-train Zypper NEVER TO WIPE THE CACHE ?
:o)
You can not. I opened a bugzilla on that, was told WONTFIX or equivalent. -- Cheers / Saludos, Carlos E. R. (from Elesar, using openSUSE Leap 15.3)
On 7/1/22 07:25, Nicolas Kovacs wrote:
Hi,
Our local school has a painfully slow Internet connection, and there's no way to get a faster one.
I did an article on setting up a local repository about 15 years ago, complete with scripts to manager removing duplicates from the cached packages to just keep the latest version (which may need tweaking with package naming changes between now and then). All of the other information is still relevant. The Cliff's Notes version: (1) on a local machine (if using a separate server as your repo server) set -k (--keep-packages) on each repo you want to cache locally. (disable use delta-rpms). That way each package you update will be save in /var/cache/zypp/pkgs. Parse and remove older version from the /var/cache/zypp/pkgs files with included script and rsync from that box to the server you want the school to use) (2) on the server that will be your repo server Configure apache or nginx to serve the repository location on your local server. Create a gpg key for repo signing. Download the latest 'deltainfo.xml.gz' (may use different compression now). Parse (with included script) and use to create local deltainfo.xml.gz from the master file. Create (or update) the repository metedata and sign with the gpg key. You now have a fully functioning local repository that includes the packages downloaded by any one machine available to update all machines in the school over the LAN. If you send me you WAN IPv4 (by private e-mail), I'll punch a hole in my firewall and send you a link to the page which includes detailed instructions and the scripts to parse and build the repo metadata (they will need tweaking as they were last updated for 11.3) Or, I could just dump the page to .pdf and send you the page and scripts as attachments. -- David C. Rankin, J.D.,P.E.
On 01.07.2022 15:25, Nicolas Kovacs wrote: ...
Now when I move on to the second desktop client, first thing I do is setup the official repos for Leap 15.4 before doing the upgrade with 'zypper addrepo --keep-packages etc.'
And here's the thing: this operation WIPES THE CACHE. Which, as far as I can tell, is Zypper being particularly stupid.
That is not the language to use for technical questions.
"Hey Zypper, please add these repositories but also please keep all the packages in the cache."
"OK I'm setting up the repositories but I'm wiping the cache NOW. Duh."
I spent some time reading and re-reading 'man zypper' but apparently there seem to be no more options to tell zypper to *never* *ever* wipe the cache in /var/cache/zypp/packages.
Any suggestions ?
a) mount /var/cache/zypper/packages after adding repository b) do not use "zypper addrepo", just drop repository definition into /etc/zypp/repos.d
Andrei Borzenkov composed on 2022-07-01 18:25 (UTC+0300):
b) do not use "zypper addrepo", just drop repository definition into /etc/zypp/repos.d
I've been doing this since April, mainly to speed up each PC's update process. Installing from the LAN is much faster than from the internet on machines unconstrained by 100MB NICs. Some few rpms somehow acquire RO status, a puzzle I have yet to try to diagnose. -- Evolution as taught in public schools is, like religion, based on faith, not based on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata
Le 01/07/2022 à 17:25, Andrei Borzenkov a écrit :
b) do not use "zypper addrepo", just drop repository definition into /etc/zypp/repos.d
That did the trick ! Thanks very much Andrei ! And also a big warm THANK YOU to everybody else for your numerous suggestions ! Cheers, Niki -- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
Le 01/07/2022 à 17:25, Andrei Borzenkov a écrit :
And here's the thing: this operation WIPES THE CACHE. Which, as far as I can tell, is Zypper being particularly stupid.
That is not the language to use for technical questions.
If a Linux tool does exactly the opposite (e. g. wipes the cache) of what I'm asking it to do in the first place (--keep-packages), then "stupid" is more likely the most *polite* word I can come up with in the context. Please note that I refrained from a selection of more colorful expletives overheard from my late Hungarian grandmother. :o) -- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
* Nicolas Kovacs <info@microlinux.fr> [07-02-22 05:10]:
Le 01/07/2022 à 17:25, Andrei Borzenkov a écrit :
And here's the thing: this operation WIPES THE CACHE. Which, as far as I can tell, is Zypper being particularly stupid.
That is not the language to use for technical questions.
If a Linux tool does exactly the opposite (e. g. wipes the cache) of what I'm asking it to do in the first place (--keep-packages), then "stupid" is more likely the most *polite* word I can come up with in the context.
Please note that I refrained from a selection of more colorful expletives overheard from my late Hungarian grandmother.
:o)
perhaps you are expecting something from a tool which was not intended. maybe better that you offer to provide the functionality you desire. -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet oftc
Le 02/07/2022 à 13:32, Patrick Shanahan a écrit :
perhaps you are expecting something from a tool which was not intended. maybe better that you offer to provide the functionality you desire.
Here's the corresponding quote from 'man zypper': -k, --keep-packages Enable RPM files caching for the repository. I think we can all agree that man pages are a good source to show the *intended* functionality of a tool. In that case, *not* empty the package cache but keep stuff that's downloaded. In daily practicy, using zypper addrepo along with this option effectively wipes the cache. *insert obscene expletive here*. Corresponding bug reports have been sent (see earlier in this thread) and answered with a WONTFIX. And no, I won't fork zypper now. The workaround posted by Andrei works, so I guess it will do. Cheers, Niki -- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
Am 02.07.22 um 14:01 schrieb Nicolas Kovacs:
Le 02/07/2022 à 13:32, Patrick Shanahan a écrit :
perhaps you are expecting something from a tool which was not intended. maybe better that you offer to provide the functionality you desire.
Here's the corresponding quote from 'man zypper':
-k, --keep-packages Enable RPM files caching for the repository.
I think we can all agree that man pages are a good source to show the *intended* functionality of a tool. In that case, *not* empty the package cache but keep stuff that's downloaded.
In daily practicy, using zypper addrepo along with this option effectively wipes the cache. *insert obscene expletive here*.
The "keep downloaded packages" means what it says. All packages you download from the repo's creation/initialization on are kept. Zypper is not looking whether this repo already exists or not, using plain logic: if this repo already existed (and contained packages), you would not create it. A directory that is already named according to the repo's name must thus be erroneous and has to be deleted (and created anew).
Corresponding bug reports have been sent (see earlier in this thread) and answered with a WONTFIX.
I'm still not sure that it should change. The behaviour can be explained, even if it's not what you would like.
And no, I won't fork zypper now. The workaround posted by Andrei works, so I guess it will do.
I do several "rsync --delete" to the opensuse systems and provide the repos' content completely (but of course not every repo, only those I need) instead of hassling with /var/zypp/packages. The repos on the hosts can be created a hundred times without deleting anything, and their baseurl points to my proxy host. Werner --
Nicolas Kovacs wrote:
Le 02/07/2022 à 13:32, Patrick Shanahan a écrit :
perhaps you are expecting something from a tool which was not intended. maybe better that you offer to provide the functionality you desire.
Here's the corresponding quote from 'man zypper':
-k, --keep-packages Enable RPM files caching for the repository.
I think we can all agree that man pages are a good source to show the *intended* functionality of a tool. In that case, *not* empty the package cache but keep stuff that's downloaded.
TBH, it sort of makes sense. You are adding a *new* repo. It is therefore safe to assume that anything already in the cache direcory is bogus stuff and should be removed. The '-k' switch only refers to the *future* behavior of that repo. To do what you want IMO it would need a new keyword --keep-existing-cache. I'd say the best route for you is the suggestion to just drop in the new repo files in /etc/zypp/repos.d and do a 'zypper ref; zypper dup'. Just make sure it's the proper one (from initial upgraded host) that has the keep-packages active.
In daily practicy, using zypper addrepo along with this option effectively wipes the cache. *insert obscene expletive here*.
As explained above, IMO this is a sensible choice. Keeping a pre-existing cache because of the use of -k option is not.
Le 03/07/2022 à 10:12, Peter Suetterlin a écrit :
TBH, it sort of makes sense. You are adding a *new* repo. It is therefore safe to assume that anything already in the cache direcory is bogus stuff and should be removed. The '-k' switch only refers to the *future* behavior of that repo. To do what you want IMO it would need a new keyword --keep-existing-cache.
Yes, that would make sense. Sort of. Except the fact that adding *one* new repo effectively wipes *all* the existing stuff from *all* the repos. - Hey, can I have a beer with my steak ? - Of course, but let me first clean your table. - But I'm eating my steak. - No problem. You can order one again anytime. :o)
I'd say the best route for you is the suggestion to just drop in the new repo files in /etc/zypp/repos.d and do a 'zypper ref; zypper dup'. Just make sure it's the proper one (from initial upgraded host) that has the keep-packages active.
I just rewrote my configuration scripts accordingly: https://gitlab.com/kikinovak/opensuse-lp154/-/blob/main/setup.sh https://gitlab.com/kikinovak/opensuse-lp154/-/blob/main/upgrade.sh Cheers, Niki -- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
On 03.07.2022 11:48, Nicolas Kovacs wrote:
Le 03/07/2022 à 10:12, Peter Suetterlin a écrit :
TBH, it sort of makes sense. You are adding a *new* repo. It is therefore safe to assume that anything already in the cache direcory is bogus stuff and should be removed. The '-k' switch only refers to the *future* behavior of that repo. To do what you want IMO it would need a new keyword --keep-existing-cache.
Yes, that would make sense. Sort of. Except the fact that adding *one* new repo effectively wipes *all* the existing stuff from *all* the repos.
I cannot reproduce it.
On 2022-07-03 11:22, Andrei Borzenkov wrote:
On 03.07.2022 11:48, Nicolas Kovacs wrote:
Le 03/07/2022 à 10:12, Peter Suetterlin a écrit :
TBH, it sort of makes sense. You are adding a *new* repo. It is therefore safe to assume that anything already in the cache direcory is bogus stuff and should be removed. The '-k' switch only refers to the *future* behavior of that repo. To do what you want IMO it would need a new keyword --keep-existing-cache.
Yes, that would make sense. Sort of. Except the fact that adding *one* new repo effectively wipes *all* the existing stuff from *all* the repos.
I cannot reproduce it.
I recall this behavior in the past. I have not tested recently. It should be in my bug report, years ago. -- Cheers / Saludos, Carlos E. R. (from Elesar, using openSUSE Leap 15.3)
Le 03/07/2022 à 16:52, Carlos E. R. a écrit :
I cannot reproduce it. I recall this behavior in the past. I have not tested recently. It should be in my bug report, years ago.
Before we're going further down this foxhole, let me just state that Andrei's suggestion works perfectly. I just setup four networked VMs on my workstation to test the setup. * One NFS server running Rocky Linux 8 * Three OpenSUSE Leap 15.3 NFS clients with shared cache. I upgraded the first client using my upgrade.sh script, which downloaded and installed about 1.200 packages. Then I launched the script on the next client, and packages were fetched as expected over the shared cache. Installation was much (!) faster. Same thing on the third desktop client. Just works. So tomorrow I'll configure this shared NFS package cache in our local school to manage dist-upgrading all our two dozen desktop clients to OpenSUSE Leap 15.4. And I'm glad to have found a working solution to make up for our painfully slow bandwidth. Cheers, Niki -- Microlinux - Solutions informatiques durables 7, place de l'église - 30730 Montpezat Site : https://www.microlinux.fr Blog : https://blog.microlinux.fr Mail : info@microlinux.fr Tél. : 04 66 63 10 32 Mob. : 06 51 80 12 12
On 03/07/2022 20.49, Nicolas Kovacs wrote:
Le 03/07/2022 à 16:52, Carlos E. R. a écrit :
...
So tomorrow I'll configure this shared NFS package cache in our local school to manage dist-upgrading all our two dozen desktop clients to OpenSUSE Leap 15.4. And I'm glad to have found a working solution to make up for our painfully slow bandwidth.
That's very nice :-) -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.3 (Legolas))
Am 03.07.2022 um 10:49 schrieb Nicolas Kovacs <info@microlinux.fr>:
Hey, can I have a beer with my steak ?
- Of course, but let me first clean your table.
- But I'm eating my steak.
- No problem. You can order one again anytime.
I like analogisms to real world 😃 Which directly leads to the natural solution: one would always order the beer first. 😀 Have a nice day, Berny
real world? Genuine men always take at least two beer! Am 03.07.22 um 13:13 schrieb Bernhard Völker:
Am 03.07.2022 um 10:49 schrieb Nicolas Kovacs <info@microlinux.fr>:
Hey, can I have a beer with my steak ?
- Of course, but let me first clean your table.
- But I'm eating my steak.
- No problem. You can order one again anytime. I like analogisms to real world 😃 Which directly leads to the natural solution: one would always order the beer first. 😀
Have a nice day, Berny
On Fri, Jul 1, 2022 at 7:26 AM Nicolas Kovacs <info@microlinux.fr> wrote:
Hi,
Our local school has a painfully slow Internet connection, and there's no way to get a faster one.
...
I spent some time reading and re-reading 'man zypper' but apparently there seem to be no more options to tell zypper to *never* *ever* wipe the cache in /var/cache/zypp/packages.
Any suggestions ?
For this exact reason, I created a small C .so that I LD_PRELOAD which implements "unlink()" and "rmdir()". I'll attach an SRPM of it. Note that the paths are hard-coded into it (in my case "/nfs/sys/zypp-cache/"), but can the wrapper script (also included) sets an environment variable that overrides it. -Nick
On 2022-07-01 14:25, Nicolas Kovacs wrote:
Hi,
...
Next, I went for a bone-headed solution consisting in sharing /var/cache/zypp/packages over NFS.
Yes, that's what I do.
On my local server (running Rocky Linux 8) I exported /srv/packages to the local network, and then all OpenSUSE clients are mounting this export on /var/cache/zypp/packages.
Yes.
Repositories are configured with the --keep-packages option, so the cache is not erased after each download/installation.
Yes.
Well, in theory.
Yes.
In practice, here's how it works (or not).
All my desktop clients have a shared /var/cache/zypp/packages mounted over NFS.
I perform my first upgrade on the first client, and things go smoothly. The cache is filled with almost 1 GB of downloaded RPMS and the corresponding metadata.
So far so good.
Now when I move on to the second desktop client, first thing I do is setup the official repos for Leap 15.4 before doing the upgrade with 'zypper addrepo --keep-packages etc.'
And here's the thing: this operation WIPES THE CACHE. Which, as far as I can tell, is Zypper being particularly stupid.
Yeah, you have to umount the nfs directory first, then add the repos. It will create a local structure, or not, and if it does, compare with the remote structure and then mount it, if it is the same names. Or, edit the repo files directly.
"Hey Zypper, please add these repositories but also please keep all the packages in the cache."
"OK I'm setting up the repositories but I'm wiping the cache NOW. Duh."
I spent some time reading and re-reading 'man zypper' but apparently there seem to be no more options to tell zypper to *never* *ever* wipe the cache in /var/cache/zypp/packages.
Yeah :-(
Any suggestions ?
I don't know how to remove delete permissions that would work for this. Keep another copy on another directory (can be hard links), and restore them when deleted. A script for creation of the backup, another for restoring. With hard links, it would be very fast and use zero disk space.
Cheers from the sunny South of France,
Cheers from the sunny South-East of Spain :-) -- Cheers / Saludos, Carlos E. R. (from Elesar, using openSUSE Leap 15.3)
On 7/1/22 14:25, Nicolas Kovacs wrote:
I spent some time reading and re-reading 'man zypper' but apparently there seem to be no more options to tell zypper to *never* *ever* wipe the cache in /var/cache/zypp/packages.
I've been using this a couple of times: (Note, I'm using Tumbleweed, so I have to use 'zypper dup' instead of 'zypper up'). In general, I use the zypper option --pkg-cache-dir, because the default /var/cache/zypp/packages is on the '/' filesystem here, and especially big updates with texlive can be 4-7G. On one "master" PC, I run 'zypper dup -d' (=download-only). All packages get downloaded to the package cache dir. Then I'm rsyncing that directory to the other PC(s), followed by 'zypper dup' there. I could also mount them via NFS or sshfs. When all other PCs are updated, I run 'zypper dup' on the first one. Well, if some packages are missing because they're only installed on PC 2, 3, ..., then they get downloaded as needed. No big deal. But the big part (3-4G) is already there. Eventually, you might want to run 'zypper clean' on each system later. I know that sounds a bit hacky, but it saved me some hours of downloading already. Yes, that's still the truth in some parts of good old Germany. In the apartments we stayed during the last couple of holidays, the internet was usually 10-20 times faster than at home. I'm glad we don't have to feed carrier pigeons in Germany. ;-) Have a nice day, Berny
W dniu 01.07.2022 o 14:25, Nicolas Kovacs pisze:
Hi,
Our local school has a painfully slow Internet connection, and there's no way to get a faster one.
We have about two dozen desktop clients, and before starting the dist-upgrade from Leap 15.3 to Leap 15.4, I'm looking for a simple solution to avoid repeated downloads which would take ages.
[...]
Any suggestions ?
Cheers from the sunny South of France,
Niki
I use squid https://etam-software.eu/blog/2021-03-24-caching-opensuse-repos-with-squid.h...
On 7/1/22 21:55, Nicolas Kovacs wrote:
Hi,
...
Any suggestions ?
Sorry if someone has already said it but this is a problem for us in Australia so my colleague wrote https://github.com/Firstyear/opensuse-proxy-cache, basically I run the container on one of my tumbleweed instances and then point everything else to that IP address as the repo location and it will cache the last X GB of packages that go through. Cheers -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
participants (19)
-
Adam Mizerski -
Andrei Borzenkov -
Bernhard Voelker -
Bernhard Völker
-
Bjoern Voigt -
Carlos E. R. -
Darin Perusich -
Dave Howorth -
David C. Rankin -
Dr. Christoph Rüdt -
Felix Miata -
Nick LeRoy -
Nicolas Kovacs -
Patrick Shanahan -
Per Jessen -
Peter Suetterlin -
Scott Bradnick -
Simon Lees -
Werner Flamme