[opensuse] How to build a "caching only" local repository for use by many machines on my LAN?
Hi I have a bunch of opensuse machines on a lan. They're all the same version of opensuse (13.2), all the same arch (x86_64), and all use the same set of repos (in /etc/zypp/repos.d). I update them all regularly. They all have a common core set of installed packages, and each machine has additional (groups of) pacakges unique to its user and usage. When updating, I can either (1) update them all individually, each machine pulling its pkgs from the repo mirrors or (2) rsync all the full source repos to a single instance on the lan, and update the clients from the local collection Each has its drawbacks: (1) leads to a lot of redundant traffic (2) leads to rsync of lots of unneeded packages What I'd LIKE to do instead is set up some sort of caching on the LAN so that when any machine downloads any new or upgraded package, the rpms are FIRST cached to a lan machine that then makes those packages available to other machines as they need it. Since all the machines are pretty similar, and there's always at least a couple of machines that need the same package, this would (1) make sure that I'm not re-downloading the same package over and over (2) prevent rsync/storing packages that are never used at all The question is -- can I do this with zypper, or some other tool? Cheers Arvind -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 05/20/2015 09:21 PM, arvindg@emailuser.net wrote:
What I'd LIKE to do instead is set up some sort of caching on the LAN so that when any machine downloads any new or upgraded package, the rpms are FIRST cached to a lan machine that then makes those packages available to other machines as they need it.
Since all the machines are pretty similar, and there's always at least a couple of machines that need the same package, this would
(1) make sure that I'm not re-downloading the same package over and over (2) prevent rsync/storing packages that are never used at all
The question is -- can I do this with zypper, or some other tool?
I'm sure this has come up, so there's be something in the archives you can google for. Here's some context to help. What you want is a "caching proxy". Rather than have each machine point to the opensuse repositories, point all machines except the proxy to the proxy when adding repositories. https://en.opensuse.org/SDB:Creating_YaST_installation_sources Or perhaps better illustrated: http://eureka.ykyuen.info/2010/01/06/opensuse-create-your-own-software-repos... http://linux.die.net/man/8/createrepo That's a start. You'll have to do a bit of background reading if you're not familiar with the types of metadata needed, how to set up Apache, and a few other things. You seem to be aware that simply downloading ALL of a remote repository is over-doing it, there are going to be packages that never get requested by any of the machines. That's why a proxy. Zypper can be told not delete the rpms when it has installed them. It can also be told to 'just' download and where to download to. back to RTFM for that :-) <quote> -d, --download-only Only download the packages for later installation. </quote> <quote> -k, --keep-packages Enable RPM files caching for the repository. </quote> <quote> /var/cache/zypp/packages If keeppackages property is set for a repository (see the modifyrepo command), all the RPM file downloaded during installation will be kept here. See also the clean command for cleaning these cache directories. </quote> You may want to link or symlink that to the right place for access ... see above link The trick, therefore is to have the proxy ALWAYS run a parametrized zypper to make sure the latest is in the right places in the cache, then let the caller access that. All that being said, you might want to try this first, usign real locations not mirrorbrain. http://sysops.pblogs.gr/2012/06/rpm-caching-proxy-for-yum-zypper-kiwi-simila... -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
arvindg@emailuser.net wrote:
What I'd LIKE to do instead is set up some sort of caching on the LAN so that when any machine downloads any new or upgraded package, the rpms are FIRST cached to a lan machine that then makes those packages available to other machines as they need it.
I have exactly such a setup with squid.
Since all the machines are pretty similar, and there's always at least a couple of machines that need the same package, this would
(1) make sure that I'm not re-downloading the same package over and over (2) prevent rsync/storing packages that are never used at all
Exactly. http://wiki.jessen.ch/index/How_to_cache_openSUSE_repositories_with_Squid It's been a while since I set this up, and looking at it now, it does seem slightly complicated. It could also do with upgrading to squid 3.x. If you don't already have a working squid proxy, I suspect just rsync'ing the repo will be easier. -- Per Jessen, Zürich (7.9°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2015-05-21 03:21, arvindg@emailuser.net wrote:
Since all the machines are pretty similar, and there's always at least a couple of machines that need the same package, this would
(1) make sure that I'm not re-downloading the same package over and over (2) prevent rsync/storing packages that are never used at all
The question is -- can I do this with zypper, or some other tool?
What I do is that one machine exports the "/var/cache/zypp/packages/" tree, and the rest import it, over nfs. Then in all machines I configure all repos to keep downloaded packages. You have to make sure that all the repos use the same "alias" (the name is irrelevant for this respect), because the directory is named after the alias, not the name. If that's not the case, you have to make symlinks. Then you only have to update machines, and anything that is already in the "/var/cache/zypp/packages/REPO_ALIAS/*" tree is used and no redownloaded. You also have to take care to only update one at a time, because if not two could be downloading the same package at the same time and writing to the same file - collision and corruption. The best thing would be having a dedicated server-daemon, designed by the YaST team, to do this task properly. Zypper would ask this server (local or not) to supply packages or files from the repositories and keep them. Similar to a proxy, but dedicated to this purpose. Per does it with Squid, but it is complicated to setup. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On Thu, 21 May 2015, Carlos E. R. wrote:
On 2015-05-21 03:21, arvindg@emailuser.net wrote:
Since all the machines are pretty similar, and there's always at least a couple of machines that need the same package, this would
(1) make sure that I'm not re-downloading the same package over and over (2) prevent rsync/storing packages that are never used at all
The question is -- can I do this with zypper, or some other tool?
What I do is that one machine exports the "/var/cache/zypp/packages/" tree, and the rest import it, over nfs.
Then in all machines I configure all repos to keep downloaded packages.
You have to make sure that all the repos use the same "alias" (the name is irrelevant for this respect), because the directory is named after the alias, not the name. If that's not the case, you have to make symlinks.
Then you only have to update machines, and anything that is already in the "/var/cache/zypp/packages/REPO_ALIAS/*" tree is used and no redownloaded.
You also have to take care to only update one at a time, because if not two could be downloading the same package at the same time and writing to the same file - collision and corruption.
The best thing would be having a dedicated server-daemon, designed by the YaST team, to do this task properly. Zypper would ask this server (local or not) to supply packages or files from the repositories and keep them. Similar to a proxy, but dedicated to this purpose.
Per does it with Squid, but it is complicated to setup.
Thanks for interesting suggestion - this is much less complicated than setting up squid. I gave NFS a try, but had some issues when yast2 tried to make some hard links from the NFS mounted file-system to other locations in /var. In the end I used rsync to accomplish the same thing. As you suggested, I made sure the machines use the same repo aliases and set keeppackages on all of the .repo files. I have one desktop I always update first. For a second desktop, I have a daily job include an rsync of /var/cache/zypp/packages/. Seems to work. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 2015-05-21 03:21, arvindg@emailuser.net wrote:
Since all the machines are pretty similar, and there's always at least a couple of machines that need the same package, this would
(1) make sure that I'm not re-downloading the same package over and over (2) prevent rsync/storing packages that are never used at all
The question is -- can I do this with zypper, or some other tool?
What I do is that one machine exports the "/var/cache/zypp/packages/" tree, and the rest import it, over nfs.
Then in all machines I configure all repos to keep downloaded packages.
You have to make sure that all the repos use the same "alias" (the name is irrelevant for this respect), because the directory is named after the alias, not the name. If that's not the case, you have to make symlinks.
Then you only have to update machines, and anything that is already in the "/var/cache/zypp/packages/REPO_ALIAS/*" tree is used and no redownloaded.
You also have to take care to only update one at a time, because if not two could be downloading the same package at the same time and writing to the same file - collision and corruption.
The best thing would be having a dedicated server-daemon, designed by the YaST team, to do this task properly. Zypper would ask this server (local or not) to supply packages or files from the repositories and keep them. Similar to a proxy, but dedicated to this purpose.
Per does it with Squid, but it is complicated to setup.
Yes, unless you already have a working squid, I wouldn't recommend my setup, I would just rsync the repos. -- Per Jessen, Zürich (15.4°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-05-25 09:44, Per Jessen wrote:
Carlos E. R. wrote:
Yes, unless you already have a working squid, I wouldn't recommend my setup, I would just rsync the repos.
rsync means a local copy on each machine. With nfs, there is only one central storage, so less resources. On the other hand, with rsync you don't depend on that machine being up, nor are restricted to update one machine at a time. There is choice :-) - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlVjOdsACgkQja8UbcUWM1xArQEAl8ktkhxhTdkIEF3gA14hCHOB UTTR08/my1S77tgBzKEBAKCuXOwBPcVZfVHjb/D3XaQ1rLa/3ZVqlgfxDoq2tPAE =uyH3 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-05-25 09:44, Per Jessen wrote:
Carlos E. R. wrote:
Yes, unless you already have a working squid, I wouldn't recommend my setup, I would just rsync the repos.
rsync means a local copy on each machine.
No, I meant rsync the repo to one machine, then point your installations to that one and serve over http. http://opensuse.carlos.com. -- Per Jessen, Zürich (16.3°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-05-25 17:15, Per Jessen wrote:
Carlos E. R. wrote:
rsync means a local copy on each machine.
No, I meant rsync the repo to one machine, then point your installations to that one and serve over http. http://opensuse.carlos.com.
Ah! :-o (third attempt to send this post) - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlVkk8cACgkQja8UbcUWM1xZEAD/W5gYL1Hdy6zsUw9USSKQ2zaV lmFjUlhpFkpFHqdfEocA/04Zx+8uNTKELt7PA8v0gCmfC33XWBe5stBL4mMDa8KK =ednW -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, 20 May 2015 18:21:54 arvindg@emailuser.net wrote:
Hi
I have a bunch of opensuse machines on a lan.
They're all the same version of opensuse (13.2), all the same arch (x86_64), and all use the same set of repos (in /etc/zypp/repos.d).
I update them all regularly.
This is where debian has it over openSuSE. Apt-cacher does exactly what you want. There are another couple of alternatives, and I think at least one can also cache rpm files for yum (and possibly zypper), but none of them are buit for openSuSE (that I know of). I manage a distributed network of debian (well, actually, rasbian) boxes at work on an internal network that is completely isolated from the internet, apart from one box running apt-cacher. The rest of the boxes query the apt- cacher box and it serves the updates to the rest of the network. It make sense from not only a bandwidth point of view but also a network security point of view too. It would be lovely to have a packaged equivalent for openSuSE. I reckon someone with some programming nous could knock something up in Ruby or Python without too much trouble, but that's not me. Cest la vie. -- ============================================================== Rodney Baker VK5ZTV rodney.baker@iinet.net.au ============================================================== -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2015-05-22 13:22, Rodney Baker wrote:
It would be lovely to have a packaged equivalent for openSuSE. I reckon someone with some programming nous could knock something up in Ruby or Python without too much trouble, but that's not me.
Absolutely! :-) - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlVfHmoACgkQtTMYHG2NR9UpdgCcDEaR/GvIiR+6gJYuA/O/E+py hYIAnReTMWhdquaEQcZR4/0a91wQ173s =hZfU -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri 22 May 2015 02:17:48 PM CDT, Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2015-05-22 13:22, Rodney Baker wrote:
It would be lovely to have a packaged equivalent for openSuSE. I reckon someone with some programming nous could knock something up in Ruby or Python without too much trouble, but that's not me.
Absolutely! :-)
Hi Which was handled by SMT on SLE, now there is SUSE Manager. This uses spacewalk which is built on OBS, but maybe an overkill? -- Cheers Malcolm °¿° LFCS, SUSE Knowledge Partner (Linux Counter #276890) SUSE Linux Enterprise Desktop 12 GNOME 3.10.1 Kernel 3.12.39-47-default up 19:18, 3 users, load average: 0.25, 0.31, 0.31 CPU AMD A4-5150M APU @ 3.3GHz | GPU Richland Radeon HD 8350G -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (8)
-
Anton Aylward -
arvindg＠emailuser.net -
Carlos E. R. -
Carlos E. R. -
Malcolm -
Michael Hamilton -
Per Jessen -
Rodney Baker