[SLE] On-demand caching local mirror?
I'm wondering if anyone can come up with an answer. I'm after a piece of software that will "mirror" an ftp directory (as an example) locally, but will not retrieve the files unless read. I'm sort of thinking of a FUSE sort of local filesystem. i.e. The SUSE updates repository. 1. Set up a local folder. 2. Get the remote file list. 3. An app requests a file. 4. If in local cache, return file, goto 3 5. If not local, and if connection closed, reopen. 6. Try to retrieve, then return file, goto 3 (every so often (on mount, once/day) repeat 2) The point being that I have multiple machines I want to update, so I don't want to waste bandwidth by a) downloading the kde updates repeatedly, or b) replicating the whole folder with updates for packages I haven't installed. It seems like this would be a sensible addition to SUSE for those of us with multiple machines, (applying to virtual ones too.) I've googled, and gone through lots of pages, but I can't find something that operates like this. The closest I seen so far is fuseftp. The two problems I can see with this are 1. I think the cache expiry time is specified in seconds, and I can't see if there is an indefinate (i.e. passing 0) 2. I think it holds the ftp connection open while it is mounted, which wouldn't be good for the servers, with people holding connections open. Any suggestions? -- Steve Boddy -- Check the headers for your unsubscription address For additional commands send e-mail to suse-linux-e-help@suse.com Also check the archives at http://lists.suse.com Please read the FAQs: suse-linux-e-faq@suse.com
On Thursday 13 July 2006 03:21, Stephen Boddy wrote:
The point being that I have multiple machines I want to update, so I don't want to waste bandwidth by a) downloading the kde updates repeatedly, or b) replicating the whole folder with updates for packages I haven't installed.
It seems like this would be a sensible addition to SUSE for those of us with multiple machines, (applying to virtual ones too.)
I agree entirely. I think it's weird that something like this, which would be really handy in a small office setup, for instance, is not a YaST module - it's almost as if people are assuming that you will either have a lonely Linux server on you LAN, or an enterprise-level 800-desk setup, but nothing in between. If you are using Smart, one thing you could do is set it not to remove packages after download: smart config --set remove-packages=false and then share that PC's /var/lib/smart/packages over NFS. Then, on the other PCs, symlink their /var/lib/smart/packages dirs to that NFS share. Everything downloaded by any PC will then be in the source PC's /var/lib/smart/packages dir, and any package slated for update on a PC will be fetched from there if it exists, rather than being downloaded. -- Pob hwyl / Best wishes Kevin Donnelly www.kyfieithu.co.uk - KDE yn Gymraeg www.eurfa.org.uk - Geiriadur rhydd i'r Gymraeg www.rhedadur.org.uk - Rhedeg berfau Cymraeg www.cymrux.org.uk - Linux Cymraeg ar un CD -- Check the headers for your unsubscription address For additional commands send e-mail to suse-linux-e-help@suse.com Also check the archives at http://lists.suse.com Please read the FAQs: suse-linux-e-faq@suse.com
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Thursday 2006-07-13 at 10:22 +0100, Kevin Donnelly wrote:
It seems like this would be a sensible addition to SUSE for those of us with multiple machines, (applying to virtual ones too.)
I agree entirely. I think it's weird that something like this, which would be really handy in a small office setup, for instance, is not a YaST module - it's almost as if people are assuming that you will either have a lonely Linux server on you LAN, or an enterprise-level 800-desk setup, but nothing in between.
Me too
If you are using Smart, one thing you could do is set it not to remove packages after download: smart config --set remove-packages=false and then share that PC's /var/lib/smart/packages over NFS. Then, on the other PCs, symlink their /var/lib/smart/packages dirs to that NFS share. Everything downloaded by any PC will then be in the source PC's /var/lib/smart/packages dir, and any package slated for update on a PC will be fetched from there if it exists, rather than being downloaded.
That reminds me. Previously, in 9.3, all updates were locally stored in "/var/lib/YaST2/you/mnt/i386/update/9.3", but in 10.1 I can't find where they are, that directory tree no longer exists. Where are patches stored now? The trick you mention for smart should be possible with yast, too. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFEti5JtTMYHG2NR9URAthvAJ9k0qZq+gXhxz3XHjcCFe/iz+W2qACdEBNJ aKa0sregTvkFUXYoxRHANr4= =RTdA -----END PGP SIGNATURE----- -- Check the headers for your unsubscription address For additional commands send e-mail to suse-linux-e-help@suse.com Also check the archives at http://lists.suse.com Please read the FAQs: suse-linux-e-faq@suse.com
On Thursday 13 July 2006 12:28, Carlos E. R. wrote:
The Thursday 2006-07-13 at 10:22 +0100, Kevin Donnelly wrote:
It seems like this would be a sensible addition to SUSE for those of us with multiple machines, (applying to virtual ones too.)
I agree entirely. I think it's weird that something like this, which would be really handy in a small office setup, for instance, is not a YaST module - it's almost as if people are assuming that you will either have a lonely Linux server on you LAN, or an enterprise-level 800-desk setup, but nothing in between.
Me too
If you are using Smart, one thing you could do is set it not to remove packages after download: smart config --set remove-packages=false and then share that PC's /var/lib/smart/packages over NFS. Then, on the other PCs, symlink their /var/lib/smart/packages dirs to that NFS share. Everything downloaded by any PC will then be in the source PC's /var/lib/smart/packages dir, and any package slated for update on a PC will be fetched from there if it exists, rather than being downloaded.
That reminds me. Previously, in 9.3, all updates were locally stored in "/var/lib/YaST2/you/mnt/i386/update/9.3", but in 10.1 I can't find where they are, that directory tree no longer exists.
Where are patches stored now?
The trick you mention for smart should be possible with yast, too.
That may work, but has some problems. The folder would have to be writable for all machines accessing, or it won't work. As a result, any machine that is misconfigured will erase the "cache". As to the location, with the changes in 10.1 it's anyones guess to where they are now. It also assumes that libzypp et al have a similar seperate cached directory of downloaded rpms, and that multiple machines accessing it won't cause the machines to trip over each other. As for smart, once I have a distribution with package management, I'm very reluctant to use a system not delivered through that distro. i.e. is smart just another "perspective" on the same files supplied by SuSE? Who updates the "perspective", and is it done in a timely fashion? I did come across this page/site though. http://minkirri.apana.org.au/~abo/projects/ It is /exactly/ the principle I'm thinking of, but it appears to be a bit out of date, and possibly incomplete. I haven't downloaded it and checked it out yet. He has some sort of python based VFS (osVFS), and a mirror program (mirrord) that do this. Probably worth a look, and some of my time if it works. I'm comfortable with Python so I should be able to get it working ;-) -- Steve Boddy -- Check the headers for your unsubscription address For additional commands send e-mail to suse-linux-e-help@suse.com Also check the archives at http://lists.suse.com Please read the FAQs: suse-linux-e-faq@suse.com
On Thursday 13 July 2006 14:43, Stephen Boddy wrote:
On Thursday 13 July 2006 12:28, Carlos E. R. wrote:
The Thursday 2006-07-13 at 10:22 +0100, Kevin Donnelly wrote:
It seems like this would be a sensible addition to SUSE for those of us with multiple machines, (applying to virtual ones too.)
I agree entirely. I think it's weird that something like this, which would be really handy in a small office setup, for instance, is not a YaST module - it's almost as if people are assuming that you will either have a lonely Linux server on you LAN, or an enterprise-level 800-desk setup, but nothing in between.
Me too
If you are using Smart, one thing you could do is set it not to remove packages after download: smart config --set remove-packages=false and then share that PC's /var/lib/smart/packages over NFS. Then, on the other PCs, symlink their /var/lib/smart/packages dirs to that NFS share. Everything downloaded by any PC will then be in the source PC's /var/lib/smart/packages dir, and any package slated for update on a PC will be fetched from there if it exists, rather than being downloaded.
That reminds me. Previously, in 9.3, all updates were locally stored in "/var/lib/YaST2/you/mnt/i386/update/9.3", but in 10.1 I can't find where they are, that directory tree no longer exists.
Where are patches stored now?
The trick you mention for smart should be possible with yast, too.
That may work, but has some problems. The folder would have to be writable for all machines accessing, or it won't work. As a result, any machine that is misconfigured will erase the "cache".
As to the location, with the changes in 10.1 it's anyones guess to where they are now. It also assumes that libzypp et al have a similar seperate cached directory of downloaded rpms, and that multiple machines accessing it won't cause the machines to trip over each other.
As for smart, once I have a distribution with package management, I'm very reluctant to use a system not delivered through that distro. i.e. is smart just another "perspective" on the same files supplied by SuSE? Who updates the "perspective", and is it done in a timely fashion?
I did come across this page/site though. http://minkirri.apana.org.au/~abo/projects/ It is /exactly/ the principle I'm thinking of, but it appears to be a bit out of date, and possibly incomplete. I haven't downloaded it and checked it out yet. He has some sort of python based VFS (osVFS), and a mirror program (mirrord) that do this. Probably worth a look, and some of my time if it works. I'm comfortable with Python so I should be able to get it working ;-) --
OK, I'm going to eat my words, and do a public turnaround. I've tried smart, and I'm very impressed. It's fast and clever. I had some conflicts on my main system (10.0) which were all over the place. 4+ packages with conflicts and recommendations to delete another dozen or so. I ran "smart fix" and it just repaired it with fewer, different removals. And it possibly didn't need to remove all of those, as I was still grokking the interface, and hadn't enabled all the repositories yet. Also on the plus side is that the tray icon is more intuitive that the green gecko. Not a problem for me, but it would be for some of the people I set up for. Love the parallel/mirrored downloads. It saturated my broadband at a constant 240 KB/s or so, as opposed to YaST limping along pulling one file at a time, and installing it before doanloading the next file, all from a single mirror. The only downsides I've discovered are: - The lack of "patch-sets", which is not a problem for my home systems, but might be in a business setting. - The gtk gui seems rather flaky. More than a couple of minutes usage accessing varied functions tends to make it go unstable and crash, and sometimes either the download or install seems to freeze up. - The gtk download gui at full screen hogs the CPU a bit. (Not sure if that's Python, GTK, X or display drivers at fault.) Even with the downsides, as soon as I'd finished with that system I went and installed on the 10.1 laptop I have. Smart forever!!!! I know Python, so I might even be able to help out with minor fixes, although having looked at the code already, it takes a little effort to get your bearings. -- Steve Boddy -- Check the headers for your unsubscription address For additional commands send e-mail to suse-linux-e-help@suse.com Also check the archives at http://lists.suse.com Please read the FAQs: suse-linux-e-faq@suse.com
On Thursday 13 July 2006 10:32, Per Jessen wrote:
Stephen Boddy wrote:
Any suggestions?
squid? I haven't thought it through, but squid was the first thing that came to mind.
This is somewhat in Squids domain of abilities, but I think you'd have to twist it into a tortuous configuration. i.e. Restricting to specified server/paths, cache huge files, remove cache expiry? I have no real practical experience with Squid to know if this would be a workable answer, and the docs I've skimmed are of the basic/advanced type, with no middle ground. i.e. "The default config will work for 95% of you; don't touch" or "parameter diskumbobulator_expiry_delta regulates the fibrilators affect on widgets", with no explanation of how it all ties together. Then you have the issue of configuring YaST to use the proxy, without other programs using it, as the config is now wholly inappropriate. Ta for the suggestion though. -- Steve Boddy -- Check the headers for your unsubscription address For additional commands send e-mail to suse-linux-e-help@suse.com Also check the archives at http://lists.suse.com Please read the FAQs: suse-linux-e-faq@suse.com
participants (4)
-
Carlos E. R. -
Kevin Donnelly -
Per Jessen -
Stephen Boddy