Mirroring updates directory
Hello. Let's suppose I want to have a mirror of *all* 6.4 suse updates. I have several SuSE machines and I'm going to set up on each machine an auto-update script which checks for newer packets by ftp'ing to my local mirror. I've created this script: LOCAL=/usr/local/ftp/pub/suse/update wget -m -nH --cut-dirs=4 -P ${LOCAL} ftp://ftp.suse.de/pub/suse/i386/update/6.4 wget -m -nH --cut-dirs=4 -P ${LOCAL} ftp://ftp.suse.com/pub/suse/i386/update/6.4 First I download packets from the German site which if I'm not wrong they are the ones with license/exports problem. Second, I download the rest of the packets from "official" site. Can I trust rpm's not being duplicated on both sites? What's the SuSE's policy about this? I think I'm right but.. Am I missing some important update packets on another directory? I need to be sure I'm not missing rpm's. The script is executed regularly by cron (one time per week is sufficient?). Is there any way of removing old packets from my mirror? This would be avoid a situation like this: -r--r--r-- 1 root root 453985 Aug 29 2000 timezone-2.1.3-150.i386.rpm -r--rw-r-- 1 root root 452109 Jan 25 17:07 timezone-2.1.3-155.i386.rpm Both files are kept in my local mirror. This is unnecessary. Is it safe to delete automatically (I'll write some script lines) the old files? (eg: compare xxx-2.1.3-150.yyy < xxx-2.1.3-155.yyy and remove the first one). Would I break something? TIA =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Wednesday 18 April 2001 13:18, RoMaN SoFt / LLFB!! wrote:
Hello.
Let's suppose I want to have a mirror of *all* 6.4 suse updates. I have several SuSE machines and I'm going to set up on each machine an auto-update script which checks for newer packets by ftp'ing to my local mirror.
I've created this script:
LOCAL=/usr/local/ftp/pub/suse/update wget -m -nH --cut-dirs=4 -P ${LOCAL} ftp://ftp.suse.de/pub/suse/i386/update/6.4 wget -m -nH --cut-dirs=4 -P ${LOCAL} ftp://ftp.suse.com/pub/suse/i386/update/6.4
First I download packets from the German site which if I'm not wrong they are the ones with license/exports problem. Second, I download the rest of the packets from "official" site. Can I trust rpm's not being duplicated on both sites? What's the SuSE's policy about this?
Why not first download from the international site, and THEN from .de, thereby avoiding the problem altogether ?
I think I'm right but.. Am I missing some important update packets on another directory? I need to be sure I'm not missing rpm's.
You get what's there, I guess... no more no less
The script is executed regularly by cron (one time per week is sufficient?). Is there any way of removing old packets from my mirror? This would be avoid a situation like this: -r--r--r-- 1 root root 453985 Aug 29 2000 timezone-2.1.3-150.i386.rpm -r--rw-r-- 1 root root 452109 Jan 25 17:07 timezone-2.1.3-155.i386.rpm
Both files are kept in my local mirror. This is unnecessary.
If you use the "mirror" perlscript it will take care of these. For my own setup, I made some shellscripts that symlink all those downloaded files into several new directories according to several criteria. Thus I try to avoid downloading huge fontfiles or other unneeded stuff like X when I only want to upgrade a firewall, etc. You get the idea. Also, those symlinks effectively merge the separate .com and .de trees into one directory my clients can download from. YMMV. Works for me.
Is it safe to delete automatically (I'll write some script lines) the old files? (eg: compare xxx-2.1.3-150.yyy < xxx-2.1.3-155.yyy and remove the first one). Would I break something?
You shouldn't end up with those two files in the first place. Use mirror, (as was said already). I second that. P.S.: been there done that. I wanted exactly the same as you want. Maarten
On Wed, 18 Apr 2001 13:59:43 +0200, you wrote:
First I download packets from the German site which if I'm not wrong they are the ones with license/exports problem. Second, I download the rest of the packets from "official" site. Can I trust rpm's not being duplicated on both sites? What's the SuSE's policy about this?
Why not first download from the international site, and THEN from .de, thereby avoiding the problem altogether ?
I don't see the issue here. Could you extend on it? I chose to mirror .de first because I prefer rpms from .com to overwrite .de than .de over .com. Really nothing of this should occur (because .de and .com seems to not have duplicated rpms).
Both files are kept in my local mirror. This is unnecessary.
If you use the "mirror" perlscript it will take care of these.
Read my response to Ralf Koch's post.
For my own setup, I made some shellscripts that symlink all those downloaded files into several new directories according to several criteria. Thus I try to avoid downloading huge fontfiles or other unneeded stuff like X when I only want to upgrade a firewall, etc. You get the idea. Also, those symlinks effectively merge the separate .com and .de trees into one directory my clients can download from. YMMV. Works for me.
Ummm, not a bad idea :-) Another problem I've not talked about is rpms like the kernel one which indeed should be installed (if not rpm fails to check dependencies) but really not used since I use to build (compile) and maintain (patches) my own kernel tree from tarball. This is more flexible, I think.
P.S.: been there done that. I wanted exactly the same as you want.
I'll keep you informed if I get it to work ok ;-) =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Hey, i also use the mirror script for this. My config file for the update of 6.4 looks like this: package=suseupdate comment=suseupdate site=ftp.suse.com local_dir=/daten/suse/suse64/update/ remote_user=anonymous remote_password=torsten@webit.de remote_dir=/pub/suse/i386/update/6.4/ passive_ftp=true mail_to=torsten@slave.xtremeweb.de If you want to exclude some files or directories, you could do that also. Hope that helps. Greetings Maarten van den Berg schrieb:
On Wednesday 18 April 2001 13:18, RoMaN SoFt / LLFB!! wrote:
Hello.
Let's suppose I want to have a mirror of *all* 6.4 suse updates. I have several SuSE machines and I'm going to set up on each machine an auto-update script which checks for newer packets by ftp'ing to my local mirror.
I've created this script:
LOCAL=/usr/local/ftp/pub/suse/update wget -m -nH --cut-dirs=4 -P ${LOCAL} ftp://ftp.suse.de/pub/suse/i386/update/6.4 wget -m -nH --cut-dirs=4 -P ${LOCAL} ftp://ftp.suse.com/pub/suse/i386/update/6.4
First I download packets from the German site which if I'm not wrong they are the ones with license/exports problem. Second, I download the rest of the packets from "official" site. Can I trust rpm's not being duplicated on both sites? What's the SuSE's policy about this?
Why not first download from the international site, and THEN from .de, thereby avoiding the problem altogether ?
I think I'm right but.. Am I missing some important update packets on another directory? I need to be sure I'm not missing rpm's.
You get what's there, I guess... no more no less
The script is executed regularly by cron (one time per week is sufficient?). Is there any way of removing old packets from my mirror? This would be avoid a situation like this: -r--r--r-- 1 root root 453985 Aug 29 2000 timezone-2.1.3-150.i386.rpm -r--rw-r-- 1 root root 452109 Jan 25 17:07 timezone-2.1.3-155.i386.rpm
Both files are kept in my local mirror. This is unnecessary.
If you use the "mirror" perlscript it will take care of these. For my own setup, I made some shellscripts that symlink all those downloaded files into several new directories according to several criteria. Thus I try to avoid downloading huge fontfiles or other unneeded stuff like X when I only want to upgrade a firewall, etc. You get the idea. Also, those symlinks effectively merge the separate .com and .de trees into one directory my clients can download from. YMMV. Works for me.
Is it safe to delete automatically (I'll write some script lines) the old files? (eg: compare xxx-2.1.3-150.yyy < xxx-2.1.3-155.yyy and remove the first one). Would I break something?
You shouldn't end up with those two files in the first place. Use mirror, (as was said already). I second that.
P.S.: been there done that. I wanted exactly the same as you want.
Maarten
--------------------------------------------------------------------- To unsubscribe, e-mail: suse-security-unsubscribe@suse.com For additional commands, e-mail: suse-security-help@suse.com
Hello, Torsten Mueller wrote:
i also use the mirror script for this. My config file for the update of 6.4 looks like this:
package=suseupdate comment=suseupdate site=ftp.suse.com local_dir=/daten/suse/suse64/update/ remote_user=anonymous remote_password=torsten@webit.de remote_dir=/pub/suse/i386/update/6.4/ passive_ftp=true mail_to=torsten@slave.xtremeweb.de
Wouldn't it be better to use a mirror site of "ftp.suse.com"? I think that German users (especially users who are connected to "DFN") have a much faster connection to SuSE's primary mirror "ftp.gwdg.de". I am writing this to avoid that many people are simply going to copy your "mirror" config file... "ftp.suse.com" has been under high load for the last months (at least this is my experience), although it became better in last weeks. After the release of "kde-2.0", "kde-2.1", "kde-2.1.1", "XFree86-4.0.3" and the ftp version of SuSE Linux 7.1 it took quite a long time until "ftp.gwdg.de" mirrored all directories completely (perhaps due to high load on "ftp.suse.com"?). Now "ftp.gwdg.de" seems to be more "up to date" again. Best regards, Steffen
Wouldn't it be better to use a mirror site of "ftp.suse.com"? I think that German users (especially users who are connected to "DFN") have a much faster connection to SuSE's primary mirror "ftp.gwdg.de".
This is correct.
I am writing this to avoid that many people are simply going to copy your "mirror" config file... "ftp.suse.com" has been under high load for the last months (at least this is my experience), although it became better in last weeks.
The limit is at 600 users. Currently, there are 468 users online, the bandwidth is always saturated. It is adviseable to use a mirror. There will be more mirrors soon, hopefully. The number of them is not the issue, but the fact that the mirror sites are out of sync very often.
After the release of "kde-2.0", "kde-2.1", "kde-2.1.1", "XFree86-4.0.3" and the ftp version of SuSE Linux 7.1 it took quite a long time until "ftp.gwdg.de" mirrored all directories completely (perhaps due to high load on "ftp.suse.com"?). Now "ftp.gwdg.de" seems to be more "up to date" again.
There have been some problems with the routing between DFN (Deutsches Forschungsnetz) and our US West Coast provider.
Best regards, Steffen
Thanks,
Roman.
--
- -
| Roman Drahtmüller
Hey, To make it short, yes you are right. I thought it was clear to use a mirror near the own (network)location. Greetings Torsten Steffen Moser schrieb:
...
Wouldn't it be better to use a mirror site of "ftp.suse.com"? I think that German users (especially users who are connected to "DFN") have a much faster connection to SuSE's primary mirror "ftp.gwdg.de".
Hi Roman, why inventing the wheel again and again? There is a package called mirror in series n which does all the work for you: Only uploading the new files, deletion of old versions etc. It's worth to have a look on it.... Cheers Ralf
Hello.
Let's suppose I want to have a mirror of *all* 6.4 suse updates. I have several SuSE machines and I'm going to set up on each machine an auto-update script which checks for newer packets by ftp'ing to my local mirror.
I've created this script:
LOCAL=/usr/local/ftp/pub/suse/update wget -m -nH --cut-dirs=4 -P ${LOCAL} ftp://ftp.suse.de/pub/suse/i386/update/6.4 wget -m -nH --cut-dirs=4 -P ${LOCAL} ftp://ftp.suse.com/pub/suse/i386/update/6.4
First I download packets from the German site which if I'm not wrong they are the ones with license/exports problem. Second, I download the rest of the packets from "official" site. Can I trust rpm's not being duplicated on both sites? What's the SuSE's policy about this?
I think I'm right but.. Am I missing some important update packets on another directory? I need to be sure I'm not missing rpm's.
The script is executed regularly by cron (one time per week is sufficient?). Is there any way of removing old packets from my mirror? This would be avoid a situation like this: -r--r--r-- 1 root root 453985 Aug 29 2000 timezone-2.1.3-150.i386.rpm -r--rw-r-- 1 root root 452109 Jan 25 17:07 timezone-2.1.3-155.i386.rpm
Both files are kept in my local mirror. This is unnecessary.
Is it safe to delete automatically (I'll write some script lines) the old files? (eg: compare xxx-2.1.3-150.yyy < xxx-2.1.3-155.yyy and remove the first one). Would I break something?
TIA
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* * Ralf 'coko' Koch * mailto:info@formel4.de * --- Windows-Error: Mouse not found - A mouse driver hasn't been installed. Please click the left mouse button to continue.
On Wed, 18 Apr 2001 13:25:26 +0100, you wrote:
why inventing the wheel again and again? There is a package called mirror in series n which does all the work for you: Only uploading the new files, deletion of old versions etc. It's worth to have a look on it....
As I want to have both sites' updates on the same local directory the "delete old version" style of mirror wouldn't help me since it would destroy all files from the other mirror too (apart from the old files of the current mirror). BTW I'm more interested in the other topics I asked. Thanks for your response. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Simply configure mirror not to delete files then. Kurt Seifried, seifried@securityportal.com Securityportal - your focal point for security on the 'net
On Wed, 18 Apr 2001 13:14:03 -0600, you wrote:
Simply configure mirror not to delete files then.
Kurt, this is the effect I got using "wget -m". Why switching to mirror script if wget is working ok? =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
mirror offers a lot more control. Exclusions, etc. Kurt
On Thursday 19 April 2001 09:36, RoMaN SoFt / LLFB!! wrote:
On Wed, 18 Apr 2001 13:14:03 -0600, you wrote:
Simply configure mirror not to delete files then.
Kurt, this is the effect I got using "wget -m". Why switching to mirror script if wget is working ok?
At least for me, wget did NOT work right. Wget insisted on retrieving the symlinks on the remote ftp server as if they were files, thus doubling the data to download. The manpage (I think) says you can avoid that, but wget subbornly kept on retrieving everything twice. That was when I switched to "mirror" (as the ftpsite-admin discouraged the use of rsync). I don't understand what you have against mirror anyway, as it's just as easy as using wget... Greetings, Maarten
Because mirror supports regex, wget doesn't. You can be much much more selective/intelligent with mirror then you can with wget. Kurt
On Wed, 18 Apr 2001, RoMaN SoFt / LLFB!! wrote:
First I download packets from the German site which if I'm not wrong they are the ones with license/exports problem. Second, I download the rest of the packets from "official" site. Can I trust rpm's not being duplicated on both sites? What's the SuSE's policy about this?
6.4 is an example where duplicated packages with differnet file-sizes exist (only source-packages so far): zq1/apache-1.3.17-0.src.rpm and zq1/apache-1.3.19-1.src.rpm So I assume you couldn't rely on the filenames being unique. c'ya sven -- The Internet treats censorship as a routing problem, and routes around it. (John Gilmore on http://www.cygnus.com/~gnu/)
On Wed, 18 Apr 2001 22:28:34 +0200 (CEST), you wrote:
6.4 is an example where duplicated packages with differnet file-sizes exist (only source-packages so far):
zq1/apache-1.3.17-0.src.rpm and zq1/apache-1.3.19-1.src.rpm
I didn't explain well. With "duplicated" I mean duplicated soft links (comparing .de to .com site) which indeed are the ones it matters (because each soft link is linked to the last version of the real .rpm or at least it should so). Therefore to choose always the latest .rpm you only have to follow the soft links. In the example you chose: lrwxrwxrwx 1 root root 23 Apr 18 13:37 apache.spm -> apache-1.3.19-1. src.rpm If .com site has duplicated soft links comparing to .de site they would be overwriten by the one which latest dating. BTW the .spm directory doesn't use to be included in automatized tasks when auto-updating system. Indeed they are not needed in many cases. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= ** RoMaN SoFt / LLFB ** roman@madrid.com http://pagina.de/romansoft ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
participants (8)
-
Kurt Seifried -
Maarten van den Berg -
Ralf Koch -
Roman Drahtmueller -
RoMaN SoFt / LLFB!! -
Steffen Moser -
Sven Koch -
Torsten Mueller