wget bug or dumb user?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I'm trying to download the whole 9.3 update tree for a friend without internet access. I'm doing it this way (mind: some very long lines): #!/bin/bash wget --continue --timestamping --quota=1800m -r -l3 --limit-rate=90k \ --no-remove-listing -nH --cut-dirs=5 --output-file=LOGFILE \ --reject "OpenOffice*-af-*rpm,OpenOffice*-ar-*rpm,OpenOffice*-be-*rpm,OpenOffice*-bg-*rpm,OpenOffice*-ca-*rpm,OpenOffice*-cs-*rpm,OpenOffice*-cy-*rpm,OpenOffice*-da-*rpm,OpenOffice*-de-*rpm,OpenOffice*-el-*rpm,OpenOffice*-et-*rpm,OpenOffice*-fi-*rpm,OpenOffice*-gu-*rpm,OpenOffice*-hr-*rpm,OpenOffice*-hu-*rpm,OpenOffice*-it-*rpm,OpenOffice*-ja-*rpm,OpenOffice*-ko-*rpm,OpenOffice*-nb-*rpm,OpenOffice*-nl-*rpm,OpenOffice*-nn-*rpm,OpenOffice*-pa-*rpm,OpenOffice*-pl-*rpm,OpenOffice*-pt-*rpm,OpenOffice*-ru-*rpm,OpenOffice*-sk-*rpm,OpenOffice*-sl-*rpm,OpenOffice*-sv-*rpm,OpenOffice*-xh-*rpm,OpenOffice*-tr-*rpm,OpenOffice*-zh-*rpm,OpenOffice*-zu-*rpm" \ --exclude-directories=/pub/linux/suse/ftp.suse.com/suse/i386/update/9.3/rpm/x86_64 \ "ftp://ftp.gwdg.de/pub/linux/suse/ftp.suse.com/suse/i386/update/9.3/" It is the "--exclude-directories" option that is causing me the biggest problem: after I added it, it never stops, it goes round and round downloading (skipping, actually) the same files. I had to make it log to a file to make sure, it was flipping by so fast. I track the download marking the ".listing" entries in the log (trimmed for clarity): - --20:30:58-- ftp://ftp.gwdg.de/pub/linux/suse/ftp.suse.com/suse/i386/update/9.3/ => `i386/update/9.3/.listing' 20:31:01 (14.83 MB/s) - `i386/update/9.3/.listing' saved [5412] 20:31:06 (71.81 KB/s) - `i386/update/9.3/deltas/.listing' saved [2574208] 20:31:16 (27.28 KB/s) - `i386/update/9.3/patches/.listing' saved [241797] 20:31:24 (27.75 KB/s) - `i386/update/9.3/patches.obsolete/.listing' saved [121069] 20:31:26 (18.86 MB/s) - `i386/update/9.3/rpm/.listing' saved [5655] 20:31:30 (69.83 KB/s) - `i386/update/9.3/rpm/i586/.listing' saved [6910062] 20:32:02 (46.33 MB/s) - `i386/update/9.3/rpm/i686/.listing' saved [34980] 20:32:04 (22.74 KB/s) - `i386/update/9.3/rpm/noarch/.listing' saved [426240] 20:32:07 (50.95 KB/s) - `i386/update/9.3/rpm/src/.listing' saved [2124577] ... Not descending to `pub/linux/suse/ftp.suse.com/suse/i386/update/9.3/rpm/x86_64' as it is excluded/not-included. ... 20:32:13 (70.47 KB/s) - `i386/update/9.3/rpm/i586/.listing' saved [7107435] 20:32:38 (79.04 KB/s) - `i386/update/9.3/rpm/i686/.listing' saved [36146] Notice that it has gone round again to download the same i386/update/9.3/rpm/i586/ directory again! And it continued... ... 20:32:48 (72.20 KB/s) - `i386/update/9.3/rpm/i586/.listing' saved [7304808] 20:33:25 (72.53 KB/s) - `i386/update/9.3/rpm/i586/.listing' saved [7502181] 20:33:57 (70.83 KB/s) - `i386/update/9.3/rpm/i586/.listing' saved [7699554] 20:34:33 (71.51 KB/s) - `i386/update/9.3/rpm/i586/.listing' saved [7896927] 20:35:14 (70.25 KB/s) - `i386/update/9.3/rpm/i586/.listing' saved [8094300] Before I killed it, it had gone round 7 times in 5 minutes - fortunately, it only really downloads the ".listing" files, due to the "--continue" switch. What is happening? :-O [...] Those ".listing" files were increasing in size... yeap, the server was syncing. A while later retried, downloaded some files related to ethereal, and finished normally. Then I tried some hours later, and it also worked normally. Weird... The problem is solved, but... why the fact that the server was adding some new files caused wget to get into a never ending loop? - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFEBMfntTMYHG2NR9URAinRAJ9KoVoidmQXcmAe53w77lS08bHvOACgj2LF tCluItIMup07DXQ3gY2gb0c= =5pZi -----END PGP SIGNATURE-----
On Tuesday 28 February 2006 01:59 pm, Carlos E. R. wrote:
Hi,
I'm trying to download the whole 9.3 update tree for a friend without internet access. I'm doing it this way (mind: some very long lines):
#!/bin/bash wget --continue --timestamping --quota=1800m -r -l3 --limit-rate=90k \ --no-remove-listing -nH --cut-dirs=5 --output-file=LOGFILE \
<snip> I guess this looks like an issue with wget. I run across wierdness myself when downloading webpages both on *nix and Wintendo. Of course, I would have simply opened Konqueror and navigated to the FTP site, copied the files, then pasted locally. :)
Weird...
The problem is solved, but... why the fact that the server was adding some new files caused wget to get into a never ending loop?
<insert Twilight Zone music here> -- kai - www.perfectreign.com www.livebeans.com - the new NetBeans community
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The Tuesday 2006-02-28 at 19:50 -0800, kai wrote:
I guess this looks like an issue with wget. I run across wierdness myself when downloading webpages both on *nix and Wintendo.
Mmm.
Of course, I would have simply opened Konqueror and navigated to the FTP site, copied the files, then pasted locally. :)
I used gftp previously, but then noticed that it was doing double downloads: one for the files, another for the links, as files, not as links. Double networks use, double space used. As for konkeror, I don't use it. I could have tried mc instead. - -- Cheers, Carlos Robinson -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (GNU/Linux) Comment: Made with pgp4pine 1.76 iD8DBQFEBZKPtTMYHG2NR9URAvskAJ9pCriR+JIbijfmxCHLHmqDGsgzqgCfYxAy 2SUfSxFuP8miV6hDXvAYft8= =rK3W -----END PGP SIGNATURE-----
participants (2)
-
Carlos E. R.
-
kai