Mailinglist Archive: opensuse (933 mails)

< Previous Next >
[opensuse] fun bash script to automate creating url lists for file download (especially from repos)
  • From: "David C. Rankin" <drankinatty@xxxxxxxxxxxxxxxxxx>
  • Date: Mon, 07 Jun 2010 01:28:30 -0500
  • Message-id: <4C0C918E.1070500@xxxxxxxxxxxxxxxxxx>

I frequently need to pull multiple files from a remote host and I like
to do it
with a list of links that I just feed to wget with the -i option (and -b) and
let it do its thing. The only pain is building the 'getfile' list. I wrote a
little script that helps. If you are interested, you can grab it at:

As the name implies, it uses 'lynx -dump' to generate the list and then
parses the return to leave just the links in the output file. Normally, it will
just create an output file with all links from the urls given on the command
line in a single file 'without' any subdirectories included. Two options modify
this behavior [-d|--dirs] causes subdirectories to be included and [-r|--rpms]
return only links to rpms. (very handy for repos).

The main timesaver of the script is the [-b|--base] option that tells
script to grab the baseURL from the next argument and use that for any
subsequent directories specified on the command line. (again, very handy for
working with repos -- you never have to enter the full url twice). For example,
if you wanted the 11.2 libusb0 packages for i586, x86_64 and the src rpms, all
you would need to do is:

lynxdump -b
src x86_64 -r

to grab the binary and src.rpm links and eliminate all the mirror and
junk. The default output file is ./lynxdump.txt [or as specified with the -o
option]. In the case above, here is the output file you would get:

01:10 alchemy:~> cat lynxdump.txt

All that is needed to retrieve the files now is 'wget -i lynxdump.txt'
and your
done. The script works equally well on 1 URL or 50 URLs. The only limitation is
you cant specify more than 1 baseURL ( two -b options are not allowed). The
script is reasonable commented and can be quickly adapted to handle time when
you just want the .tar.gz files, .pdf files, etc...

For those wanting to learn bash scripting a little better, this is a
script to dissect for a look at command line parameter handling with an array,
array indexing, bash functions and returns, etc. Here the option specification
order or position in the command line is unimportant. (obviously if you specify
-b or -o, the next parameter must be a URL or output file, respectively.

DNH, all, you see anything I could do better, let me know. I tried to
the getdump() function to simply build the dumpSTR (see ## Scraps) at the bottom
of the script, but I couldn't get lynx to accept a variable containing the
pipes, etc... I was wanting to eliminate the 3 calls to lynx -dump in the
function and simple pass the dumpSTR to a single call of lynx -dump at the
bottom of the function, but ... "no dice." Thanks for any feedback you can

Enjoy :p

David C. Rankin, J.D.,P.E.
Rankin Law Firm, PLLC
510 Ochiltree Street
Nacogdoches, Texas 75961
Telephone: (936) 715-9333
Facsimile: (936) 715-9339
To unsubscribe, e-mail: opensuse+unsubscribe@xxxxxxxxxxxx
For additional commands, e-mail: opensuse+help@xxxxxxxxxxxx

< Previous Next >
Follow Ups