[opensuse] Trying to use aria2c instead of wget to download a pattern of files from a site
Ok, the goal is to download these patterns: "kernel*rpm" "nfs-kernel*rpm" "virtualbox*rpm" from "http://download.opensuse.org/update/leap/15.1/oss/x86_64/" The advantage with aria2c is that it talks to the MirrorBrain and downloads from several severs at max speed. The problem is that aria2c doesn't do recursive or pattern listing of the site. I mean, that I would have to download the index.html file and parse it to obtain the exact names to then feed aria2c. That is too much for me. Somebody has done this before? -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.0 (Legolas))
On Tue, 11 Jun 2019 18:46:57 +0200 "Carlos E. R." <robin.listas@telefonica.net> wrote:
Ok,
the goal is to download these patterns:
"kernel*rpm" "nfs-kernel*rpm" "virtualbox*rpm"
from
"http://download.opensuse.org/update/leap/15.1/oss/x86_64/"
The advantage with aria2c is that it talks to the MirrorBrain and downloads from several severs at max speed.
The problem is that aria2c doesn't do recursive or pattern listing of the site. I mean, that I would have to download the index.html file and parse it to obtain the exact names to then feed aria2c. That is too much for me.
Somebody has done this before?
I just discovered that I have aria2c installed, so presumably I tried it at some time (perhaps you mentioned it before?) but I never use it and don't remember anything about it. I also discovered I have uget-gtk installed. But when I start it and go to the Help menu, pretty much everything I click on takes me to: "Nothing Found "It seems we can’t find what you’re looking for. Perhaps searching can help." and indeed clicking on the links on that site to Documentation get the same response. There's no man page installed either. Seems like abandonware? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 11/06/2019 19.28, Dave Howorth wrote:
On Tue, 11 Jun 2019 18:46:57 +0200 "Carlos E. R." <> wrote:
Somebody has done this before?
I just discovered that I have aria2c installed, so presumably I tried it at some time (perhaps you mentioned it before?) but I never use it and don't remember anything about it.
Yes, I have mentioned it often, soon after it was invented :-) It is the optimal tool for downloading the openSUSE installation isos. As I said, it talks to the MirrorBrain and selects several optimal mirrors to effectuate the download at max speed. wget instead downloads from a single mirror, and if the one it selects is bad, you are stuck.
I also discovered I have uget-gtk installed. But when I start it and go to the Help menu, pretty much everything I click on takes me to:
"Nothing Found
I don't know that tool. I don't have it installed.
"It seems we can’t find what you’re looking for. Perhaps searching can help."
and indeed clicking on the links on that site to Documentation get the same response. There's no man page installed either. Seems like abandonware?
No idea :-? -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
11.06.2019 19:46, Carlos E. R. пишет:
Ok,
the goal is to download these patterns:
"kernel*rpm" "nfs-kernel*rpm" "virtualbox*rpm"
from
"http://download.opensuse.org/update/leap/15.1/oss/x86_64/"
The advantage with aria2c is that it talks to the MirrorBrain and downloads from several severs at max speed.
The problem is that aria2c doesn't do recursive or pattern listing of the site. I mean, that I would have to download the index.html file and parse it to obtain the exact names to then feed aria2c. That is too much for me.
lynx -listonly -dump http://download.opensuse.org/update/leap/15.1/oss/x86_64
Somebody has done this before?
* Andrei Borzenkov <arvidjaar@gmail.com> [06-11-19 14:02]:
11.06.2019 19:46, Carlos E. R. пишет:
Ok,
the goal is to download these patterns:
"kernel*rpm" "nfs-kernel*rpm" "virtualbox*rpm"
from
"http://download.opensuse.org/update/leap/15.1/oss/x86_64/"
The advantage with aria2c is that it talks to the MirrorBrain and downloads from several severs at max speed.
The problem is that aria2c doesn't do recursive or pattern listing of the site. I mean, that I would have to download the index.html file and parse it to obtain the exact names to then feed aria2c. That is too much for me.
lynx -listonly -dump http://download.opensuse.org/update/leap/15.1/oss/x86_64
Somebody has done this before?
lynx -listonly -dump -nonumbers \ http://download.opensuse.org/update/leap/15.1/oss/x86_64/ -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Registered Linux User #207535 @ http://linuxcounter.net Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 11/06/2019 21.21, Patrick Shanahan wrote:
* Andrei Borzenkov <> [06-11-19 14:02]:
lynx -listonly -dump http://download.opensuse.org/update/leap/15.1/oss/x86_64
Somebody has done this before?
lynx -listonly -dump -nonumbers \ http://download.opensuse.org/update/leap/15.1/oss/x86_64/
Thanks! :-) I'm impressed you two came with that so fast :-o :-) It produced a list in less than a second, I'm also impressed at that. 171755 bytes. Ah, ok, the update repository is still small, that's why it goes so fast. still, wget thinks longer. This concoction generates the wanted download list: grep -e "kernel.*rpm\|nfs-kernel.*rpm\|virtualbox.*rpm " list | \ grep -v "\.mirrorlist\|.drpm" But aria2c downloads duplicates, which it renames. :-? -rw-r--r-- 1 cer users 67263444 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.1.rpm -rw-r--r-- 1 cer users 67263444 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.rpm -rw-r--r-- 1 cer users 27761 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.rpm.1.meta4 -rw-r--r-- 1 cer users 27761 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.rpm.meta4 -rw-r--r-- 1 cer users 26482500 Jun 11 22:30 kernel-debug-base-4.12.14-lp151.28.4.1.x86_64.1.rpm -rw-r--r-- 1 cer users 26482500 Jun 11 22:29 kernel-debug-base-4.12.14-lp151.28.4.1.x86_64.rpm ... Indeed, the list file contains duplicates, if file seems to be listed twice: http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-4.12.1... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-4.12.1... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-4.12.1... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-base-4... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-base-4... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-base-4... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-devel-... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-devel-... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-devel-... So I will have to use "uniq". ... There is a remaining problem: a second run downloads a second copy, renamed *.1.rpm. I fail to see the adequate option in the aria2c man :-? Maybe... --allow-overwrite [true|false] Restart download from scratch if the corresponding control file doesn’t exist. See also --auto-file-renaming option. Default: false --auto-file-renaming [true|false] Rename file name if the same file already exists. This option works only in HTTP(S)/FTP download. The new file name has a dot and a number(1..9999) appended after the name, but before the file extension, if any. Default: true Ok, got something that works: lynx -listonly -dump -nonumbers "http://download.opensuse.org/update/leap/15.1/oss/x86_64/" > 0list_1 uniq 0list_1 0list_2 grep -e "kernel.*rpm\|nfs-kernel.*rpm\|virtualbox.*rpm" 0list_2 | grep -v "\.mirrorlist\|.drpm" > 0list_3 aria2c --continue --allow-overwrite=false --auto-file-renaming=false --input-file=0list_3 lynx -listonly -dump -nonumbers http://download.opensuse.org/update/leap/15.1/oss/noarch/ > 0list_1 uniq 0list_1 0list_2 grep -e "kernel.*rpm\|nfs-kernel.*rpm\|virtualbox.*rpm" 0list_2 | grep -v "\.mirrorlist\|.drpm" > 0list_3 aria2c --continue --allow-overwrite=false --auto-file-renaming=false --input-file=0list_3 rm *meta4 *.aria2 -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
* Carlos E. R. <robin.listas@telefonica.net> [06-11-19 16:59]:
On 11/06/2019 21.21, Patrick Shanahan wrote:
* Andrei Borzenkov <> [06-11-19 14:02]:
lynx -listonly -dump http://download.opensuse.org/update/leap/15.1/oss/x86_64
Somebody has done this before?
lynx -listonly -dump -nonumbers \ http://download.opensuse.org/update/leap/15.1/oss/x86_64/
Thanks! :-)
I'm impressed you two came with that so fast :-o :-)
It produced a list in less than a second, I'm also impressed at that. 171755 bytes. Ah, ok, the update repository is still small, that's why it goes so fast. still, wget thinks longer.
This concoction generates the wanted download list:
grep -e "kernel.*rpm\|nfs-kernel.*rpm\|virtualbox.*rpm " list | \ grep -v "\.mirrorlist\|.drpm"
But aria2c downloads duplicates, which it renames. :-?
-rw-r--r-- 1 cer users 67263444 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.1.rpm -rw-r--r-- 1 cer users 67263444 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.rpm -rw-r--r-- 1 cer users 27761 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.rpm.1.meta4 -rw-r--r-- 1 cer users 27761 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.rpm.meta4 -rw-r--r-- 1 cer users 26482500 Jun 11 22:30 kernel-debug-base-4.12.14-lp151.28.4.1.x86_64.1.rpm -rw-r--r-- 1 cer users 26482500 Jun 11 22:29 kernel-debug-base-4.12.14-lp151.28.4.1.x86_64.rpm ...
Indeed, the list file contains duplicates, if file seems to be listed twice:
http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-4.12.1... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-4.12.1... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-4.12.1... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-base-4... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-base-4... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-base-4... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-devel-... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-devel-... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-devel-...
So I will have to use "uniq".
...
There is a remaining problem: a second run downloads a second copy, renamed *.1.rpm. I fail to see the adequate option in the aria2c man :-?
Maybe...
--allow-overwrite [true|false] Restart download from scratch if the corresponding control file doesn’t exist. See also --auto-file-renaming option. Default: false
--auto-file-renaming [true|false] Rename file name if the same file already exists. This option works only in HTTP(S)/FTP download. The new file name has a dot and a number(1..9999) appended after the name, but before the file extension, if any. Default: true
Ok, got something that works:
lynx -listonly -dump -nonumbers "http://download.opensuse.org/update/leap/15.1/oss/x86_64/" > 0list_1 uniq 0list_1 0list_2
grep -e "kernel.*rpm\|nfs-kernel.*rpm\|virtualbox.*rpm" 0list_2 | grep -v "\.mirrorlist\|.drpm" > 0list_3 aria2c --continue --allow-overwrite=false --auto-file-renaming=false --input-file=0list_3
lynx -listonly -dump -nonumbers http://download.opensuse.org/update/leap/15.1/oss/noarch/ > 0list_1 uniq 0list_1 0list_2 grep -e "kernel.*rpm\|nfs-kernel.*rpm\|virtualbox.*rpm" 0list_2 | grep -v "\.mirrorlist\|.drpm" > 0list_3 aria2c --continue --allow-overwrite=false --auto-file-renaming=false --input-file=0list_3
rm *meta4 *.aria2
or maybe the man page for aria2c --continue=true and/or --auto-file-renaming=false you test and you choose -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Registered Linux User #207535 @ http://linuxcounter.net Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
* Patrick Shanahan <paka@opensuse.org> [06-11-19 17:20]:
* Carlos E. R. <robin.listas@telefonica.net> [06-11-19 16:59]:
On 11/06/2019 21.21, Patrick Shanahan wrote:
* Andrei Borzenkov <> [06-11-19 14:02]:
lynx -listonly -dump http://download.opensuse.org/update/leap/15.1/oss/x86_64
Somebody has done this before?
lynx -listonly -dump -nonumbers \ http://download.opensuse.org/update/leap/15.1/oss/x86_64/
Thanks! :-)
I'm impressed you two came with that so fast :-o :-)
It produced a list in less than a second, I'm also impressed at that. 171755 bytes. Ah, ok, the update repository is still small, that's why it goes so fast. still, wget thinks longer.
This concoction generates the wanted download list:
grep -e "kernel.*rpm\|nfs-kernel.*rpm\|virtualbox.*rpm " list | \ grep -v "\.mirrorlist\|.drpm"
But aria2c downloads duplicates, which it renames. :-?
-rw-r--r-- 1 cer users 67263444 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.1.rpm -rw-r--r-- 1 cer users 67263444 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.rpm -rw-r--r-- 1 cer users 27761 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.rpm.1.meta4 -rw-r--r-- 1 cer users 27761 Jun 11 22:29 kernel-debug-4.12.14-lp151.28.4.1.x86_64.rpm.meta4 -rw-r--r-- 1 cer users 26482500 Jun 11 22:30 kernel-debug-base-4.12.14-lp151.28.4.1.x86_64.1.rpm -rw-r--r-- 1 cer users 26482500 Jun 11 22:29 kernel-debug-base-4.12.14-lp151.28.4.1.x86_64.rpm ...
Indeed, the list file contains duplicates, if file seems to be listed twice:
http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-4.12.1... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-4.12.1... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-4.12.1... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-base-4... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-base-4... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-base-4... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-devel-... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-devel-... http://download.opensuse.org/update/leap/15.1/oss/x86_64/kernel-debug-devel-...
So I will have to use "uniq".
...
There is a remaining problem: a second run downloads a second copy, renamed *.1.rpm. I fail to see the adequate option in the aria2c man :-?
Maybe...
--allow-overwrite [true|false] Restart download from scratch if the corresponding control file doesn’t exist. See also --auto-file-renaming option. Default: false
--auto-file-renaming [true|false] Rename file name if the same file already exists. This option works only in HTTP(S)/FTP download. The new file name has a dot and a number(1..9999) appended after the name, but before the file extension, if any. Default: true
Ok, got something that works:
lynx -listonly -dump -nonumbers "http://download.opensuse.org/update/leap/15.1/oss/x86_64/" > 0list_1 uniq 0list_1 0list_2
grep -e "kernel.*rpm\|nfs-kernel.*rpm\|virtualbox.*rpm" 0list_2 | grep -v "\.mirrorlist\|.drpm" > 0list_3 aria2c --continue --allow-overwrite=false --auto-file-renaming=false --input-file=0list_3
lynx -listonly -dump -nonumbers http://download.opensuse.org/update/leap/15.1/oss/noarch/ > 0list_1 uniq 0list_1 0list_2 grep -e "kernel.*rpm\|nfs-kernel.*rpm\|virtualbox.*rpm" 0list_2 | grep -v "\.mirrorlist\|.drpm" > 0list_3 aria2c --continue --allow-overwrite=false --auto-file-renaming=false --input-file=0list_3
rm *meta4 *.aria2
or maybe the man page for aria2c --continue=true and/or --auto-file-renaming=false
you test and you choose
doh! too much cruft! -- (paka)Patrick Shanahan Plainfield, Indiana, USA @ptilopteri http://en.opensuse.org openSUSE Community Member facebook/ptilopteri Registered Linux User #207535 @ http://linuxcounter.net Photos: http://wahoo.no-ip.org/piwigo paka @ IRCnet freenode -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (4)
-
Andrei Borzenkov -
Carlos E. R. -
Dave Howorth -
Patrick Shanahan