http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c2 Josef Möllers changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |Ulrich.Windl@rz.uni-regensb | |urg.de Flags| |needinfo?(Ulrich.Windl@rz.u | |ni-regensburg.de) --- Comment #2 from Josef Möllers --- What is the size of your local copy of suu_14.12.200.69.iso? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 Ulrich Windl changed: What |Removed |Added ---------------------------------------------------------------------------- Flags|needinfo?(Ulrich.Windl@rz.u | |ni-regensburg.de) | -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c5 Ulrich Windl changed: What |Removed |Added ---------------------------------------------------------------------------- Status|RESOLVED |REOPENED Resolution|INVALID |--- --- Comment #5 from Ulrich Windl --- (In reply to Josef Möllers from comment #4)

This looks pretty much like a bug on Dell's side:

Whatever: My complaint was not about seeing the message "416 Requested Range Not Satisfiable", but with wget's conclusion "The file is already fully retrieved; nothing to do." (which is obviously wrong). -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c7 Josef Möllers changed: What |Removed |Added ---------------------------------------------------------------------------- Status|REOPENED |IN_PROGRESS --- Comment #7 from Josef Möllers --- Ah, OK, I see. Sorry for the misinterpretation. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c9 Josef Möllers changed: What |Removed |Added ---------------------------------------------------------------------------- Flags| |needinfo?(Ulrich.Windl@rz.u | |ni-regensburg.de) --- Comment #9 from Josef Möllers --- Created attachment 740645 --> http://bugzilla.opensuse.org/attachment.cgi?id=740645&action=edit Webserver which returns 416 if asked to continue If asked to continue (Range: bytes=...-), this webserver will return a 416 status and also some hideously large Content-Length in the response header. You'll need glib2-devel and microhttpd-devel to build, then just start it in its own directory: "webserver". It will open port 8888 and wait for requests. In a second window, change into the same directory and run "wget --continue --tries=5 http://localhost:8888/webserver". It should retry 5 times and then fail. As the problem seems to have been resolved on the server side, I think I'll close this bug and submit my changes. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c11 --- Comment #11 from Ulrich Windl --- Also: # zypper install /tmp/wget-1.19.1-98.1.x86_64.rpm Loading repository data... Reading installed packages... Resolving package dependencies... Problem: nothing provides libidn2.so.0()(64bit) needed by wget-1.19.1-98.1.x86_64 Solution 1: do not install wget-1.19.1-98.1.x86_64 Solution 2: break wget-1.19.1-98.1.x86_64 by ignoring some of its dependencies Choose from above solutions by number or cancel [1/2/c] (c): -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c13 --- Comment #13 from Ulrich Windl --- I think the web server deliberately refuses service for Linux ;-)

User-Agent: Wget/1.14 (linux-gnu) Range: bytes=8338718720-

Installation worked: Resolving package dependencies... The following 3 NEW packages are going to be installed: libidn2-0 libmetalink3 libpsl5 The following package is going to be upgraded: wget The following package is going to change vendor: wget openSUSE -> obs://build.opensuse.org/network:utilities 1 package to upgrade, 3 new, 1 to change vendor. Unfortunately it did not help for the problem:

wget -N --continue https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO --2017-09-15 09:11:41-- https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Resolving downloads.dell.com (downloads.dell.com)... 172.227.84.40 Connecting to downloads.dell.com (downloads.dell.com)|172.227.84.40|:443... connected. HTTP request sent, awaiting response... 304 Not Modified File ‘SUU_14.12.200.69.ISO’ not modified on server. Omitting download.

(--continue still does not continue) -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c15 --- Comment #15 from Josef Möllers --- (In reply to Ulrich Windl from comment #14)

Without "-N" it does continue:

...

wget --continue https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO --2017-09-15 09:14:28-- https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Resolving downloads.dell.com (downloads.dell.com)... 172.227.84.40 Connecting to downloads.dell.com (downloads.dell.com)|172.227.84.40|:443... connected. HTTP request sent, awaiting response... 206 Partial Content Length: 10589175808 (9.9G), 2250457088 (2.1G) remaining [application/octet-stream] Saving to: ‘SUU_14.12.200.69.ISO’

SUU_14.12.200.69.ISO 78%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++> ] 7.78G 393KB/s eta 94m 5s

(The original wget seemed to continue even with "-N". I'd prefer that logic: If the file is downloaded completely and if there's no newer file, you are done; otherwise if the file on server is newer, get if from start; otherwise continue download of existing file) Actually I'd prefer if "-N" alone would be clever enough to continue a partial download.

I wonder if this is a valid use-case. Wouldn't one do a "wget -N" and then, if that terminated prematurely, do a "wget --continue"? That would need to be discussed upstream as it is a change in behaviour. With this patch, this BZ could then be closed? -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c17 Josef Möllers changed: What |Removed |Added ---------------------------------------------------------------------------- Attachment #740643|0 |1 is obsolete| | --- Comment #17 from Josef Möllers --- Created attachment 740767 --> http://bugzilla.opensuse.org/attachment.cgi?id=740767&action=edit wget version 1.14 with patch applied (In reply to Ulrich Windl from comment #16)

As seen in comment #0 (original version), "-N --continue" tries to resume the download, while in comment #13 (new version) "--continue" is ignored if "-N" is present. When using a script, it's hard for the script to tell whether the last download completed or not, so you would use -N to prevent creating a new file for every download if the file is already there, also avoiding to download a version that is already there, and you would use --continue to continue an incomplete download. Most people won't care about such differences, but it took two days (with interruptions) to download that big file. Or in short: Your fix seems to change the semantics of wget.

As you may have noticed the original version was 1.14, this is now 1.19. I'll have to dig into the sources to see the actual differences, but I've built a 1.14 version and replaced the 1.19 version by that one. Please try and check if the semantics have changed. Sorry for the inconvenience. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c19 --- Comment #19 from Josef Möllers --- (In reply to Ulrich Windl from comment #18)

Sorry:

...

wget -N --continue https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO --2017-09-18 17:02:27-- https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Resolving downloads.dell.com (downloads.dell.com)... 172.227.84.40 Connecting to downloads.dell.com (downloads.dell.com)|172.227.84.40|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable

The file is already fully retrieved; nothing to do.

...

rpm -qf $(which wget) wget-1.14-21.4.1.x86_64

Then, Houston, we have a problem: I tried this on two systems: a SLES12-SP2 and a Leap42.2 and both gave me this: jmoellers@adler:/tmp$ ~/usr/bin/wget -N --continue https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO --2017-09-19 08:35:03-- https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Resolving downloads.dell.com (downloads.dell.com)... 2.21.184.9 Connecting to downloads.dell.com (downloads.dell.com)|2.21.184.9|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable Retrying. --2017-09-19 08:35:04-- (try: 2) https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Connecting to downloads.dell.com (downloads.dell.com)|2.21.184.9|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable Retrying. This was the SLES and the wget wasn't installed but extracted using rpm2cpio and then run from the private copy, but on the Leap42.2 wget was installed from the RPM attached to this BZ (rpm -Uvh) and there the very same happened: retries until the retry counter (eg --retries=5) was exhausted! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c21 --- Comment #21 from Ulrich Windl --- Created attachment 741034 --> http://bugzilla.opensuse.org/attachment.cgi?id=741034&action=edit Compressed output of "ltrace wget ..." I ran wget under ltrace (ltrace -s200 -o wget.ltrace wget -N --continue https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO). The result is reproducible all the time, here. -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c23 --- Comment #23 from Josef Möllers --- Thanks for helping on your side to solve this issue! No matter from where I try (home office in Paderborn or a server in Nürnberg), I keep getting retries! My patch (in src/http.c:2704) is very simple: + if (statcode == HTTP_STATUS_RANGE_NOT_SATISFIABLE + && hs->restval < (contlen + contrange)) + { + /* The file was not completely downloaded, + yet the server claims the range is invalid. + Bail out. */ + CLOSE_INVALIDATE (sock); + xfree (head); + xfree_null (type); + return RANGEERR; + } This part is inserted BEFORE the check for "The file is already fully retrieved; nothing to do." happens. In case of a 416, it compares the size of the local file ("hs->restval" is the "restart value") with the size of the remote file based upon the value in the "Content-Length" header field (contlen) plus the start of the requested "Range". Maybe the server you hit just doesn't send any "Content-Length" in the response header and then "contlen" is left at "-1" (its initial value) and "contrange" is identical to "hs->restval", so the condition is NOT satisfied and the retry does not happen. This would be visible in the reponse header. Can you please save the server's response header(s) and post it/them here? The option is "-S" or "--server-response". Here's the output from "my" server: HTTP request sent, awaiting response... HTTP/1.1 416 Requested Range Not Satisfiable Content-Length: 10589175808 Content-Type: application/octet-stream ... Thanks again! -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c25 --- Comment #25 from Ulrich Windl --- (In reply to Josef Möllers from comment #24)

which indicates that wget is right: the file is complete!

In fact the file is complete now. The point is without the patch the same message was printed when the file was not complete. Maybe I was expecting too much (like first checking the remote file length and the modification time, then if the file is up-to-date don't request any more). When I tried a final test, I ran into a new problem: wget doesn't seem to finish:

truncate -c -s -1 SUU_14.12.200.69.ISO wget -N --continue https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO; echo $? --2017-09-19 12:27:55-- https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Resolving downloads.dell.com (downloads.dell.com)... 23.67.128.54 Connecting to downloads.dell.com (downloads.dell.com)|23.67.128.54|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable Retrying.

--2017-09-19 12:27:56-- (try: 2) https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Connecting to downloads.dell.com (downloads.dell.com)|23.67.128.54|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable Retrying. --2017-09-19 12:27:59-- (try: 3) https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Connecting to downloads.dell.com (downloads.dell.com)|23.67.128.54|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable Retrying. --2017-09-19 12:28:02-- (try: 4) https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Connecting to downloads.dell.com (downloads.dell.com)|23.67.128.54|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable Retrying. --2017-09-19 12:28:06-- (try: 5) https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Connecting to downloads.dell.com (downloads.dell.com)|23.67.128.54|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable Retrying. --2017-09-19 12:28:11-- (try: 6) https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Connecting to downloads.dell.com (downloads.dell.com)|23.67.128.54|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable Retrying. --2017-09-19 12:28:18-- (try: 7) https://downloads.dell.com/FOLDER02721216M/1/SUU_14.12.200.69.ISO Connecting to downloads.dell.com (downloads.dell.com)|23.67.128.54|:443... connected. HTTP request sent, awaiting response... 416 Requested Range Not Satisfiable Retrying. ^C -- You are receiving this mail because: You are on the CC list for the bug.

http://bugzilla.opensuse.org/show_bug.cgi?id=1058204 http://bugzilla.opensuse.org/show_bug.cgi?id=1058204#c27 Josef Möllers changed: What |Removed |Added ---------------------------------------------------------------------------- Status|IN_PROGRESS |RESOLVED Resolution|--- |FIXED --- Comment #27 from Josef Möllers --- Closing this bug. The patch is submitted (and accepted) upstream. -- You are receiving this mail because: You are on the CC list for the bug.