On Mon, Mar 08, 2010 at 01:53:55PM +0100, Roger Oberholtzer wrote:

On Sat, 2010-02-27 at 10:45 +0530, Arun Khan wrote:

...

After a lot of frustrating events like yours, I point my repos to the appropriate sub dirs under one of the two below - decent speed, no time outs ...

http://mirrors.kernel.org/opensuse/ http://mirror.anl.gov/pub/opensuse/opensuse/

Well, I have been away a week. I was hoping that my earlier issue with the timeouts from this system would have been resolved (expecting them to be mirror issues). This is not the case.

We fixed a configuration issue last Friday, thus the response time should be much better now. I'm surprised that you still seem to have problems. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Hi, Am 08.03.2010 um 15:52 schrieb Roger Oberholtzer:

On Mon, 2010-03-08 at 14:12 +0100, Michael Schroeder wrote:

...

On Mon, Mar 08, 2010 at 01:53:55PM +0100, Roger Oberholtzer wrote:

...

On Sat, 2010-02-27 at 10:45 +0530, Arun Khan wrote:

...

After a lot of frustrating events like yours, I point my repos to the appropriate sub dirs under one of the two below - decent speed, no time outs ...

http://mirrors.kernel.org/opensuse/ http://mirror.anl.gov/pub/opensuse/opensuse/

Well, I have been away a week. I was hoping that my earlier issue with the timeouts from this system would have been resolved (expecting them to be mirror issues). This is not the case.

We fixed a configuration issue last Friday, thus the response time should be much better now. I'm surprised that you still seem to have problems.

Me too. I just tried again, and it is failing at the same point. Here is what zypper prints when it fails. Some packages download, and others do not...

Retrieving: kiwi-doc-4.21-80.1.i586.rpm [error (6.2 MiB/s)] Failed to download ./i586/kiwi-doc-4.21-80.1.i586.rpm from http://download.opensuse.org/repositories/Virtualization%3a/Appliances/openS... Abort, retry, ignore? [a/r/i/?] (a): r Retrieving: kiwi-doc-4.21-80.1.i586.rpm [error (0 B/s)] Failed to download ./i586/kiwi-doc-4.21-80.1.i586.rpm from http://download.opensuse.org/repositories/Virtualization%3a/Appliances/openS... Abort, retry, ignore? [a/r/i/?] (a):

At first, this failure message (which is meaningless) should be fixed to provide details. It should say what the failure exactly is, which will give a hint to the answer why the failure occured and what can be done about it. With an error message unspecific like this, it is difficult to do something about it. Having said that, I checked that all mirrors that have the file http://download.opensuse.org/repositories/Virtualization%3a/Appliances/openS... deliver it correctly (md5 sum 782dc51b4c020372a57af17ae99e3d8e) and swiftly. This included the Swedish mirror (I think you might be in Sweden, but I don't know since you didn't say so). I noticed that ftp.nux.ipb.pt is very slow in sending data. But for files > 1MB (like the kiwi-doc package), that mirror is specially configured as reserve mirror, so it is always given lowest priority in the download description. It is always the last in the list. So I don't think that your download client would ever try that mirror at all. And even if so, I don't know whether that a very slow download might lead to a "Failure to download" (I don't think so). In addition, I notice that all download description (metalinks) which download.opensuse.org serves are lacking verification hashes. That should be fixed as soon as possible. But that's again something which is unlikely to be related to the problems that you are seeing. (For files in repositories/Virtualization:/Appliances, the hashes aren't needed.) What you could do meanwhile is trying out whether your download client works correctly by repeatedly issueing the following commands, and observing the output for speed and errors: aria2c http://download.opensuse.org/repositories/Virtualization:/Appliances/openSUS... rm kiwi-doc-4.21-80.1.i586.rpm* Peter-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Mon, 2010-03-08 at 18:15 +0100, Roger Oberholtzer wrote:

On Mon, 2010-03-08 at 18:02 +0100, Peter Pöml wrote:

...

aria2c http://download.opensuse.org/repositories/Virtualization:/Appliances/openSUS...

This fails consistently. I attached the listing. I am no0t sure if the list allows an attachment. I did it as one so the line breaks would be unmolested.

When it fails, it is very quick. The whole process takes less than a minute. All files are not failing. But many do, making updates impossible.

A bit more info: If I check the size of the downloaded RPM as stated in the metalink file, I see that the downloaded RPM size matches. So, it seems to have been downloaded. But aria2c is saying otherwise... -- Roger Oberholtzer OPQ Systems / Ramböll RST Ramböll Sverige AB Krukmakargatan 21 P.O. Box 17009 SE-104 62 Stockholm, Sweden Office: Int +46 10-615 60 20 Mobile: Int +46 70-815 1696 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Mon, 2010-03-08 at 19:26 +0100, Peter Pöml wrote:

Am 08.03.2010 um 18:15 schrieb Roger Oberholtzer:

...

On Mon, 2010-03-08 at 18:02 +0100, Peter Pöml wrote:

...

aria2c http://download.opensuse.org/repositories/Virtualization:/Appliances/openSUS...

This fails consistently. I attached the listing. I am no0t sure if the list allows an attachment. I did it as one so the line breaks would be unmolested.

The attachment was fine, thanks for caring.

...

When it fails, it is very quick. The whole process takes less than a minute. All files are not failing. But many do, making updates impossible.

-> [HttpResponse.cc:93] Invalid range header. Request: 3,145,728-0/11,498,702, Response: 0-11,498,701/11,498,702 [...] -> [HttpResponse.cc:93] Invalid range header. Request: 3,145,728-0/11,498,702, Response: 0-11,498,701/11,498,702

(This is the least that libzypp should have reported, together with the mirror name.)

Let me risk a guess. (I've seen this before.) There is an intermediate (proxy cache) between your client and the mirrors.

It is a very old Squid cache, which doesn't handle HTTP/1.1 partial requests properly. I'm relatively sure that it works with newer Squid.

That or a bug in aria2c that occurs in such a setting.

Can you confirm that this is due to an intermediate?

Hard to say. This is a company-wide intranet through which I am accessing things. I am pretty sure it is some sort of NAT implemented in Cisco devices. Unfortunately, it also goes through something called Webwasher that checks all downloads for viri and such unwanted cruft. I can inquire if they have changed something around the time my problem surfaced. Perhaps there now is a cache, or a cache gone bad. The IT guys are always a bit guarded in what they tell. Security through obscurity is part of their working model. -- Roger Oberholtzer Ramböll RST/OPQ Ramböll Sverige AB Krukmakargatan 21 P.O. Box 17009 SE-104 62 Stockholm, Sweden Office: Int +46 10-615 60 20 Mobile: Int +46 70-815 1696 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On 03/08/2010 06:02 PM, Peter Pöml wrote:

Am 08.03.2010 um 15:52 schrieb Roger Oberholtzer:

...

On Mon, 2010-03-08 at 14:12 +0100, Michael Schroeder wrote:

...

On Mon, Mar 08, 2010 at 01:53:55PM +0100, Roger Oberholtzer wrote:

...

On Sat, 2010-02-27 at 10:45 +0530, Arun Khan wrote:

...

After a lot of frustrating events like yours, I point my repos to the appropriate sub dirs under one of the two below - decent speed, no time outs ...

http://mirrors.kernel.org/opensuse/ http://mirror.anl.gov/pub/opensuse/opensuse/

Well, I have been away a week. I was hoping that my earlier issue with the timeouts from this system would have been resolved (expecting them to be mirror issues). This is not the case.

We fixed a configuration issue last Friday, thus the response time should be much better now. I'm surprised that you still seem to have problems.

Me too. I just tried again, and it is failing at the same point. Here is what zypper prints when it fails. Some packages download, and others do not...

Retrieving: kiwi-doc-4.21-80.1.i586.rpm [error (6.2 MiB/s)] Failed to download ./i586/kiwi-doc-4.21-80.1.i586.rpm from http://download.opensuse.org/repositories/Virtualization%3a/Appliances/openS... Abort, retry, ignore? [a/r/i/?] (a): r Retrieving: kiwi-doc-4.21-80.1.i586.rpm [error (0 B/s)] Failed to download ./i586/kiwi-doc-4.21-80.1.i586.rpm from http://download.opensuse.org/repositories/Virtualization%3a/Appliances/openS... Abort, retry, ignore? [a/r/i/?] (a):

At first, this failure message (which is meaningless) should be fixed to provide details.

It should already do so in --verbose mode, and on (a)bort. Anyway, i'll try to improve it.

It should say what the failure exactly is, which will give a hint to the answer why the failure occured and what can be done about it. With an error message unspecific like this, it is difficult to do something about it.

-- cheers, jano Ján Kupec YaST team ---------------------------------------------------------(PGP)--- Key ID: 637EE901 Fingerprint: 93B9 C79B 2D20 51C3 800B E09B 8048 46A6 637E E901 ---------------------------------------------------------(IRC)--- Server: irc.freenode.net Nick: jniq Channels: #zypp #yast #suse #susecz ---------------------------------------------------------(EOF)---

Hi Roger, On Mon, Mar 08, 2010 at 06:15:01 +0100, Roger Oberholtzer wrote:

On Mon, 2010-03-08 at 18:02 +0100, Peter Pöml wrote:

...

aria2c http://download.opensuse.org/repositories/Virtualization:/Appliances/openSUS...

This fails consistently. I attached the listing. I am no0t sure if the list allows an attachment. I did it as one so the line breaks would be unmolested.

When it fails, it is very quick. The whole process takes less than a minute. All files are not failing. But many do, making updates impossible.

Could you please re-run the command with "--log=aria2c.log" and provide the resulting long log file? I need to see the full HTTP headers. For a short comparison, the range headers that are passed around normally look like this: % aria2c --log=- http://download.opensuse.org/repositories/Virtualization:/Appliances/openSUS... | grep Range Range: bytes=2883584- Range: bytes=8650752- Range: bytes=5767168- Accept-Ranges: bytes Accept-Ranges: bytes Content-Range: bytes 2883584-11498701/11498702 Accept-Ranges: bytes Content-Range: bytes 5767168-11498701/11498702 Range: bytes=1572864- Accept-Ranges: bytes Content-Range: bytes 1572864-11498701/11498702 Content-Range: bytes 8650752-11498701/11498702 Range: bytes=4718592- Accept-Ranges: bytes Content-Range: bytes 4718592-11498701/11498702 Range: bytes=2359296- Accept-Ranges: bytes Content-Range: bytes 2359296-11498701/11498702 Range: bytes=4194304- Accept-Ranges: bytes Content-Range: bytes 4194304-11498701/11498702 Range: bytes=2097152- Content-Range: bytes 2097152-11498701/11498702

-> [HttpResponse.cc:93] Invalid range header. Request: 3,145,728-0/11,498,702, Response: 0-11,498,701/11,498,702

From the aria2c output you attached, I have the impression that all responses come as if all mirrors don't support byte ranges. The 0-11,498,701 in aria2c's error message indicates that it receives "Content-Range: bytes 0-11498701/11498702" responses. While I _have_ seen this with an old corporate squid cache, it doesn't mean this is due to a squid cache. In fact, any proxy could elicit this effect, by not passing the partial GET request but prefetching the entire content and then passing it back in full. If the HTTP/1.1 protocol is followed correctly by the proxy, aria2c should be able to deal with this case, since support for byte ranges is not obligatory in HTTP/1.1. So the bug might be in several places, and we'd need to see the HTTP headers for details. The aria2c verbose log contains them. Which aria2c version is this? Peter

Hi, Am 10.03.2010 um 13:27 schrieb Roger Oberholtzer:

On Wed, 2010-03-10 at 12:39 +0100, Peter Poeml wrote:

...

aria2c.log

This confirms the suspicions I uttered earlier.

I have attached the file. The download still failed. Popular wisdom (not final verdict) in the IT department is that nothing has changed. Still, I ran it on another computer on the same network, and it also failed. At home, with a different computer but same repos and, until this issue, the same software, it works fine.

The odd thing is that it is failing so quickly. Not a tcp timeout. Some protocol failure.

That's because the protocol is manipulated by an intermediary that your IT department set up, as explained in the other mail. They use a product called Ironport[1] that intercepts requests to the Internet, and interferes with them. It is like "forced" proxy, so to speak. Range requests are disabled in this proxy. The IT department could change the configuration, by tweaking the "rangerequestdownload" parameter as described in the product documentation (I found one at http://www.headtechnology.ru/download/ironport/AsyncOS_Web_UserGuide.pdf - see p. 117). That configuration change would eliminate the issue. Another possible configuration change might be that you bypass the proxy (by not configuring your client to use it), although I assume that's not possible for your client because the proxy intercepts the requests without the clients knowledge. Short of changing the configuration, there are two things to examine. 1) Is Ironport's interception broken, in that it changes the passed headers to something invalid? 2) Is aria2c not correctly dealing with servers that ignore byte ranges I guess that 1) is the case, because if Ironport removes byte range headers from the client, it clearly should also remove the server's announcement for support of byte ranges (which practically all mirrors announce): the "Accept-Ranges: bytes" header. So IMO what we see is a violation of RFC2616 by the Ironport appliance. Might be worthwhile to open a bug with its manufacturer. Peter [1] http://www.ironport.com/products/web_security_appliances.html-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Wed, 2010-03-10 at 14:18 +0100, Peter Pöml wrote:

They use a product called Ironport[1] that intercepts requests to the Internet, and interferes with them. It is like "forced" proxy, so to speak.

This is typically referred to as a "transparent" proxy.

1) Is Ironport's interception broken, in that it changes the passed headers to something invalid? 2) Is aria2c not correctly dealing with servers that ignore byte ranges I guess that 1) is the case, because if Ironport removes byte range headers from the client, it clearly should also remove the server's announcement for support of byte ranges (which practically all mirrors announce): the "Accept-Ranges: bytes" header. So IMO what we see is a violation of RFC2616 by the Ironport appliance. Might be worthwhile to open a bug with its manufacturer.

-- Adam Tauno Williams LPIC-1, Novell CLA http://www.whitemiceconsulting.com OpenGroupware, Cyrus IMAPd, Postfix, OpenLDAP, Samba -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

I am responding to the list even though Radule did not post there. He explains why. But I wanted it in the thread: I am not alone. On Fri, 2010-03-12 at 13:27 +0100, Radule Šoškić wrote:

On 03/12/2010 08:30 AM, Roger Oberholtzer wrote:

...

... But I will start by seeing about the range request issue. Anyone have any ideas on how I can tell that aria2c is even trying to detect if range requests are allowed?

Hi, Roger

Sorry for replying and off the list, but at the moment I can't post there, since our national domain changed from .yu to .sr recently and the list still knows me as a member of .yu. I'll fix it later, hope that I'll figure how to do that without going through all the unsubscribing/subscribing mess...Ok.

I went through this thread and read all the messages. I have exactly the same situation and symptoms: we also have Ironport in the company network and the same troubles with suse updates.

I don't know about the range issue, but I guess that the part of the problem is also in aria2c.

Namely, when I do this:

export ZYPP_ARIA2C=0

and start update in such environment, everything goes ok. What happens with this variable value set to zero is that zypper stops using aria2c and goes back to its old ways (the name of the program is curl, I think).

The other issue with curl is that it needs babysitting. Ie, on every timeout expiration you need to manually confirm that you want it to go on. My guess is that suse (zypper) switch to using aria2c was in an attempt to avoid exactly this curl's babysitting issue.

I bet aria2c tries to do range requests so it can download from multiple sources to improve efficency, and curl does not. In our case, I think that is the significant difference. The question is if aria2c is doing something wrong when ranges are not supported, or if the IronPort is not doing them, but not reporting this properly. Don't know where to point the finger... -- Roger Oberholtzer OPQ Systems / Ramböll RST Ramböll Sverige AB Krukmakargatan 21 P.O. Box 17009 SE-104 62 Stockholm, Sweden Office: Int +46 10-615 60 20 Mobile: Int +46 70-815 1696 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Fri, 2010-03-12 at 13:19 +0000, Dave Howorth wrote:

Roger Oberholtzer wrote:

...

I bet aria2c tries to do range requests so it can download from multiple sources to improve efficency, and curl does not. In our case, I think that is the significant difference. The question is if aria2c is doing something wrong when ranges are not supported, or if the IronPort is not doing them, but not reporting this properly. Don't know where to point the finger...

Why not report the problem to both communities, even if you don't know where to point the finger? Somebody else may have a related problem or be in a better position to investigate the problem. Especially since you have now confirmed that it is not your own finger trouble!

This is my plan. The IronPort issue has been reported to our IT department, who are investigating the IronPort side of things. Now that I know how to manipulate the problem, I can decide what to do. I was hoping to see if I got any info here on aria2c (as I have requested earlier in the thread) before adding a bug report. This is where I think things stand. If I do not move forward any more via the list, I will file a bug report at Novell for aria2c. Not that I know it is a bug... -- Roger Oberholtzer OPQ Systems / Ramböll RST Ramböll Sverige AB Krukmakargatan 21 P.O. Box 17009 SE-104 62 Stockholm, Sweden Office: Int +46 10-615 60 20 Mobile: Int +46 70-815 1696 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Fri, 2010-03-12 at 13:49 +0000, Dave Howorth wrote:

Roger Oberholtzer wrote:

...

If I do not move forward any more via the list, I will file a bug report at Novell for aria2c. Not that I know it is a bug...

I was meaning reporting it upstream directly:

Ahh. I thought aria2c was an opensuse component. Not something managed on sourceforge. Thanks for that pointer. I will do as you suggest.

http://sourceforge.net/projects/aria2/support - "aria2 - CLI Metalink/BitTorrent client says the best way to get help with its software is by creating a new item in the Bugs Tracker"

http://sourceforge.net/apps/phpbb/aria2/ - "Ask for help. If you found bugs, use bug tracker instead."

http://sourceforge.net/tracker/?func=add&group_id=159897&atid=813673

But a Novell report would probably also be useful.

Cheers, Dave

-- Roger Oberholtzer OPQ Systems / Ramböll RST Ramböll Sverige AB Krukmakargatan 21 P.O. Box 17009 SE-104 62 Stockholm, Sweden Office: Int +46 10-615 60 20 Mobile: Int +46 70-815 1696 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Hi, Am 17.03.2010 um 15:08 schrieb Peter Pöml:

Am 12.03.2010 um 08:30 schrieb Roger Oberholtzer:

...

On Wed, 2010-03-10 at 14:18 +0100, Peter Pöml wrote:

...

Hi,

...

So IMO what we see is a violation of RFC2616 by the Ironport appliance. Might be worthwhile to open a bug with its manufacturer.

I got the IT guys to let me out without going through the proxy. And magically all worked. That does not mean the IronPort is the source of the problem. Do we know that aria2c does the right thing when range requests are disabled? The reason the IronPort disables range requests is that when they are on it cannot check the downloads for malware. It needs the whole thing to do so.

As mentioned before, disabling byte ranges is fine, however then the proxy must also remove the header line that claims that byte ranges are supported. This is a very simple header manipulation that the manufacturer of the appliance just overlooked. Maybe your IT guys could even work around by configuring the removal of the header manually -- if that's possible.

I'd suggest asking them whether they can completely suppress that header line by filtering it.

...

When the range issue is sorted, another issue could be that arias2c would have to wait for the proxy to download the whole item and check it before aria2c sees the first byte. For large/slow items, I guess that could be an issue. Or not.

That would be a principal issue with any intercepting intermediary that scans for malware.

...

But I will start by seeing about the range request issue. Anyone have any ideas on how I can tell that aria2c is even trying to detect if range requests are allowed?

I should note that I don't claim that fixing the problem in IronPort will necessarily make aria2c work in your situation. There could of course also be a problem with aria2c, in addition to the evident problem in IronPort. I tried to download a file http://opensuse.mirrors.proxad.net/opensuse/, which happens to have byte ranges disabled. Just pick a file from there and run aria2c with the URL. aria2c reports some warnings on the way, but downloads correctly in the end (I verified with a cryptographic hash). I interrupt the download, I can't restart it later - which makes sense in a way, because without byte range support there is no way to complete a download other than starting from the beginning.

So I'd say, aria2c does the best it can and works correctly here (despite some warnings that it expresses).

Peter--

Recap (& further analysis after a little bit of testing): - aria2c can download fine from behind the intercepting intermediary or from a mirror that doesn't support byte ranges at all - aria2c also copes with the situation of the broken intermediary in your case - libzypp leaves last attempts downloads around (I strongly suppose), so when _retrying_ a download, aria2c tries to complete the existing file; in order to do that, it _has_ to use byte range requests (or resume in FTP). - that fails if aria2c is behind a proxy like yours (which leaves byterange headers in, but doesn't allow them), or when no mirror supports byte ranges (nearly all do) - aria2c tries all mirrors in that situation, which of course takes a long time. - (quoting Tatsuhiro here) it would be an easy solution for libzypp to delete partial file before download, but if the file size is big and the connection is slow, resuming file will help users. Apparently, the folks over at Mandriva ran into the same situation. I have good news though: Tatsuhiro Tsujikawa (author of aria2c) has added an interesting feature that allows dealing with the situation:

Here is changelog:

Added --always-resume and --max-resume-failure-tries option. If --always-resume=false is given, when all given URIs do not support resume or aria2 encounters N URIs which does not support resume(N is the value specified using --max-resume-failure-tries option), aria2 downloads file from scratch. The default behavior is --always-resume=true, which means if all URIs do not support resume, download fails. I think this is OK because user normally don't like to see that partially downloaded file is overwritten(this is particularly true if file size is big). This option is useful when aria2 is used as download backend and graceful falling back to overwritten behavior is preferable. Added exit status value 8, which means download failed

That should help. Jano, I expect the change to turn up with the next aria2c release. So the new feature could be used in 11.3! Peter-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org