[opensuse] Leap and ZYPP_MULTICURL=0
Does anyone know if zypper in Leap supports ZYPP_MULTICURL=0 ? I ask because I am trying to update a system where I have needed this in the past, and I think it looks like it is exhibiting the same behavior as when this option is not used. I would at least like to eliminate that as a source of problems for zypper downloading RPMs. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Perhaps bad form to reply to one's own post. I might add that google was no help with a search limited to ZYPP_MULTICURL and leap. I don't mind looking for this in the source to see what is going on. But I do not know where to look. I doubt it is in zypper, as it applies to yast as well. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Nov 18, 2015 at 10:07 AM, Roger Oberholtzer <roger.oberholtzer@gmail.com> wrote:
Perhaps bad form to reply to one's own post. I might add that google was no help with a search limited to ZYPP_MULTICURL and leap. I don't mind looking for this in the source to see what is going on. But I do not know where to look. I doubt it is in zypper, as it applies to yast as well.
libzypp? https://github.com/openSUSE/libzypp -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Roger Oberholtzer wrote:
Perhaps bad form to reply to one's own post. I might add that google was no help with a search limited to ZYPP_MULTICURL and leap.
It's probably a little early to expect any useful results on that search. When I googled "ZYPP_MULTICURL", the first hit was http://lists.opensuse.org/opensuse/2015-11/msg00442.html :-) The 2nd hit weas about libzypp, as Andrei suggested. -- Per Jessen, Zürich (14.3°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Nov 18, 2015 at 8:28 AM, Per Jessen <per@computer.org> wrote:
Roger Oberholtzer wrote:
Perhaps bad form to reply to one's own post. I might add that google was no help with a search limited to ZYPP_MULTICURL and leap.
It's probably a little early to expect any useful results on that search. When I googled "ZYPP_MULTICURL", the first hit was
http://lists.opensuse.org/opensuse/2015-11/msg00442.html :-)
The 2nd hit weas about libzypp, as Andrei suggested.
I see that ZYPP_MULTICURL is still checked, and it seems that if it is true (not set to 0), the connection is made via MediaMultiCurl::MediaMultiCurl, which I would suspect is the same as it has been. The problem I am having is that I have installed Leap in a VirtualBox. I have done this before with other versions of openSUSE, and used zypper without issue. We always make one of these as the 'official' build environment of the various releases we support. When I try to install some packages that we want in this VBox, the status when downloading is crazy. It goes up and down in percent. In larger packages, it never to completes, and zypper fails. This looks like what I saw when we did not use ZYPP_MULTICURL=0. Our systems are behind a Cisco firewall (called something like ironhorse, if memory serves). It wants to download the entire item on it's own and check for virus before passing the bits to the client. When libzypp makes multiple connections, this causes problems. Earlier discussions here came to the conclusion that ZYPP_MULTICURL=0 could help. And, up until Leap it has. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Nov 18, 2015 at 11:17 AM, Roger Oberholtzer <roger.oberholtzer@gmail.com> wrote:
multiple connections, this causes problems. Earlier discussions here came to the conclusion that ZYPP_MULTICURL=0 could help. And, up until Leap it has.
If you want to check Leap version it is here: https://build.opensuse.org/package/show/openSUSE:Leap:42.1/libzypp -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Nov 18, 2015 at 9:21 AM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
On Wed, Nov 18, 2015 at 11:17 AM, Roger Oberholtzer <roger.oberholtzer@gmail.com> wrote:
multiple connections, this causes problems. Earlier discussions here came to the conclusion that ZYPP_MULTICURL=0 could help. And, up until Leap it has.
If you want to check Leap version it is here: https://build.opensuse.org/package/show/openSUSE:Leap:42.1/libzypp
That is the version I checked. And as I wrote, it looks like ZYPP_MULTICURL=0 is still accepted. So perhaps the issue is elsewhere. It is just that the behavior looked so similar to what I saw when not using ZYPP_MULTICURL=0 -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Roger Oberholtzer wrote:
On Wed, Nov 18, 2015 at 8:28 AM, Per Jessen <per@computer.org> wrote:
Roger Oberholtzer wrote:
Perhaps bad form to reply to one's own post. I might add that google was no help with a search limited to ZYPP_MULTICURL and leap.
It's probably a little early to expect any useful results on that search. When I googled "ZYPP_MULTICURL", the first hit was
http://lists.opensuse.org/opensuse/2015-11/msg00442.html :-)
The 2nd hit weas about libzypp, as Andrei suggested.
I see that ZYPP_MULTICURL is still checked, and it seems that if it is true (not set to 0), the connection is made via MediaMultiCurl::MediaMultiCurl, which I would suspect is the same as it has been.
The problem I am having is that I have installed Leap in a VirtualBox. I have done this before with other versions of openSUSE, and used zypper without issue. We always make one of these as the 'official' build environment of the various releases we support. When I try to install some packages that we want in this VBox, the status when downloading is crazy. It goes up and down in percent. In larger packages, it never to completes, and zypper fails. This looks like what I saw when we did not use ZYPP_MULTICURL=0. Our systems are behind a Cisco firewall (called something like ironhorse, if memory serves).
"Ironport" I suspect. It's not a firewall as such, it's an email and web security filter. I guess it could be getting in the way, although it can't possibly have an issue with multiple concurrent requests. Most browsers will issue multiple concurrent requests for every page. It's odd that zypper should behave differently depending on the environment, i.e. virtual or physical.
It wants to download the entire item on it's own and check for virus before passing the bits to the client. When libzypp makes multiple connections, this causes problems.
Wouldn't this also happen during a normal installation? (it sounds to me as if ZYPP_MULTICURL=0 cures the symptom, but not the problem). -- Per Jessen, Zürich (14.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Nov 18, 2015 at 9:31 AM, Per Jessen <per@computer.org> wrote:
Wouldn't this also happen during a normal installation? (it sounds to me as if ZYPP_MULTICURL=0 cures the symptom, but not the problem).
I did the install from an ISO image. I did not choose to do updates at that time because I knew I could not specify ZYPP_MULTICURL=0 to the installer. So I thought I would do the updates after install. It is big RPMs that seem to be most effected. For example, my update wants texlive-cm-super-fonts. It gets to +90% percent, and then immediately fails, retires to 90%, fails, etc. If I tell it to ignore the failed RPM it continues to download the others. But of course the install will fail because an RPM is missing... -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Op woensdag 18 november 2015 10:02:10 schreef Roger Oberholtzer:
On Wed, Nov 18, 2015 at 9:31 AM, Per Jessen <per@computer.org> wrote:
Wouldn't this also happen during a normal installation? (it sounds to me as if ZYPP_MULTICURL=0 cures the symptom, but not the problem).
I did the install from an ISO image. I did not choose to do updates at that time because I knew I could not specify ZYPP_MULTICURL=0 to the installer. So I thought I would do the updates after install.
It is big RPMs that seem to be most effected. For example, my update wants texlive-cm-super-fonts. It gets to +90% percent, and then immediately fails, retires to 90%, fails, etc. If I tell it to ignore the failed RPM it continues to download the others. But of course the install will fail because an RPM is missing...
I once had a similar problem in which the downloaded rpm many times failed the security check; also big rpms. In that case I found the location where the rpm should go, don't remember where, and searched for the URL of the rpm on the download.opensuse.org site. After that I used: curl -L <URL> -o <local_destination> to get the rpm, performed the security check, and started zypper again. Apparently zypper found the rpm was OK and continued with the rest. I may have needed to repeat the procedure with other rpms, but finally zypper could perform the update. -- fr.gr. member openSUSE Freek de Kruijf -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Roger Oberholtzer wrote:
On Wed, Nov 18, 2015 at 9:31 AM, Per Jessen <per@computer.org> wrote:
Wouldn't this also happen during a normal installation? (it sounds to me as if ZYPP_MULTICURL=0 cures the symptom, but not the problem).
I did the install from an ISO image. I did not choose to do updates at that time because I knew I could not specify ZYPP_MULTICURL=0 to the installer. So I thought I would do the updates after install.
Ah, I see - I am so unaccustomed to installing from physical media that I forget they exist :-(
It is big RPMs that seem to be most effected.
Big files are often broken into multiple downloads, one segment each. For instance, if an rpm is 16Mb, it could be downloaded as 16 x 1Mb segments from 16 different mirrors. It played merry hell with our squid cache until I figured out what was going on.
For example, my update wants texlive-cm-super-fonts. It gets to +90% percent, and then immediately fails, retires to 90%, fails, etc.
I'll have to check one of my machines, but others should be seeing this too. -- Per Jessen, Zürich (15.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Nov 18, 2015 at 11:30 AM, Per Jessen <per@computer.org> wrote:
Big files are often broken into multiple downloads, one segment each. For instance, if an rpm is 16Mb, it could be downloaded as 16 x 1Mb segments from 16 different mirrors. It played merry hell with our squid cache until I figured out what was going on.
But the Cisco Ironport firewall gets the full file on it's own and then repackages it for the client as needed. IIRC, the Ironport does not deal correctly with some HTTP directives correctly set by zypper/curl when this is happening. So it breaks things. zypper/curl are not the only victims of the Cisco Ironport bug. Which seems to remain unfixed. At least on the ones used by our company. -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Roger Oberholtzer wrote:
On Wed, Nov 18, 2015 at 11:30 AM, Per Jessen <per@computer.org> wrote:
Big files are often broken into multiple downloads, one segment each. For instance, if an rpm is 16Mb, it could be downloaded as 16 x 1Mb segments from 16 different mirrors. It played merry hell with our squid cache until I figured out what was going on.
But the Cisco Ironport firewall gets the full file on it's own and then repackages it for the client as needed.
I'm surprised Ironport can work that out, it's fairly complex. You're certain it does that? Ah, maybe it does indeed fetch the whole file first, but that would be ridiculous. If a 16Mb file is split into 16 segments, that could potentially cause 16 full downloads :-)
IIRC, the Ironport does not deal correctly with some HTTP directives correctly set by zypper/curl when this is happening. So it breaks things. zypper/curl are not the only victims of the Cisco Ironport bug.
Aha, that's interesting. zypper is certainly not the only one to use the segmented download, but the only other one I can think of right now is youtube. Anyway, yes, it sounds like the ZYPP_MULTICURL option is in fact the solution you need. -- Per Jessen, Zürich (15.8°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Nov 18, 2015 at 1:05 PM, Per Jessen <per@computer.org> wrote:
I'm surprised Ironport can work that out, it's fairly complex. You're certain it does that? Ah, maybe it does indeed fetch the whole file first, but that would be ridiculous. If a 16Mb file is split into 16 segments, that could potentially cause 16 full downloads :-)
Yep. The reason it does this is that it wants to scan the item for virus. I guess it needs the whole file for this. I recall something about information in the HTTP commands generated by zypper that indicate this is what is happening. I think that zypper was doing the right thing. It was the Cisco device that did not get this correct. Either in it's replies or in intercepting the HTTP commands used by libcurl. If memory serves...
Aha, that's interesting. zypper is certainly not the only one to use the segmented download, but the only other one I can think of right now is youtube.
Yep. Other clients had problems, which was how it was discovered to be an Ironport issue.
Anyway, yes, it sounds like the ZYPP_MULTICURL option is in fact the solution you need.
Except it seems not to be working well with big files, as it had been doing previously. Of course, maybe the IT guys here have changed something as well. They can be rather secretive about these things. Moving targets everywhere... -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
Roger Oberholtzer wrote:
On Wed, Nov 18, 2015 at 11:30 AM, Per Jessen <per@computer.org> wrote:
Big files are often broken into multiple downloads, one segment each. For instance, if an rpm is 16Mb, it could be downloaded as 16 x 1Mb segments from 16 different mirrors. It played merry hell with our squid cache until I figured out what was going on.
But the Cisco Ironport firewall gets the full file on it's own and then repackages it for the client as needed.
I'm surprised Ironport can work that out, it's fairly complex. You're certain it does that? Ah, maybe it does indeed fetch the whole file first, but that would be ridiculous. If a 16Mb file is split into 16 segments, that could potentially cause 16 full downloads :-)
IIRC, the Ironport does not deal correctly with some HTTP directives correctly set by zypper/curl when this is happening. So it breaks things. zypper/curl are not the only victims of the Cisco Ironport bug.
Aha, that's interesting. zypper is certainly not the only one to use the segmented download, but the only other one I can think of right now is youtube.
Anyway, yes, it sounds like the ZYPP_MULTICURL option is in fact the solution you need.
To test whether it works, if you can get to run tcpdump/wireshark, you should be able to see the http requests and the 206 responses. The 206 response is a clear indicator that segmenting is active. -- Per Jessen, Zürich (15.9°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, Nov 18, 2015 at 3:20 PM, Per Jessen <per@computer.org> wrote:
To test whether it works, if you can get to run tcpdump/wireshark, you should be able to see the http requests and the 206 responses.
Unless SSL is in use. But with intercepting appliance it would need to fake certificates ... so probably it is plain text. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (4)
-
Andrei Borzenkov -
Freek de Kruijf -
Per Jessen -
Roger Oberholtzer