[opensuse] net install via (transparent) proxy
I've just done two identical network installations of 11.4 after one another, and would have expected the second one to benefit from the fact that the first one had already retrieved everything required. However, according to squids logs, I had 61 hits and roughly 2500 misses. Not a single RPM was cached. I'm guessing the issue is the metalink, but I haven't found any way of making squid work with the metalink. Question: How do I get squid to cache the repo data during a network installation? -- Per Jessen, Zürich (10.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tue, May 03, 2011 at 01:09:37PM +0200, Per Jessen wrote:
I've just done two identical network installations of 11.4 after one another, and would have expected the second one to benefit from the fact that the first one had already retrieved everything required. However, according to squids logs, I had 61 hits and roughly 2500 misses. Not a single RPM was cached.
If you used download.openSUSE.org this might be the side effect of the download redirector (MirrorBrain). I would check the squid log for the same rpm (e.g. rpm-4.8.0-27.1). If MirrorBrain redirected to two different servers the http proxy can't know that the content is the same.
I'm guessing the issue is the metalink, but I haven't found any way of making squid work with the metalink.
Question: How do I get squid to cache the repo data during a network installation?
The only workaround I see at the moment is to use a particular mirror instead of download.openSUSE.org. Looks at all like a bug or better a missing feature to me. openFATE is eager to get your input. Lars -- Lars Müller [ˈlaː(r)z ˈmʏlɐ] Samba Team SUSE Linux, Maxfeldstraße 5, 90409 Nürnberg, Germany
Lars Müller wrote:
On Tue, May 03, 2011 at 01:09:37PM +0200, Per Jessen wrote:
I've just done two identical network installations of 11.4 after one another, and would have expected the second one to benefit from the fact that the first one had already retrieved everything required. However, according to squids logs, I had 61 hits and roughly 2500 misses. Not a single RPM was cached.
If you used download.openSUSE.org this might be the side effect of the download redirector (MirrorBrain).
I used the vanilla ISO as downloaded from http://download.opensuse.org/.
I would check the squid log for the same rpm (e.g. rpm-4.8.0-27.1).
If MirrorBrain redirected to two different servers the http proxy can't know that the content is the same.
It uses a few different mirrors - e.g. for kernel-default: ftp.halifax.rwth-aachen.de ftp.tu-chemnitz.de ftp.uni-bremen.de gd.tuwien.ac.at mirror.leaseweb.com mirror.ox.ac.uk mirror.switch.ch Other packages are equally widespread: ftp4.gwdg.de ftp.belnet.be ftp.cc.uoc.gr ftp.esat.net ftp.icm.edu.pl ftp.linux.ee ftp.uni-kl.de mirror.karneval.cz mirror.switch.ch
I'm guessing the issue is the metalink, but I haven't found any way of making squid work with the metalink.
Question: How do I get squid to cache the repo data during a network installation?
The only workaround I see at the moment is to use a particular mirror instead of download.openSUSE.org.
Yeah, that's easy, I could also just run my own local mirror, but the idea here is to make this happen transparently without any additional typing. Put in the USB or CD, boot and start the installation, and let squid handle the cacheing.
Looks at all like a bug or better a missing feature to me. openFATE is eager to get your input.
Sounds more like a bug to me - "we provide the proxy= command line option, but cacheing is disabled due to our infrastructure". Anyway, is this really mirrorbrain causing the problem? I'm sure I've seen Peter Pöml claim that mirrorbrain had no problems with squid. /Per -- Per Jessen, Zürich (13.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tue, May 03, 2011 at 02:47:17PM +0200, Per Jessen wrote:
Lars Müller wrote:
On Tue, May 03, 2011 at 01:09:37PM +0200, Per Jessen wrote:
I've just done two identical network installations of 11.4 after one another, and would have expected the second one to benefit from the fact that the first one had already retrieved everything required. However, according to squids logs, I had 61 hits and roughly 2500 misses. Not a single RPM was cached.
If you used download.openSUSE.org this might be the side effect of the download redirector (MirrorBrain).
I used the vanilla ISO as downloaded from http://download.opensuse.org/.
Then you only download the updates released after general availabilty of 11.4. And the big pieces are handled very well by patch RPMs. Therefore your download at all must be small if you only use the default oss, non-oss and update package repositories.
I would check the squid log for the same rpm (e.g. rpm-4.8.0-27.1).
If MirrorBrain redirected to two different servers the http proxy can't know that the content is the same.
It uses a few different mirrors - e.g. for kernel-default:
a) which version (e.g. 2.6.37.6-0.5.1)? b) full or delta rpm?
The only workaround I see at the moment is to use a particular mirror instead of download.openSUSE.org.
Yeah, that's easy, I could also just run my own local mirror, but the idea here is to make this happen transparently without any additional typing. Put in the USB or CD, boot and start the installation, and let squid handle the cacheing.
Looks at all like a bug or better a missing feature to me. openFATE is eager to get your input.
Sounds more like a bug to me - "we provide the proxy= command line option, but cacheing is disabled due to our infrastructure".
Anyway, is this really mirrorbrain causing the problem? I'm sure I've seen Peter Pöml claim that mirrorbrain had no problems with squid.
It's not MirrorBrain which is causing this. MirrorBrain redirects you each time to a different server. Try this: download_url="http://download.opensuse.org/pub/opensuse/update/11.4/rpm/x86_64/yast2-contr..." while [ 1 ]; do wget ${download_url} rm -v ${download_url##*/} sleep 2 done You'll see it gets redirected each time to a different server if your region is covered by multiple server. To check this follow the "Details" link for example from http://download.opensuse.org/pub/opensuse/update/11.4/rpm/x86_64/ at the right most column. Lars -- Lars Müller [ˈlaː(r)z ˈmʏlɐ] Samba Team SUSE Linux, Maxfeldstraße 5, 90409 Nürnberg, Germany
Lars Müller wrote:
On Tue, May 03, 2011 at 02:47:17PM +0200, Per Jessen wrote:
Lars Müller wrote:
On Tue, May 03, 2011 at 01:09:37PM +0200, Per Jessen wrote:
I've just done two identical network installations of 11.4 after one another, and would have expected the second one to benefit from the fact that the first one had already retrieved everything required. However, according to squids logs, I had 61 hits and roughly 2500 misses. Not a single RPM was cached.
If you used download.openSUSE.org this might be the side effect of the download redirector (MirrorBrain).
I used the vanilla ISO as downloaded from http://download.opensuse.org/.
Then you only download the updates released after general availabilty of 11.4. And the big pieces are handled very well by patch RPMs.
Therefore your download at all must be small if you only use the default oss, non-oss and update package repositories.
The NET iso only has about 100Mb. Everything else is downloaded.
I would check the squid log for the same rpm (e.g. rpm-4.8.0-27.1). If MirrorBrain redirected to two different servers the http proxy can't know that the content is the same.
It uses a few different mirrors - e.g. for kernel-default:
a) which version (e.g. 2.6.37.6-0.5.1)?
kernel-default-2.6.37.1-1.2.2.i586.rpm (GA 11.4).
b) full or delta rpm?
Full. But it's irrelevant, I can pick any other bigger package, it is still split across multiple different mirrors. (isn't that what metalink is good for?)
The only workaround I see at the moment is to use a particular mirror instead of download.openSUSE.org.
Yeah, that's easy, I could also just run my own local mirror, but the idea here is to make this happen transparently without any additional typing. Put in the USB or CD, boot and start the installation, and let squid handle the cacheing.
Looks at all like a bug or better a missing feature to me. openFATE is eager to get your input.
Sounds more like a bug to me - "we provide the proxy= command line option, but cacheing is disabled due to our infrastructure".
Anyway, is this really mirrorbrain causing the problem? I'm sure I've seen Peter Pöml claim that mirrorbrain had no problems with squid.
It's not MirrorBrain which is causing this.
Right, I thought so too. I guess I'd better open that bugreport. -- Per Jessen, Zürich (14.3°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Right, I thought so too. I guess I'd better open that bugreport.
https://bugzilla.novell.com/show_bug.cgi?id=691685 -- Per Jessen, Zürich (14.8°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
Per Jessen wrote:
Right, I thought so too. I guess I'd better open that bugreport.
I'm slightly surprised to nobody else has commented on this so far - I guess proxying is not overly popular amongst openSUSE users? (that it doesn't work might explain that of course :-) ) -- Per Jessen, Zürich (9.7°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
I'm slightly surprised to nobody else has commented on this so far - I guess proxying is not overly popular amongst openSUSE users? (that it doesn't work might explain that of course :-) )
I can think of two reasons that would prevent me noticing: (1) I haven't tried any 11.4 installs yet :) (2) I'm not the network admin guy who would notice that the proxy was bypassed, and he doesn't do linux installs so wouldn't notice either :( -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Dave Howorth wrote:
Per Jessen wrote:
I'm slightly surprised to nobody else has commented on this so far - I guess proxying is not overly popular amongst openSUSE users? (that it doesn't work might explain that of course :-) )
I can think of two reasons that would prevent me noticing:
(1) I haven't tried any 11.4 installs yet :)
Hehe, I have a suspicion the issue might date a bit further back.
(2) I'm not the network admin guy who would notice that the proxy was bypassed, and he doesn't do linux installs so wouldn't notice either :(
With your email-address, you probably have a really thick internet pipe, otherwise you would notice. Here in the office I have a 6Mbit downstream, so the difference between internet download and hitting the cache is quite significant. -- Per Jessen, Zürich (15.7°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Thu, May 05, 2011 at 12:27:38PM +0100, Dave Howorth wrote:
Per Jessen wrote:
I'm slightly surprised to nobody else has commented on this so far - I guess proxying is not overly popular amongst openSUSE users? (that it doesn't work might explain that of course :-) )
I can think of two reasons that would prevent me noticing:
(1) I haven't tried any 11.4 installs yet :)
(2) I'm not the network admin guy who would notice that the proxy was bypassed, and he doesn't do linux installs so wouldn't notice either :(
No proxy is bypassed. This "it's me again, please redirect me to the same download source again to benefit from my local proxy" is a new freature and has to be tracked by https://features.opensuse.org/ Lars -- Lars Müller [ˈlaː(r)z ˈmʏlɐ] Samba Team SUSE Linux, Maxfeldstraße 5, 90409 Nürnberg, Germany
Lars Müller wrote:
On Thu, May 05, 2011 at 12:27:38PM +0100, Dave Howorth wrote:
Per Jessen wrote:
I'm slightly surprised to nobody else has commented on this so far - I guess proxying is not overly popular amongst openSUSE users? (that it doesn't work might explain that of course :-) )
I can think of two reasons that would prevent me noticing:
(1) I haven't tried any 11.4 installs yet :)
(2) I'm not the network admin guy who would notice that the proxy was bypassed, and he doesn't do linux installs so wouldn't notice either :(
No proxy is bypassed.
Uh, mine is. 61 hits, 2500 misses. That equates to "bypassing the proxy" in my book.
This "it's me again, please redirect me to the same download source again to benefit from my local proxy" is a new freature and has to be tracked by https://features.opensuse.org/
Sorry, it's a bug - we have had the proxy option on the command line for years, and it has stopped working as intended, presumably after we changed the distribution infrastructure. Bug. https://bugzilla.novell.com/show_bug.cgi?id=691685 -- Per Jessen, Zürich (16.4°C) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
Per Jessen wrote:
With your email-address, you probably have a really thick internet pipe, otherwise you would notice. Here in the office I have a 6Mbit downstream, so the difference between internet download and hitting the cache is quite significant.
Good point. IIRC, we have at least 100 Mbps. :-P -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (3)
-
Dave Howorth -
Lars Müller -
Per Jessen