[opensuse-buildservice] Redirector issues.
Greetings All, The redirector is causing problems again. Now with installation for many users. Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations. The real problem is the mirror "sticky" that means people always get the same mirror. This means that if the user gets a bad mirror and a package download timeout then the installation has essentially failed. If the user clicks retry he/she will get the same mirror, and a timeout again, and again. There is approximately 1minute timeout between attempts, and several hundred packages to install from online sources by default. This means even if the user clicks "skip" for each package it will take hours to complete the install. So the install has effectively failed because of the redirector's mirror sticky. Quite a number of users have commented on this behaviour, I have experienced it myself with 2 installs already. It has also been mentioned in some reviews, contributing to overall bad reviews of 10.3. Other distribution's solutions: 1: Round Robin DNS + No problem with redirector server going down + If one of the mirrors in rotation is broken clicking "retry" will likely select a good mirror. + Location based mirror selection not possible but can have e.g. eu.download.opensuse.org, us.download.opensuse.org ... etc - Frequently changing repositories such as on the build service, could bounce between new metadata & old packages and vice versa. 2: Mirror list files Other distributions use mirror-list-files which the package manager interprets. + No sticking-to-bad-mirror problem of the redirector. + Avoids the bouncing between mirrors problem of RR DNS. - The server with the mirror list on needs to be up when it is requested. - Cannot be achieved without modifying the client package management software. Sticky breaks installations, no sticky breaks general package management - what is the solution? Can the redirector do better checking on mirror availability and status? any thoughts? A solution that can be achieved by only modifying d.o.o so that future 10.3 installations can go smoothly is obviously preferable. _ Benjamin Weber --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2007-10-09 02:10:10 +0100, Benji Weber wrote:
The redirector is causing problems again. Now with installation for many users.
Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations.
The real problem is the mirror "sticky" that means people always get the same mirror. This means that if the user gets a bad mirror and a package download timeout then the installation has essentially failed. If the user clicks retry he/she will get the same mirror, and a timeout again, and again. There is approximately 1minute timeout between attempts, and several hundred packages to install from online sources by default. This means even if the user clicks "skip" for each package it will take hours to complete the install. So the install has effectively failed because of the redirector's mirror sticky.
Quite a number of users have commented on this behaviour, I have experienced it myself with 2 installs already. It has also been mentioned in some reviews, contributing to overall bad reviews of 10.3.
Other distribution's solutions:
1: Round Robin DNS + No problem with redirector server going down + If one of the mirrors in rotation is broken clicking "retry" will likely select a good mirror. + Location based mirror selection not possible but can have e.g. eu.download.opensuse.org, us.download.opensuse.org ... etc
- Frequently changing repositories such as on the build service, could bounce between new metadata & old packages and vice versa.
2: Mirror list files
Other distributions use mirror-list-files which the package manager interprets.
+ No sticking-to-bad-mirror problem of the redirector. + Avoids the bouncing between mirrors problem of RR DNS.
- The server with the mirror list on needs to be up when it is requested. - Cannot be achieved without modifying the client package management software.
Sticky breaks installations, no sticky breaks general package management - what is the solution? Can the redirector do better checking on mirror availability and status? any thoughts? A solution that can be achieved by only modifying d.o.o so that future 10.3 installations can go smoothly is obviously preferable.
just a few first thoughts. i will think more about it: 1. the frequency for those checks would be too high to avoid the problem. 2. my first approach would be some history in the redirector. if a client retries we know that the client requested the same file again and send it to a different mirror. the information could be stored in the same session structure as the sticky mirror. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Tuesday 09 October 2007 06:52:50 wrote Marcus Rueckert:
On 2007-10-09 02:10:10 +0100, Benji Weber wrote: ...
Sticky breaks installations, no sticky breaks general package management - what is the solution? Can the redirector do better checking on mirror availability and status? any thoughts? A solution that can be achieved by only modifying d.o.o so that future 10.3 installations can go smoothly is obviously preferable.
just a few first thoughts. i will think more about it:
1. the frequency for those checks would be too high to avoid the problem.
2. my first approach would be some history in the redirector. if a client retries we know that the client requested the same file again and send it to a different mirror.
the information could be stored in the same session structure as the sticky mirror.
2. sounds like a good idea ... let's hear a comment from Peter about this next week. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Tue, Oct 09, 2007 at 09:08:09AM +0200, Adrian Schröter wrote:
On Tuesday 09 October 2007 06:52:50 wrote Marcus Rueckert:
On 2007-10-09 02:10:10 +0100, Benji Weber wrote: ...
Sticky breaks installations, no sticky breaks general package management - what is the solution? Can the redirector do better checking on mirror availability and status? any thoughts? A solution that can be achieved by only modifying d.o.o so that future 10.3 installations can go smoothly is obviously preferable.
just a few first thoughts. i will think more about it:
1. the frequency for those checks would be too high to avoid the problem.
2. my first approach would be some history in the redirector. if a client retries we know that the client requested the same file again and send it to a different mirror.
the information could be stored in the same session structure as the sticky mirror.
2. sounds like a good idea ... let's hear a comment from Peter about this next week.
Interesting idea indeed. Let me think about it :-) Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Tuesday 09 October 2007 03:10:10 wrote Benji Weber:
Greetings All,
The redirector is causing problems again. Now with installation for many users.
Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations.
I would be happy, if you can name some ;)
The real problem is the mirror "sticky" that means people always get the same mirror. This means that if the user gets a bad mirror and a package download timeout then the installation has essentially failed. If the user clicks retry he/she will get the same mirror, and a timeout again, and again. There is approximately 1minute timeout between attempts, and several hundred packages to install from online sources by default. This means even if the user clicks "skip" for each package it will take hours to complete the install. So the install has effectively failed because of the redirector's mirror sticky.
Quite a number of users have commented on this behaviour, I have experienced it myself with 2 installs already. It has also been mentioned in some reviews, contributing to overall bad reviews of 10.3.
Do you you have URLs ?
Other distribution's solutions:
1: Round Robin DNS + No problem with redirector server going down + If one of the mirrors in rotation is broken clicking "retry" will likely select a good mirror. + Location based mirror selection not possible but can have e.g. eu.download.opensuse.org, us.download.opensuse.org ... etc
- Frequently changing repositories such as on the build service, could bounce between new metadata & old packages and vice versa.
- no check if a mirror is complete - no possibility to disable a mirror fast, when it goes down - no support for mirrors which provide only a subset of files (what are quite a lot)
2: Mirror list files
Other distributions use mirror-list-files which the package manager interprets.
+ No sticking-to-bad-mirror problem of the redirector.
well, this is more a client side task, a client can already request the list of mirrors which provide a certain file. So it would be thinkable, that libzypp checks this list and asks for another mirror if it got an error. This would have the advantage that we get also broken or unreliable mirrors reported from the clients and can do some actions like disabling or rating them down.
+ Avoids the bouncing between mirrors problem of RR DNS.
- The server with the mirror list on needs to be up when it is requested. - Cannot be achieved without modifying the client package management software.
yep :) Klaus, Duncan, any comments from your side here ?
Sticky breaks installations, no sticky breaks general package management - what is the solution? Can the redirector do better checking on mirror availability and status? any thoughts? A solution that can be achieved by only modifying d.o.o so that future 10.3 installations can go smoothly is obviously preferable.
well, no sticky means just that we would get unreproducable errors and random points. This is IMHO even worse, since it is hard to find out which mirror is actually broken. So in short, I see to options on the high level * Either we can guarantee for only non-broken mirrors (or at least for a reasonable level) * We can handle broken mirrors on the client side. But the discussion about sticky or not sticky is only moving the problem IMHO. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (cross-posting on purpose, see below) Adrian Schröter wrote:
On Tuesday 09 October 2007 03:10:10 wrote Benji Weber: [...]
Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations.
I would be happy, if you can name some ;)
[...]
Quite a number of users have commented on this behaviour, I have experienced it myself with 2 installs already. It has also been mentioned in some reviews, contributing to overall bad reviews of 10.3.
Do you you have URLs ?
Lots, seriously. Gut feeling is that currently, there are even more bad reviews about 10.3 than 10.2 (which is quite surprising, given that 10.3 definitely hasn't got the package management issues 10.2 had). I think we should collect URLs to bad reviews somewhere (on the wiki ?) and comment them there, to try to identify what the most annoying problems were -- and whether it's just motivated by the usual "anti-Novell-because-it's-in-bed-with-MS" FUD from so-called journalists or whether they actually have a point. To me, many of the reviews are motivated by the hatred against Novell and you can feel the tone right from the start, they're just looking for things to criticize, but even in those, they may have a point with 2 or 3 things. Shall we discuss this on -project ? (already answered, I'm cross-posting) Oh, and getting rid of compiz would probably remove 50% of the problems people have with openSUSE 10.3 but.. ok, let's keep that for -project :) cheers - -- -o) Pascal Bleser http://linux01.gwdg.de/~pbleser/ /\\ <pascal.bleser@skynet.be> <guru@unixtech.be> _\_v The more things change, the more they stay insane. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHCywwr3NMWliFcXcRAh9bAJ9w10oa41pbyQ9zwKCrLSLxKMVOagCfcqdE wTY7hngMyu2+SBIE/YvENDY= =GDwF -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
* Adrian Schröter <adrian@suse.de> [Oct 09. 2007 09:07]: [...]
This would have the advantage that we get also broken or unreliable mirrors reported from the clients and can do some actions like disabling or rating them down.
[...]
- Cannot be achieved without modifying the client package management software.
yep :)
Klaus, Duncan, any comments from your side here ?
We are very well aware of it and plan to fix it in the future. For 10.3, there is not much we can do :-( Klaus --- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Please make sure there is a bug (well, RFE ;-) in place for OpenSUSE 11. Best -F Klaus Kaempf wrote:
* Adrian Schröter <adrian@suse.de> [Oct 09. 2007 09:07]: [...]
This would have the advantage that we get also broken or unreliable mirrors reported from the clients and can do some actions like disabling or rating them down.
[...]
- Cannot be achieved without modifying the client package management software.
yep :)
Klaus, Duncan, any comments from your side here ?
We are very well aware of it and plan to fix it in the future. For 10.3, there is not much we can do :-(
Klaus
--- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg)
--------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-- _________________________________________ -- "'Problem' is a bleak word for challenge" - Richard Fish (Federico L. Lucifredi) - flucifredi@novell.com --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Benji Weber wrote:
The redirector is causing problems again. Now with installation for many users.
Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations. [...]
And as a side note: most US mirrors listed on the wiki [1] are incomplete again (we already had this when 10.2 came out): ftp://mirrors.kernel.org/opensuse/distribution/10.3/repo/oss/ ok, seems to have it now, was empty yesterday ftp://mirror.colorado.edu/pub/opensuse/distribution/10.3/repo/oss/ ok, was empty yesterday ftp://suse.mirrors.tds.net/pub/opensuse/distribution/10.3/repo/oss/ ok, but frequently errors out with timeouts, was empty yesterday ftp://linux.nssl.noaa.gov/opensuse/distribution/10.3/repo/oss/ empty ftp://mirror.nyi.net/opensuse/distribution/10.3/repo/oss/ ok http://opensuse.berkeley.edu/opensuse/distribution/10.3/repo/oss/ empty (404) [1]http://en.opensuse.org/Mirrors_Released_Version#USA I swear I did check yesterday evening because someone located in the US asked for help as he couldn't add the 10.3 oss repository using download.o.o, and mirror.nyi.net was the only one that actually had the files (original error was with tds.net, and I think I didn't check colorado.edu). Well, there are still 2 bad mirrors left, and yesterday it was at least 4 broken mirrors (unless I was too tired and did something stupid ^^). Today, 5 days after the official release, and more than a week after the mirrors could start syncing, this still means having a 33% chance of hitting a broken mirror (with the consequences explained in Benjamin's mail). We know how it goes when a release comes out: lots and lots of people jump on it, eagerly waiting to download and try it out. Which means that it is very important in terms of visibility and having good reviews (which isn't really the case for 10.3, but that's a somewhat broader topic as the redirector and broken mirrors aren't the only reasons) that the mirrors are available and fully synced as soon as the official release is published. Correct me if I'm wrong, but since 10.2 the sync-out to mirrors is much better prepared in conjunction with the mirror admins. It just doesn't seem to work for US mirrors. For 11.0, I think it would be better not to add the US mirrors in the redirector (if that's possible per subdir) at release time -- it's better to redirect users in the US to fast, reliable and fully synced servers in Europe (*.de, skynet, belnet) than broken ones in the US -- and then check the mirrors listed above and only add them when they're complete. Would something like that even be possible with the redirector (list of mirrors per subdir (/10.3/ or /11.0/) instead of global) ? cheers - -- -o) Pascal Bleser http://linux01.gwdg.de/~pbleser/ /\\ <pascal.bleser@skynet.be> <guru@unixtech.be> _\_v The more things change, the more they stay insane. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHCykrr3NMWliFcXcRAsPZAJ0SilzAAE9P4qIm1WBEnlGoowlKQQCdEUJ7 rMori4Mb20S6nL74JZ56qlA= =ghNs -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Tuesday 09 October 2007 09:09:31 wrote Pascal Bleser:
Benji Weber wrote:
The redirector is causing problems again. Now with installation for many users.
Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations.
[...]
And as a side note: most US mirrors listed on the wiki [1] are incomplete again (we already had this when 10.2 came out):
We may should remove most of them and point to http://download.opensuse.org/distribution/10.3/iso/cd/MD5SUMS?mirrorlist (does not work atm, need to check why) ... (I will walk along the mirrors you mentioned)
[1]http://en.opensuse.org/Mirrors_Released_Version#USA
I swear I did check yesterday evening because someone located in the US asked for help as he couldn't add the 10.3 oss repository using download.o.o, and mirror.nyi.net was the only one that actually had the files (original error was with tds.net, and I think I didn't check colorado.edu). Well, there are still 2 bad mirrors left, and yesterday it was at least 4 broken mirrors (unless I was too tired and did something stupid ^^). Today, 5 days after the official release, and more than a week after the mirrors could start syncing, this still means having a 33% chance of hitting a broken mirror (with the consequences explained in Benjamin's mail).
We know how it goes when a release comes out: lots and lots of people jump on it, eagerly waiting to download and try it out. Which means that it is very important in terms of visibility and having good reviews (which isn't really the case for 10.3, but that's a somewhat broader topic as the redirector and broken mirrors aren't the only reasons) that the mirrors are available and fully synced as soon as the official release is published.
Correct me if I'm wrong, but since 10.2 the sync-out to mirrors is much better prepared in conjunction with the mirror admins. It just doesn't seem to work for US mirrors.
For 11.0, I think it would be better not to add the US mirrors in the redirector (if that's possible per subdir) at release time -- it's better to redirect users in the US to fast, reliable and fully synced servers in Europe (*.de, skynet, belnet) than broken ones in the US -- and then check the mirrors listed above and only add them when they're complete.
Would something like that even be possible with the redirector (list of mirrors per subdir (/10.3/ or /11.0/) instead of global) ?
well, the redirector works already based on files. So, if it redirects to mirror, which does not provide the file one of this happened: * mirror was working but broke down recently. We permanantly ping the mirrors and disable them, if they do not answer to limit this problem. But this does not solve the problem, if suddenly the content gets broken. * We misconfigured something, for example we scan the mirror via rsync, but the content gets not delievered via http/ftp * content got removed/disabled on the mirror. This will discovered by the next scan, but in the meantime it is broken. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Tuesday 09 October 2007 09:28:19 wrote Adrian Schröter: ...
Would something like that even be possible with the redirector (list of mirrors per subdir (/10.3/ or /11.0/) instead of global) ?
well, the redirector works already based on files. So, if it redirects to mirror, which does not provide the file one of this happened:
* mirror was working but broke down recently. We permanantly ping the mirrors and disable them, if they do not answer to limit this problem. But this does not solve the problem, if suddenly the content gets broken.
* We misconfigured something, for example we scan the mirror via rsync, but the content gets not delievered via http/ftp
* content got removed/disabled on the mirror. This will discovered by the next scan, but in the meantime it is broken.
In general, we can have also a bug somewhere else (zypp or our redirector setup or ...) Please ask for y2log files when someone reports this and send them to us. Before we know more about the problem, we can not really do something reasonable. thanks adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Hi Benji, On Tue, Oct 09, 2007 at 02:10:10AM +0100, Benji Weber wrote:
Greetings All,
The redirector is causing problems again. Now with installation for many users.
Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations.
The real problem is the mirror "sticky" that means people always get the same mirror. This means that if the user gets a bad mirror and a package download timeout then the installation has essentially failed. If the user clicks retry he/she will get the same mirror, and a timeout again, and again. There is approximately 1minute timeout between attempts, and several hundred packages to install from online sources by default. This means even if the user clicks "skip" for each package it will take hours to complete the install. So the install has effectively failed because of the redirector's mirror sticky.
Quite a number of users have commented on this behaviour, I have experienced it myself with 2 installs already. It has also been mentioned in some reviews, contributing to overall bad reviews of 10.3.
Thank you for the input. The current status can be summarized like this: - Most which can be done on the server side has been done. - Now it is time to work on the client side and add robustness which can be achieved only there -- on the client side. Thus, there is work to be done in yast/libzypp, which the yast/libzypp team had no time to even look at so far. Unfortunately, they have been busy with working on real essential stuff, like installing a system, or making sure they don't retrieve each file 5 times from the webserver... they didn't get around to such more advanced, high-level functionality. The situation is regrettable, but let's hope that libzypp stabilizes somewhat, and they can start working on these issues.
Other distribution's solutions:
1: Round Robin DNS + No problem with redirector server going down + If one of the mirrors in rotation is broken clicking "retry" will likely select a good mirror. + Location based mirror selection not possible but can have e.g. eu.download.opensuse.org, us.download.opensuse.org ... etc
- Frequently changing repositories such as on the build service, could bounce between new metadata & old packages and vice versa.
RRDNS is out of question, much too inflexible and it requires an intelligent client anyway, which handles errors sensibly. As RRDNS can't work based on type of requested file, it would only work for "complete" mirrors anyway. All mirrors are not equal!
2: Mirror list files
Other distributions use mirror-list-files which the package manager interprets.
+ No sticking-to-bad-mirror problem of the redirector. + Avoids the bouncing between mirrors problem of RR DNS.
- The server with the mirror list on needs to be up when it is requested. - Cannot be achieved without modifying the client package management software.
Exactly. What I would like you to understand is that there is no further client robustness achievable without modifying the client. So, stop bashing on "the redirector" and start bashing in the right direction ;-) What I have been advocating so far looks like this: - The means of providing a list of potential mirrors to try are there. It is just a matter of implementing client support to make use of it. (You could see this as a contemporary equivalent of the static mirror lists which were used in the past.) - Instead of just requesting a file, and crash upon failure, request from the redirector a list of 3 or 5 mirrors which have the file. Try the first mirror, if it fails, try the second one, and so on. - Note that I want this to work transparently; _not_ requiring anyone hitting a "retry" button. Optionally, - it could backlist mirrors which are broken or give a timeout. - It could cache mirror base urls in case to try if the redirector is unavailable. - It could even use the mirror which turns out to be fastest, if it wants. This would bring libzypp on par with yum, and more. In addition, interaction between the client (yast/libzypp) and the download infrastructure is conceivable: - yast/libzypp can report broken mirrors / missing files to the redirector. - without further effort, all other clients (yum, smart, ...) would benefit from this, once yast/zypp users (which comprise the majority) have caused an issue to be fixed. From all what I have seen while working on the download infrastructure, looking at different download infrastructures, using and debugging different clients, with and without proxies, considering and discussing different architectural takes, this is what seems to be the most promising, powerful, scalable approach which makes yast/zypp robust to result in a pleasurable user experience. This concept needs to be refined and discussed, of course. Input and further ideas welcome.
Sticky breaks installations, no sticky breaks general package management - what is the solution? Can the redirector do better checking on mirror availability and status? any thoughts? A solution that can be achieved by only modifying d.o.o so that future 10.3 installations can go smoothly is obviously preferable.
Um, no, problems rather become worse, if mirrors are not sticky. Suddenly everybody is affected from such a problem, while otherwise it would hit only users in the country of the broken mirror (plus users in countries without any mirror). Not something you want. By the way, we have been running without mirror stickiness for many weeks (you just didn't notice), and it yielded the expected results. Thanks, Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
* Dr. Peter Poeml <poeml@suse.de> [Oct 23. 2007 15:51]:
Hi Benji,
On Tue, Oct 09, 2007 at 02:10:10AM +0100, Benji Weber wrote: [...]
The real problem is the mirror "sticky" that means people always get the same mirror.
[...]
Thank you for the input.
The current status can be summarized like this: - Most which can be done on the server side has been done.
Please help me understand where the 'stickyness' comes from. Is the redirector pointing to a mirror once or issuing a http redirect for every request ? Klaus --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On 2007-10-25 16:56:02 +0200, Klaus Kaempf wrote:
* Dr. Peter Poeml <poeml@suse.de> [Oct 23. 2007 15:51]:
Hi Benji,
On Tue, Oct 09, 2007 at 02:10:10AM +0100, Benji Weber wrote: [...]
The real problem is the mirror "sticky" that means people always get the same mirror.
[...]
Thank you for the input.
The current status can be summarized like this: - Most which can be done on the server side has been done.
Please help me understand where the 'stickyness' comes from.
Is the redirector pointing to a mirror once or issuing a http redirect for every request ?
stickiness means that once we have choosen a mirror for you. all further redirects will use the same mirror. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Oct 25, 2007 at 04:56:02PM +0200, Klaus Kaempf wrote:
Please help me understand where the 'stickyness' comes from.
In this context it means: Once a client has redirected to a certain mirror, it is redirected to the same mirror again on the next request, and not to another randomly chosen one. Obviously, the server checks for each request if the mirror actually has the file, because otherwise it would send the client into the void. Also, not all files can be trusted on the mirrors, some are not redirected at all. The association mirror id <-> client ip as memorized by the server) times out after a certain time of inactivity (10 minutes I think).
Is the redirector pointing to a mirror once or issuing a http redirect for every request ?
For every request. Thus working on a file basis. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Thu, Oct 25, 2007 at 05:11:17PM +0200, Dr. Peter Poeml wrote:
Once a client has redirected to a certain mirror, it is redirected to the same mirror again on the next request, and not to another randomly chosen one.
Obviously, the server checks for each request if the mirror actually has the file, because otherwise it would send the client into the void. c
Also, not all files can be trusted on the mirrors, some are not redirected at all.
The association mirror id <-> client ip as memorized by the server) times out after a certain time of inactivity (10 minutes I think).
Is the redirector pointing to a mirror once or issuing a http redirect for every request ?
For every request. Thus working on a file basis.
Peter
* Dr. Peter Poeml <poeml@suse.de> [Oct 25. 2007 17:11]:
On Thu, Oct 25, 2007 at 04:56:02PM +0200, Klaus Kaempf wrote:
Please help me understand where the 'stickyness' comes from.
Thanks for the explanation !
In this context it means:
Once a client has redirected to a certain mirror, it is redirected to the same mirror again on the next request, and not to another randomly chosen one.
Am I right to rephrase "it is redirected to the same mirror again on the next request" as "the redirector is introducing the stickyness" ? [...]
Is the redirector pointing to a mirror once or issuing a http redirect for every request ?
For every request. Thus working on a file basis.
Understood, thanks ! Klaus --- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Thu, Oct 25, 2007 at 05:39:12PM +0200, Klaus Kaempf wrote:
* Dr. Peter Poeml <poeml@suse.de> [Oct 25. 2007 17:11]:
On Thu, Oct 25, 2007 at 04:56:02PM +0200, Klaus Kaempf wrote:
Please help me understand where the 'stickyness' comes from.
Thanks for the explanation !
I have added the text to http://en.opensuse.org/Build_Service/Redirector also.
In this context it means:
Once a client has redirected to a certain mirror, it is redirected to the same mirror again on the next request, and not to another randomly chosen one.
Am I right to rephrase "it is redirected to the same mirror again on the next request" as "the redirector is introducing the stickyness" ?
Yes, and it is optional behaviour, which can be switched off in the Apache module. --Which is what I meant with "we have been running with and without stickiness in the past". Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
* Dr. Peter Poeml <poeml@suse.de> [Oct 25. 2007 17:45]:
On Thu, Oct 25, 2007 at 05:39:12PM +0200, Klaus Kaempf wrote:
Am I right to rephrase "it is redirected to the same mirror again on the next request" as "the redirector is introducing the stickyness" ?
Yes, and it is optional behaviour, which can be switched off in the Apache module. --Which is what I meant with "we have been running with and without stickiness in the past".
So at least the "retry button doesn't switch mirror" problem mentioned by Benji could be solved in the Redirector until the client tools catch up. Klaus --- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
On Fri, Oct 26, 2007 at 09:25:19AM +0200, Klaus Kaempf wrote:
* Dr. Peter Poeml <poeml@suse.de> [Oct 25. 2007 17:45]:
On Thu, Oct 25, 2007 at 05:39:12PM +0200, Klaus Kaempf wrote:
Am I right to rephrase "it is redirected to the same mirror again on the next request" as "the redirector is introducing the stickyness" ?
Yes, and it is optional behaviour, which can be switched off in the Apache module. --Which is what I meant with "we have been running with and without stickiness in the past".
So at least the "retry button doesn't switch mirror" problem mentioned by Benji could be solved in the Redirector until the client tools catch up.
That doesn't solve anything. It only alleviates the problem, on the cost that it hits much more people (like nearly everybody). Not a good trade. And as a side effect, it makes it nearly impossible to track down problems with mirrors. Thus, it increases the damage. No, it is important to find out about broken mirrors as soon as possible. I have been through it: I tracked down a "broken mirror" during the time we ran *without* stickiness. A mirror had a funny firewall which partly swallowed some server responses. It took *weeks* until I had tracked it down. At first, there were only sporadic reports that "something didn't work", and you had no idea where to start looking. Fortunately, it happened to myself sometimes, because it was a German mirror. Note that a broken mirror would be chosen irregardless of stickiness, randomly. And the fewer mirrors a country has, the more often that happens. Stickiness helps to track down problems and to fix them. ... Of course, not all users even *think* of notifying us in case they encounter problems with a mirror. Instead, they might just keep asking around "how they can force another one". Still, if we get a report, or somehow find out about it by other means, we can do something about it pretty quickly. What we also need, is more active monitoring of mirrors, which includes issuing real requests to files. Just a few days ago, I have seen for the second time a mirror where 80% of requests where handled normally, while 20% were truncated. That could be easily detected with a script checking mirrors for correct and reliable replies by means of test requests. It would help if clients show how they follow through redirects, because that could make it apparent to users which mirror is causing trouble. Right now, it requires debugging capabilities to find out which server the request went to. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
On Fri, 26 Oct 2007, Dr. Peter Poeml wrote:
So at least the "retry button doesn't switch mirror" problem mentioned by Benji could be solved in the Redirector until the client tools catch up.
That doesn't solve anything. It only alleviates the problem, on the cost that it hits much more people (like nearly everybody). Not a good trade. And as a side effect, it makes it nearly impossible to track down problems with mirrors. Thus, it increases the damage. No, it is important to find out about broken mirrors as soon as possible.
[...]
From the debuggers point of view this is correct, but not from users point of view. The users does not want to debug. He wants, that it works.
Theidea already proposed by someone else would help here: Use stickiness, but switch to another mirror, when same file gets requested would fit both needs. The server could make a note in a log file somwhere, which helps debugging and the user has a working solution. No client support is needed. If a script then finds to many such entries for one server, it sends a mail to the mirror-operator, or whatever you call him/her :-) Ciao -- http://www.dstoecker.eu/ (PGP key available) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Dirk Stoecker wrote:
Theidea already proposed by someone else would help here:
Use stickiness, but switch to another mirror, when same file gets requested would fit both needs. The server could make a note in a log file somwhere, which helps debugging and the user has a working solution. No client support is needed.
Unfortunately libzypp requests some files multiple times during it's normal operation, see https://bugzilla.novell.com/show_bug.cgi?id=307098. So this idea will work for *.rpm, but not for all of the metadata files. Michal --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org
Hi, for the record, here is another idea which is related in this context. This one is more of an optimization, than avoiding breakage: Add an affinity to a local mirror. Both a server-side approach (automatic) and a client-side approach (with user intervention) is conceivable. https://bugzilla.novell.com/show_bug.cgi?id=328850#c8 https://bugzilla.novell.com/show_bug.cgi?id=328850#c9 I would be interested in discussion about this! Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development
participants (9)
-
Adrian Schröter -
Benji Weber -
Dirk Stoecker -
Dr. Peter Poeml -
Federico Lucifredi -
Klaus Kaempf -
Marcus Rueckert -
Michal Marek -
Pascal Bleser