On Tuesday 09 October 2007 03:10:10 wrote Benji Weber:

Greetings All,

The redirector is causing problems again. Now with installation for many users.

Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations.

I would be happy, if you can name some ;)

The real problem is the mirror "sticky" that means people always get the same mirror. This means that if the user gets a bad mirror and a package download timeout then the installation has essentially failed. If the user clicks retry he/she will get the same mirror, and a timeout again, and again. There is approximately 1minute timeout between attempts, and several hundred packages to install from online sources by default. This means even if the user clicks "skip" for each package it will take hours to complete the install. So the install has effectively failed because of the redirector's mirror sticky.

Quite a number of users have commented on this behaviour, I have experienced it myself with 2 installs already. It has also been mentioned in some reviews, contributing to overall bad reviews of 10.3.

Do you you have URLs ?

Other distribution's solutions:

1: Round Robin DNS + No problem with redirector server going down + If one of the mirrors in rotation is broken clicking "retry" will likely select a good mirror. + Location based mirror selection not possible but can have e.g. eu.download.opensuse.org, us.download.opensuse.org ... etc

- Frequently changing repositories such as on the build service, could bounce between new metadata & old packages and vice versa.

- no check if a mirror is complete - no possibility to disable a mirror fast, when it goes down - no support for mirrors which provide only a subset of files (what are quite a lot)

2: Mirror list files

Other distributions use mirror-list-files which the package manager interprets.

+ No sticking-to-bad-mirror problem of the redirector.

well, this is more a client side task, a client can already request the list of mirrors which provide a certain file. So it would be thinkable, that libzypp checks this list and asks for another mirror if it got an error. This would have the advantage that we get also broken or unreliable mirrors reported from the clients and can do some actions like disabling or rating them down.

+ Avoids the bouncing between mirrors problem of RR DNS.

- The server with the mirror list on needs to be up when it is requested. - Cannot be achieved without modifying the client package management software.

yep :) Klaus, Duncan, any comments from your side here ?

Sticky breaks installations, no sticky breaks general package management - what is the solution? Can the redirector do better checking on mirror availability and status? any thoughts? A solution that can be achieved by only modifying d.o.o so that future 10.3 installations can go smoothly is obviously preferable.

well, no sticky means just that we would get unreproducable errors and random points. This is IMHO even worse, since it is hard to find out which mirror is actually broken. So in short, I see to options on the high level * Either we can guarantee for only non-broken mirrors (or at least for a reasonable level) * We can handle broken mirrors on the client side. But the discussion about sticky or not sticky is only moving the problem IMHO. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Benji Weber wrote:

The redirector is causing problems again. Now with installation for many users.

Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations. [...]

And as a side note: most US mirrors listed on the wiki [1] are incomplete again (we already had this when 10.2 came out): ftp://mirrors.kernel.org/opensuse/distribution/10.3/repo/oss/ ok, seems to have it now, was empty yesterday ftp://mirror.colorado.edu/pub/opensuse/distribution/10.3/repo/oss/ ok, was empty yesterday ftp://suse.mirrors.tds.net/pub/opensuse/distribution/10.3/repo/oss/ ok, but frequently errors out with timeouts, was empty yesterday ftp://linux.nssl.noaa.gov/opensuse/distribution/10.3/repo/oss/ empty ftp://mirror.nyi.net/opensuse/distribution/10.3/repo/oss/ ok http://opensuse.berkeley.edu/opensuse/distribution/10.3/repo/oss/ empty (404) [1]http://en.opensuse.org/Mirrors_Released_Version#USA I swear I did check yesterday evening because someone located in the US asked for help as he couldn't add the 10.3 oss repository using download.o.o, and mirror.nyi.net was the only one that actually had the files (original error was with tds.net, and I think I didn't check colorado.edu). Well, there are still 2 bad mirrors left, and yesterday it was at least 4 broken mirrors (unless I was too tired and did something stupid ^^). Today, 5 days after the official release, and more than a week after the mirrors could start syncing, this still means having a 33% chance of hitting a broken mirror (with the consequences explained in Benjamin's mail). We know how it goes when a release comes out: lots and lots of people jump on it, eagerly waiting to download and try it out. Which means that it is very important in terms of visibility and having good reviews (which isn't really the case for 10.3, but that's a somewhat broader topic as the redirector and broken mirrors aren't the only reasons) that the mirrors are available and fully synced as soon as the official release is published. Correct me if I'm wrong, but since 10.2 the sync-out to mirrors is much better prepared in conjunction with the mirror admins. It just doesn't seem to work for US mirrors. For 11.0, I think it would be better not to add the US mirrors in the redirector (if that's possible per subdir) at release time -- it's better to redirect users in the US to fast, reliable and fully synced servers in Europe (*.de, skynet, belnet) than broken ones in the US -- and then check the mirrors listed above and only add them when they're complete. Would something like that even be possible with the redirector (list of mirrors per subdir (/10.3/ or /11.0/) instead of global) ? cheers - -- -o) Pascal Bleser http://linux01.gwdg.de/~pbleser/ /\\ _\_v The more things change, the more they stay insane. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iD8DBQFHCykrr3NMWliFcXcRAsPZAJ0SilzAAE9P4qIm1WBEnlGoowlKQQCdEUJ7 rMori4Mb20S6nL74JZ56qlA= =ghNs -----END PGP SIGNATURE----- --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

On Tuesday 09 October 2007 09:09:31 wrote Pascal Bleser:

Benji Weber wrote:

...

The redirector is causing problems again. Now with installation for many users.

Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations.

[...]

And as a side note: most US mirrors listed on the wiki [1] are incomplete again (we already had this when 10.2 came out):

We may should remove most of them and point to http://download.opensuse.org/distribution/10.3/iso/cd/MD5SUMS?mirrorlist (does not work atm, need to check why) ... (I will walk along the mirrors you mentioned)

[1]http://en.opensuse.org/Mirrors_Released_Version#USA

I swear I did check yesterday evening because someone located in the US asked for help as he couldn't add the 10.3 oss repository using download.o.o, and mirror.nyi.net was the only one that actually had the files (original error was with tds.net, and I think I didn't check colorado.edu). Well, there are still 2 bad mirrors left, and yesterday it was at least 4 broken mirrors (unless I was too tired and did something stupid ^^). Today, 5 days after the official release, and more than a week after the mirrors could start syncing, this still means having a 33% chance of hitting a broken mirror (with the consequences explained in Benjamin's mail).

We know how it goes when a release comes out: lots and lots of people jump on it, eagerly waiting to download and try it out. Which means that it is very important in terms of visibility and having good reviews (which isn't really the case for 10.3, but that's a somewhat broader topic as the redirector and broken mirrors aren't the only reasons) that the mirrors are available and fully synced as soon as the official release is published.

Correct me if I'm wrong, but since 10.2 the sync-out to mirrors is much better prepared in conjunction with the mirror admins. It just doesn't seem to work for US mirrors.

For 11.0, I think it would be better not to add the US mirrors in the redirector (if that's possible per subdir) at release time -- it's better to redirect users in the US to fast, reliable and fully synced servers in Europe (*.de, skynet, belnet) than broken ones in the US -- and then check the mirrors listed above and only add them when they're complete.

Would something like that even be possible with the redirector (list of mirrors per subdir (/10.3/ or /11.0/) instead of global) ?

well, the redirector works already based on files. So, if it redirects to mirror, which does not provide the file one of this happened: * mirror was working but broke down recently. We permanantly ping the mirrors and disable them, if they do not answer to limit this problem. But this does not solve the problem, if suddenly the content gets broken. * We misconfigured something, for example we scan the mirror via rsync, but the content gets not delievered via http/ftp * content got removed/disabled on the mirror. This will discovered by the next scan, but in the meantime it is broken. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

* Adrian Schröter [Oct 09. 2007 09:07]: [...]

This would have the advantage that we get also broken or unreliable mirrors reported from the clients and can do some actions like disabling or rating them down.

[...]

...

- Cannot be achieved without modifying the client package management software.

yep :)

Klaus, Duncan, any comments from your side here ?

We are very well aware of it and plan to fix it in the future. For 10.3, there is not much we can do :-( Klaus --- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Hi Benji, On Tue, Oct 09, 2007 at 02:10:10AM +0100, Benji Weber wrote:

Greetings All,

The redirector is causing problems again. Now with installation for many users.

Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations.

The real problem is the mirror "sticky" that means people always get the same mirror. This means that if the user gets a bad mirror and a package download timeout then the installation has essentially failed. If the user clicks retry he/she will get the same mirror, and a timeout again, and again. There is approximately 1minute timeout between attempts, and several hundred packages to install from online sources by default. This means even if the user clicks "skip" for each package it will take hours to complete the install. So the install has effectively failed because of the redirector's mirror sticky.

Quite a number of users have commented on this behaviour, I have experienced it myself with 2 installs already. It has also been mentioned in some reviews, contributing to overall bad reviews of 10.3.

Thank you for the input. The current status can be summarized like this: - Most which can be done on the server side has been done. - Now it is time to work on the client side and add robustness which can be achieved only there -- on the client side. Thus, there is work to be done in yast/libzypp, which the yast/libzypp team had no time to even look at so far. Unfortunately, they have been busy with working on real essential stuff, like installing a system, or making sure they don't retrieve each file 5 times from the webserver... they didn't get around to such more advanced, high-level functionality. The situation is regrettable, but let's hope that libzypp stabilizes somewhat, and they can start working on these issues.

Other distribution's solutions:

1: Round Robin DNS + No problem with redirector server going down + If one of the mirrors in rotation is broken clicking "retry" will likely select a good mirror. + Location based mirror selection not possible but can have e.g. eu.download.opensuse.org, us.download.opensuse.org ... etc

- Frequently changing repositories such as on the build service, could bounce between new metadata & old packages and vice versa.

RRDNS is out of question, much too inflexible and it requires an intelligent client anyway, which handles errors sensibly. As RRDNS can't work based on type of requested file, it would only work for "complete" mirrors anyway. All mirrors are not equal!

2: Mirror list files

Other distributions use mirror-list-files which the package manager interprets.

+ No sticking-to-bad-mirror problem of the redirector. + Avoids the bouncing between mirrors problem of RR DNS.

- The server with the mirror list on needs to be up when it is requested. - Cannot be achieved without modifying the client package management software.

Exactly. What I would like you to understand is that there is no further client robustness achievable without modifying the client. So, stop bashing on "the redirector" and start bashing in the right direction ;-) What I have been advocating so far looks like this: - The means of providing a list of potential mirrors to try are there. It is just a matter of implementing client support to make use of it. (You could see this as a contemporary equivalent of the static mirror lists which were used in the past.) - Instead of just requesting a file, and crash upon failure, request from the redirector a list of 3 or 5 mirrors which have the file. Try the first mirror, if it fails, try the second one, and so on. - Note that I want this to work transparently; _not_ requiring anyone hitting a "retry" button. Optionally, - it could backlist mirrors which are broken or give a timeout. - It could cache mirror base urls in case to try if the redirector is unavailable. - It could even use the mirror which turns out to be fastest, if it wants. This would bring libzypp on par with yum, and more. In addition, interaction between the client (yast/libzypp) and the download infrastructure is conceivable: - yast/libzypp can report broken mirrors / missing files to the redirector. - without further effort, all other clients (yum, smart, ...) would benefit from this, once yast/zypp users (which comprise the majority) have caused an issue to be fixed. From all what I have seen while working on the download infrastructure, looking at different download infrastructures, using and debugging different clients, with and without proxies, considering and discussing different architectural takes, this is what seems to be the most promising, powerful, scalable approach which makes yast/zypp robust to result in a pleasurable user experience. This concept needs to be refined and discussed, of course. Input and further ideas welcome.

Sticky breaks installations, no sticky breaks general package management - what is the solution? Can the redirector do better checking on mirror availability and status? any thoughts? A solution that can be achieved by only modifying d.o.o so that future 10.3 installations can go smoothly is obviously preferable.

Um, no, problems rather become worse, if mirrors are not sticky. Suddenly everybody is affected from such a problem, while otherwise it would hit only users in the country of the broken mirror (plus users in countries without any mirror). Not something you want. By the way, we have been running without mirror stickiness for many weeks (you just didn't notice), and it yielded the expected results. Thanks, Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development

* Dr. Peter Poeml [Oct 23. 2007 15:51]:

Hi Benji,

On Tue, Oct 09, 2007 at 02:10:10AM +0100, Benji Weber wrote: [...]

...

The real problem is the mirror "sticky" that means people always get the same mirror.

[...]

Thank you for the input.

The current status can be summarized like this: - Most which can be done on the server side has been done.

Please help me understand where the 'stickyness' comes from. Is the redirector pointing to a mirror once or issuing a http redirect for every request ? Klaus --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

On Thu, Oct 25, 2007 at 04:56:02PM +0200, Klaus Kaempf wrote:

Please help me understand where the 'stickyness' comes from.

In this context it means: Once a client has redirected to a certain mirror, it is redirected to the same mirror again on the next request, and not to another randomly chosen one. Obviously, the server checks for each request if the mirror actually has the file, because otherwise it would send the client into the void. Also, not all files can be trusted on the mirrors, some are not redirected at all. The association mirror id <-> client ip as memorized by the server) times out after a certain time of inactivity (10 minutes I think).

Is the redirector pointing to a mirror once or issuing a http redirect for every request ?

For every request. Thus working on a file basis. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development

* Dr. Peter Poeml [Oct 25. 2007 17:11]:

On Thu, Oct 25, 2007 at 04:56:02PM +0200, Klaus Kaempf wrote:

...

Please help me understand where the 'stickyness' comes from.

Thanks for the explanation !

In this context it means:

Once a client has redirected to a certain mirror, it is redirected to the same mirror again on the next request, and not to another randomly chosen one.

Am I right to rephrase "it is redirected to the same mirror again on the next request" as "the redirector is introducing the stickyness" ? [...]

...

Is the redirector pointing to a mirror once or issuing a http redirect for every request ?

For every request. Thus working on a file basis.

Understood, thanks ! Klaus --- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

* Dr. Peter Poeml [Oct 25. 2007 17:45]:

On Thu, Oct 25, 2007 at 05:39:12PM +0200, Klaus Kaempf wrote:

...

Am I right to rephrase "it is redirected to the same mirror again on the next request" as "the redirector is introducing the stickyness" ?

Yes, and it is optional behaviour, which can be switched off in the Apache module. --Which is what I meant with "we have been running with and without stickiness in the past".

So at least the "retry button doesn't switch mirror" problem mentioned by Benji could be solved in the Redirector until the client tools catch up. Klaus --- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

On Fri, 26 Oct 2007, Dr. Peter Poeml wrote:

...

So at least the "retry button doesn't switch mirror" problem mentioned by Benji could be solved in the Redirector until the client tools catch up.

That doesn't solve anything. It only alleviates the problem, on the cost that it hits much more people (like nearly everybody). Not a good trade. And as a side effect, it makes it nearly impossible to track down problems with mirrors. Thus, it increases the damage. No, it is important to find out about broken mirrors as soon as possible.

[...]

From the debuggers point of view this is correct, but not from users point of view. The users does not want to debug. He wants, that it works.

Theidea already proposed by someone else would help here: Use stickiness, but switch to another mirror, when same file gets requested would fit both needs. The server could make a note in a log file somwhere, which helps debugging and the user has a working solution. No client support is needed. If a script then finds to many such entries for one server, it sends a mail to the mirror-operator, or whatever you call him/her :-) Ciao -- http://www.dstoecker.eu/ (PGP key available) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Hi, for the record, here is another idea which is related in this context. This one is more of an optimization, than avoiding breakage: Add an affinity to a local mirror. Both a server-side approach (automatic) and a client-side approach (with user intervention) is conceivable. https://bugzilla.novell.com/show_bug.cgi?id=328850#c8 https://bugzilla.novell.com/show_bug.cgi?id=328850#c9 I would be interested in discussion about this! Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development