Hi Benji, On Tue, Oct 09, 2007 at 02:10:10AM +0100, Benji Weber wrote:
Greetings All,
The redirector is causing problems again. Now with installation for many users.
Some of the mirrors are broken or under heavy load, but the redirector is still redirecting people to those mirrors. This coupled with the fact that the openSUSE installer now selects "use online repositories" by default (a good thing) means that many people are experiencing failed installations.
The real problem is the mirror "sticky" that means people always get the same mirror. This means that if the user gets a bad mirror and a package download timeout then the installation has essentially failed. If the user clicks retry he/she will get the same mirror, and a timeout again, and again. There is approximately 1minute timeout between attempts, and several hundred packages to install from online sources by default. This means even if the user clicks "skip" for each package it will take hours to complete the install. So the install has effectively failed because of the redirector's mirror sticky.
Quite a number of users have commented on this behaviour, I have experienced it myself with 2 installs already. It has also been mentioned in some reviews, contributing to overall bad reviews of 10.3.
Thank you for the input. The current status can be summarized like this: - Most which can be done on the server side has been done. - Now it is time to work on the client side and add robustness which can be achieved only there -- on the client side. Thus, there is work to be done in yast/libzypp, which the yast/libzypp team had no time to even look at so far. Unfortunately, they have been busy with working on real essential stuff, like installing a system, or making sure they don't retrieve each file 5 times from the webserver... they didn't get around to such more advanced, high-level functionality. The situation is regrettable, but let's hope that libzypp stabilizes somewhat, and they can start working on these issues.
Other distribution's solutions:
1: Round Robin DNS + No problem with redirector server going down + If one of the mirrors in rotation is broken clicking "retry" will likely select a good mirror. + Location based mirror selection not possible but can have e.g. eu.download.opensuse.org, us.download.opensuse.org ... etc
- Frequently changing repositories such as on the build service, could bounce between new metadata & old packages and vice versa.
RRDNS is out of question, much too inflexible and it requires an intelligent client anyway, which handles errors sensibly. As RRDNS can't work based on type of requested file, it would only work for "complete" mirrors anyway. All mirrors are not equal!
2: Mirror list files
Other distributions use mirror-list-files which the package manager interprets.
+ No sticking-to-bad-mirror problem of the redirector. + Avoids the bouncing between mirrors problem of RR DNS.
- The server with the mirror list on needs to be up when it is requested. - Cannot be achieved without modifying the client package management software.
Exactly. What I would like you to understand is that there is no further client robustness achievable without modifying the client. So, stop bashing on "the redirector" and start bashing in the right direction ;-) What I have been advocating so far looks like this: - The means of providing a list of potential mirrors to try are there. It is just a matter of implementing client support to make use of it. (You could see this as a contemporary equivalent of the static mirror lists which were used in the past.) - Instead of just requesting a file, and crash upon failure, request from the redirector a list of 3 or 5 mirrors which have the file. Try the first mirror, if it fails, try the second one, and so on. - Note that I want this to work transparently; _not_ requiring anyone hitting a "retry" button. Optionally, - it could backlist mirrors which are broken or give a timeout. - It could cache mirror base urls in case to try if the redirector is unavailable. - It could even use the mirror which turns out to be fastest, if it wants. This would bring libzypp on par with yum, and more. In addition, interaction between the client (yast/libzypp) and the download infrastructure is conceivable: - yast/libzypp can report broken mirrors / missing files to the redirector. - without further effort, all other clients (yum, smart, ...) would benefit from this, once yast/zypp users (which comprise the majority) have caused an issue to be fixed. From all what I have seen while working on the download infrastructure, looking at different download infrastructures, using and debugging different clients, with and without proxies, considering and discussing different architectural takes, this is what seems to be the most promising, powerful, scalable approach which makes yast/zypp robust to result in a pleasurable user experience. This concept needs to be refined and discussed, of course. Input and further ideas welcome.
Sticky breaks installations, no sticky breaks general package management - what is the solution? Can the redirector do better checking on mirror availability and status? any thoughts? A solution that can be achieved by only modifying d.o.o so that future 10.3 installations can go smoothly is obviously preferable.
Um, no, problems rather become worse, if mirrors are not sticky. Suddenly everybody is affected from such a problem, while otherwise it would hit only users in the country of the broken mirror (plus users in countries without any mirror). Not something you want. By the way, we have been running without mirror stickiness for many weeks (you just didn't notice), and it yielded the expected results. Thanks, Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development