Mailinglist Archive: opensuse-buildservice (284 mails)

< Previous Next >
Re: [opensuse-buildservice] Redirector issues.
  • From: "Dr. Peter Poeml" <poeml@xxxxxxx>
  • Date: Tue, 23 Oct 2007 15:51:02 +0200
  • Message-id: <20071023134646.GY5856@xxxxxxx>
Hi Benji,

On Tue, Oct 09, 2007 at 02:10:10AM +0100, Benji Weber wrote:
Greetings All,

The redirector is causing problems again. Now with installation for many
users.

Some of the mirrors are broken or under heavy load, but the redirector
is still redirecting people to those mirrors. This coupled with the
fact that the openSUSE installer now selects "use online repositories"
by default (a good thing) means that many people are experiencing
failed installations.

The real problem is the mirror "sticky" that means people always get
the same mirror. This means that if the user gets a bad mirror and a
package download timeout then the installation has essentially failed.
If the user clicks retry he/she will get the same mirror, and a
timeout again, and again. There is approximately 1minute timeout
between attempts, and several hundred packages to install from online
sources by default. This means even if the user clicks "skip" for each
package it will take hours to complete the install. So the install has
effectively failed because of the redirector's mirror sticky.

Quite a number of users have commented on this behaviour, I have
experienced it myself with 2 installs already. It has also been
mentioned in some reviews, contributing to overall bad reviews of
10.3.


Thank you for the input.

The current status can be summarized like this:
- Most which can be done on the server side has been done.
- Now it is time to work on the client side and add robustness which can
be achieved only there -- on the client side.

Thus, there is work to be done in yast/libzypp, which the yast/libzypp
team had no time to even look at so far. Unfortunately, they have been
busy with working on real essential stuff, like installing a system, or
making sure they don't retrieve each file 5 times from the webserver...
they didn't get around to such more advanced, high-level functionality.
The situation is regrettable, but let's hope that libzypp stabilizes
somewhat, and they can start working on these issues.

Other distribution's solutions:

1: Round Robin DNS
+ No problem with redirector server going down
+ If one of the mirrors in rotation is broken clicking "retry" will
likely select a good mirror.
+ Location based mirror selection not possible but can have e.g.
eu.download.opensuse.org, us.download.opensuse.org ... etc

- Frequently changing repositories such as on the build service, could
bounce between new metadata & old packages and vice versa.


RRDNS is out of question, much too inflexible and it requires an
intelligent client anyway, which handles errors sensibly.
As RRDNS can't work based on type of requested file, it would only work
for "complete" mirrors anyway. All mirrors are not equal!

2: Mirror list files

Other distributions use mirror-list-files which the package manager
interprets.

+ No sticking-to-bad-mirror problem of the redirector.
+ Avoids the bouncing between mirrors problem of RR DNS.

- The server with the mirror list on needs to be up when it is requested.
- Cannot be achieved without modifying the client package management software.


Exactly.

What I would like you to understand is that there is no further
client robustness achievable without modifying the client. So, stop
bashing on "the redirector" and start bashing in the right direction ;-)


What I have been advocating so far looks like this:

- The means of providing a list of potential mirrors to try are there. It
is just a matter of implementing client support to make use of it.
(You could see this as a contemporary equivalent of the static mirror
lists which were used in the past.)
- Instead of just requesting a file, and crash upon failure, request
from the redirector a list of 3 or 5 mirrors which have the file. Try
the first mirror, if it fails, try the second one, and so on.
- Note that I want this to work transparently; _not_ requiring anyone
hitting a "retry" button.

Optionally,
- it could backlist mirrors which are broken or give a timeout.
- It could cache mirror base urls in case to try if the redirector is
unavailable.
- It could even use the mirror which turns out to be fastest, if it
wants.

This would bring libzypp on par with yum, and more.

In addition, interaction between the client (yast/libzypp) and the
download infrastructure is conceivable:
- yast/libzypp can report broken mirrors / missing files to the
redirector.
- without further effort, all other clients (yum, smart, ...) would
benefit from this, once yast/zypp users (which comprise the majority)
have caused an issue to be fixed.


From all what I have seen while working on the download infrastructure,
looking at different download infrastructures, using and debugging
different clients, with and without proxies, considering and discussing
different architectural takes, this is what seems to be the most
promising, powerful, scalable approach which makes yast/zypp robust to
result in a pleasurable user experience.

This concept needs to be refined and discussed, of course.
Input and further ideas welcome.

Sticky breaks installations, no sticky breaks general package
management - what is the solution? Can the redirector do better
checking on mirror availability and status? any thoughts? A solution
that can be achieved by only modifying d.o.o so that future 10.3
installations can go smoothly is obviously preferable.


Um, no, problems rather become worse, if mirrors are not sticky.
Suddenly everybody is affected from such a problem, while otherwise
it would hit only users in the country of the broken mirror (plus users
in countries without any mirror). Not something you want.

By the way, we have been running without mirror stickiness for many
weeks (you just didn't notice), and it yielded the expected results.

Thanks,
Peter
--
"WARNING: This bug is visible to non-employees. Please be respectful!"

SUSE LINUX Products GmbH
Research & Development
< Previous Next >
Follow Ups
References