[opensuse-buildservice] Redirector availability

older
[opensuse-buildservice] Permission...

Benji Weber

15 Sep 2007 15 Sep '07

10:43

Greetings all, So the redirector has gone down /again/ this weekend. And of course we get all the users coming asking why their package management isn't working while there are no suse people around to fix it. Having a high availability redirector is probably the single most important thing the project needs right now. It goes down far too often, and with our current strategy of adding repositories from the community repositories module, installation, and YMPs using the redirector this means that *every openSUSE user* is without package management on a regular basis. This inconveniences people and damages the reputation of the distribution amongst users. it also means that users cannot download the distribution at all as there is no static fallback for download.opensuse.org. The fact that it always goes down at weekends is a problem too, as most people have jobs this is when "hobbyist" people are likely to be wanting to try things out etc. Given how much adding repositories is automated in 10.3 we can no longer rely on users distributing themselves amongst mirrors. Therefore I think we either need to: - Have a co-hosted redirector, with monitoring to ensure uptime in place for 10.3 or very soon after. or - Have the build service YMP & installation/community repositories suggestions use alternating mirrors instead of the redirector. Whilst this would be suboptimal as we would lose tracking of statistics etc it would be better than the situation currently where everyone loses package management regularly. _ Benjamin Weber --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Show replies by date

Marcus Rueckert

15 Sep 15 Sep

18:38

On 2007-09-15 11:43:46 +0100, Benji Weber wrote:

...

So the redirector has gone down /again/ this weekend.

wrong. the redirector is still working: http://download.opensuse.org/repositories/ many people still have http://software.opensuse.org in their repository list. that host is broken. yes. (it looks like hardware problems)

...

And of course we get all the users coming asking why their package management isn't working while there are no suse people around to fix it.

Having a high availability redirector is probably the single most important thing the project needs right now. It goes down far too often, and with our current strategy of adding repositories from the community repositories module, installation, and YMPs using the redirector this means that *every openSUSE user* is without package management on a regular basis.

This inconveniences people and damages the reputation of the distribution amongst users. it also means that users cannot download the distribution at all as there is no static fallback for download.opensuse.org.

The fact that it always goes down at weekends is a problem too, as most people have jobs this is when "hobbyist" people are likely to be wanting to try things out etc.

Given how much adding repositories is automated in 10.3 we can no longer rely on users distributing themselves amongst mirrors. Therefore I think we either need to:

- Have a co-hosted redirector, with monitoring to ensure uptime in place for 10.3 or very soon after.

we are working on a HA setup for download.o.o already. but so far the host is holding up pretty good. peter is doing a good job to optimize and improve the redirector.

...

- Have the build service YMP & installation/community repositories suggestions use alternating mirrors instead of the redirector. Whilst this would be suboptimal as we would lose tracking of statistics etc it would be better than the situation currently where everyone loses package management regularly.

the ymps should be using download.opensuse.org already. if not that would be a critical bug. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Adrian Schröter

16 Sep 16 Sep

07:42

On Saturday 15 September 2007 12:43:46 wrote Benji Weber:

...

Greetings all,

So the redirector has gone down /again/ this weekend.

hm, it is working right now, and I do not find a signal that it was broken. Keep in mind that http://download.opensuse.org/ without a path redirects to software.o.o (which was indeed broken, seems a new suddenly appear hardware or kernel issue). Where there really problems with getting packages/repositories from download.o.o ? ...

...

Having a high availability redirector is probably the single most important thing the project needs right now. It goes down far too often, and with our current strategy of adding repositories from the community repositories module, installation, and YMPs using the redirector this means that *every openSUSE user* is without package management on a regular basis.

Unfortunatly, our planned HA solution for download.o.o will not be in place when 10.3 gets released. But it is on our list of most urgently needed things. So it will still happen this year, I am sure. bye adrian -- Adrian Schroeter SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) email: adrian@suse.de --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Marcus Rueckert

16:39

On 2007-09-16 09:42:08 +0200, Adrian Schröter wrote:

...

Where there really problems with getting packages/repositories from download.o.o ?

...

...
Having a high availability redirector is probably the single most important thing the project needs right now. It goes down far too often, and with our current strategy of adding repositories from the community repositories module, installation, and YMPs using the redirector this means that *every openSUSE user* is without package management on a regular basis.

Unfortunatly, our planned HA solution for download.o.o will not be in place when 10.3 gets released. But it is on our list of most urgently needed things. So it will still happen this year, I am sure.

good. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Dr. Peter Poeml

17 Sep 17 Sep

09:10

On Sat, Sep 15, 2007 at 11:43:46AM +0100, Benji Weber wrote:

...

Greetings all,

So the redirector has gone down /again/ this weekend.

No, the redirector had nothing to do with the failure of software.opensuse.org. The latter is a different host (since a while), providing the search interface stuff. (software.o.o was previously served by the redirector.) The redirector worked just fine over the weekend.

...

And of course we get all the users coming asking why their package management isn't working while there are no suse people around to fix it.

I agree that this situation is regrettable. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development

Benji Weber

15:34

On 17/09/2007, Dr. Peter Poeml wrote:

...

No, the redirector had nothing to do with the failure of software.opensuse.org. The latter is a different host (since a while), providing the search interface stuff. (software.o.o was previously served by the redirector.)

The redirector worked just fine over the weekend.

Yes, so it would seem. although a large number of people were still using software.opensuse.org that I believe used to be a redirector prior to the deployment of the new web interface. - I'm not sure this change was communicated clearly enough, as we need to always use download.opensuse.org now in that case. The points regarding necessary availability still stand even though it was in fact up though. :) _ Benjamin Weber --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Dr. Peter Poeml

18 Sep 18 Sep

09:53

On Mon, Sep 17, 2007 at 04:34:17PM +0100, Benji Weber wrote:

...

On 17/09/2007, Dr. Peter Poeml wrote:

...
No, the redirector had nothing to do with the failure of software.opensuse.org. The latter is a different host (since a while), providing the search interface stuff. (software.o.o was previously served by the redirector.)

The redirector worked just fine over the weekend.

Yes, so it would seem. although a large number of people were still using software.opensuse.org that I believe used to be a redirector prior to the deployment of the new web interface. - I'm not sure this change was communicated clearly enough, as we need to always use download.opensuse.org now in that case.

The points regarding necessary availability still stand even though it was in fact up though. :)

Sure. There are some interdependencies, which we should review carefully, and minimize where possible. * download.opensuse.org redirects (since recently) to the web frontend on software.opensuse.org for requests to '/'. Requests to '/*' go to the downlaod redirector. As far as availability of e.g. the update repository and other repositories is concerned, it doesn't depend on software.o.o. The redirector depends on a mirror database server, but can live without it for a certain period, until the network connection is sufficiently congested so it becomes to slow to respond. The latter effect can be mitigated somewhat by fallback mirrors to redirect to in the error case (to be implemented). Did we experience any availability problem since the current redirector is in place (about March)? I can't remember of any outage of the "over the weekend" kind... I can assure you that I'm against all infrastructure changes which add more dependencies than necessary :-) * software.opensuse.org relies on the availability of the internal buildservice backend, for the search and ymp generation. Thus, when the buildservice hangs, the software.opensuse.org search hangs as well. This is unfortunate IMO, but I don't know if there is a way around this. I don't know if YaST/zypper depend on the availability of software.opensuse.org or the search on software.opensuse.org. If so, how? For ymp generation? * software.opensuse.org/repositories/ redirects straight to the redirector. This kind of redirection is what is (should) only be happening when this obsolete URLs are used. Obviously, it can only work if software.o.o is responding at all. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development

Michael Schroeder

10:02

On Tue, Sep 18, 2007 at 11:53:14AM +0200, Dr. Peter Poeml wrote:

...

* software.opensuse.org relies on the availability of the internal buildservice backend, for the search and ymp generation. Thus, when the buildservice hangs, the software.opensuse.org search hangs as well. This is unfortunate IMO, but I don't know if there is a way around this.

Just to clarify things: it relies on the availablility of the API server (where we had some out of memory trouble) and the repository server (rock solid so far). It doesn't rely on the scheduler where things aren't so stable due to lots of code changes. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX Products GmbH, GF Markus Rex, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Martin Mohring

10:10

Michael Schroeder wrote:

...

On Tue, Sep 18, 2007 at 11:53:14AM +0200, Dr. Peter Poeml wrote:

...
* software.opensuse.org relies on the availability of the internal buildservice backend, for the search and ymp generation. Thus, when the buildservice hangs, the software.opensuse.org search hangs as well. This is unfortunate IMO, but I don't know if there is a way around this.

Just to clarify things: it relies on the availablility of the API server (where we had some out of memory trouble) and the repository server (rock solid so far). It doesn't rely on the scheduler where things aren't so stable due to lots of code changes.

Cheers, Michael.

Just for information: the scheduler is in "buildservice/src/backend", correct? Martin --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Marcus Rueckert

11:29

On 2007-09-18 12:10:15 +0200, Martin Mohring wrote:

...

Just for information: the scheduler is in "buildservice/src/backend", correct?

yes darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Martin Mohring

11:46

Marcus Rueckert wrote:

...

On 2007-09-18 12:10:15 +0200, Martin Mohring wrote:

...
Just for information: the scheduler is in "buildservice/src/backend", correct?

yes

darix

Do you have a branch in the Subversion repository for the deployment of OBS on .opensuse.org? I tried to understand, which changes in the scheduler you are testing (seems not to be in the Subversion repo). Are these changes deployed on OBS/.opensuse.org? Martin --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Klaas Freitag

19 Sep 19 Sep

18:53

Am Dienstag, 18. September 2007 13:46 schrieb Martin Mohring:

> Do you have a branch in the Subversion repository for the deployment of 
> OBS on .opensuse.org?
No, not yet. A proper release process of the buildservice code with for example
- tagging and opening a branch in svn
- proper release numbers
- documentation etc.
is one of our next targets, namely "Improve the development infrastructure".

Klaas
> 

-- 
Klaas Freitag                              Architect OPS/IPD
                        SUSE LINUX Products GmbH - Nuernberg
---------------------------------------------------------------------
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org
For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Cornelius Schumacher

21 Sep 21 Sep

09:43

On Tuesday 18 September 2007 13:46, Martin Mohring wrote:

...

Marcus Rueckert wrote:

...
On 2007-09-18 12:10:15 +0200, Martin Mohring wrote:

...
Just for information: the scheduler is in "buildservice/src/backend", correct?

yes

darix

Do you have a branch in the Subversion repository for the deployment of OBS on .opensuse.org?

There are branches for the deployment of frontend and web interface at https://forgesvn1.novell.com/svn/opensuse/branches/deploy. They unfortunately don't include the backend, though. -- Cornelius Schumacher --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Klaas Freitag

19 Sep 19 Sep

19:21

Am Dienstag, 18. September 2007 12:02 schrieb Michael Schroeder:

...

On Tue, Sep 18, 2007 at 11:53:14AM +0200, Dr. Peter Poeml wrote:

...
* software.opensuse.org relies on the availability of the internal buildservice backend, for the search and ymp generation. Thus, when the buildservice hangs, the software.opensuse.org search hangs as well. This is unfortunate IMO, but I don't know if there is a way around this.

Just to clarify things: it relies on the availablility of the API server (where we had some out of memory trouble) and the repository server (rock solid so far). It doesn't rely on the scheduler where things aren't so stable due to lots of code changes. If we would change that to a search_server somehow that combines search relevant data and functionality from api and repo server (and src server for build.o.o relevant searches which again calls the repo server for the work) - wouldn't that make sense?

At least searches for build.o.o have to search in the API database and the backend AFAIK which is not optimal and the repo server shouldn't be stressed by searches anyway. Anything against thinking in that direction? Did I miss something? Have fun, Klaas

...

-- Klaas Freitag Architect OPS/IPD SUSE LINUX Products GmbH - Nuernberg --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Dirk Stoecker

18 Sep 18 Sep

12:10

On Tue, 18 Sep 2007, Dr. Peter Poeml wrote:

...

* download.opensuse.org redirects (since recently) to the web frontend on software.opensuse.org for requests to '/'. Requests to '/*' go to the downlaod redirector. As far as availability of e.g. the update repository and other repositories is concerned, it doesn't depend on software.o.o.

The redirector depends on a mirror database server, but can live without it for a certain period, until the network connection is sufficiently congested so it becomes to slow to respond. The latter effect can be mitigated somewhat by fallback mirrors to redirect to in the error case (to be implemented).

Did we experience any availability problem since the current redirector is in place (about March)? I can't remember of any outage of the "over the weekend" kind...

I can assure you that I'm against all infrastructure changes which add more dependencies than necessary :-)

Well, this setup has one problem I experienced. For new packages the installation may fail, as meta files are up-to-date (from suse server directly), but the download is redirected to ftp5.gwdg.de (for me) and gwdg was not yet up-to-date. This can happy anytime, for everybody and every repository. Can the redirector be dynamic and introduce a little grace interval (15-30 minutes I think) for new packages? In the grace interval the redirector should redirect to SUSE download server for packages also instead of mirror. Ciao -- http://www.dstoecker.eu/ (PGP key available) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

Dr. Peter Poeml

19 Sep 19 Sep

08:17

Hi Dirk, On Tue, Sep 18, 2007 at 02:10:42PM +0200, Dirk Stoecker wrote:

...

Well, this setup has one problem I experienced.

For new packages the installation may fail, as meta files are up-to-date (from suse server directly), but the download is redirected to ftp5.gwdg.de (for me) and gwdg was not yet up-to-date.

This is a bug which I also observed with ftp5.gwdg.de, recently. We worked on a fix during last week, and it should be 99.9% fixed since Monday morning. A _pulling_ rsync from the site has been stepping on the toes of our rsync _push_, sometimes, and let freshly synced files disappear. The two rsyncs have been synchronized somewhat now, in coordination with the site's admin. It could principally happen with other mirrors which get the repositories pushed, but I don't have evidence that any other of those mirrors pulls via rsync, let alone every four hours. Anyway, I observed the problem only with ftp5.gwdg.de. FYI, except than in this buggy scenario, the redirector does only redirect to a mirror for files which it has successfully probed on that mirror before.

...

This can happy anytime, for everybody and every repository.

Yes, true. Peter -- "WARNING: This bug is visible to non-employees. Please be respectful!" SUSE LINUX Products GmbH Research & Development

Dirk Stoecker

20 Sep 20 Sep

06:21

On Wed, 19 Sep 2007, Dr. Peter Poeml wrote:

...

FYI, except than in this buggy scenario, the redirector does only redirect to a mirror for files which it has successfully probed on that mirror before.

Ah, that is the other alternative I would have suggested :-) Glad to hear it is a fixable bug and no design issue. Ciao -- http://www.dstoecker.eu/ (PGP key available) --------------------------------------------------------------------- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-buildservice+help@opensuse.org

6062

Age (days ago)

6068

Last active (days ago)

List overview

Download

16 comments

9 participants

participants (9)

Adrian Schröter
Benji Weber
Cornelius Schumacher
Dirk Stoecker
Dr. Peter Poeml
Klaas Freitag
Marcus Rueckert
Martin Mohring
Michael Schroeder