Hello, Am Sonntag, 7. Januar 2018, 19:06:26 CET schrieb Per Jessen:
Christian Boltz wrote:
This also means the only things I can do are updating status.o.o (already done) and waiting for someone who fixes the galera cluster and writes the next postmortem.
Earlier today I received the status notifications "Progress - The Project management tool status changed from Major Outage to Operational."
I never received any saying "changed from Operational to Major Outage" ?
Because of the database outage, connect.o.o, progress.o.o and events.o.o were down. While this doesn't sound too bad, the connect.o.o downtime also broke the @opensuse.org mail aliases :-( You are subscribed to status notifications with your opensuse.org mail address, and therefore... [log shortened and slightly edited to avoid feeding spambots] 2018-01-05T20:49:26.390029+00:00 status2 ... dsn=5.1.1, status=bounced (host mx2.suse.de[195.135.220.15] said: 550 5.1.1 <per@o....org>: Recipient address rejected: User unknown in virtual alias table (in reply to RCPT TO command)) The member mail aliases work again since yesterday evening (thanks to Theo for bringing the database master up again!), but we'll obviously have to improve the setup to make it more reliable and errorproof. Oh, and today I noticed that handling status.o.o via haproxy and distributing the load between status1 and status2 is a bad idea. Both use their own standalone MySQL database and the load balancing therefore resulted in a brain split. Fixed now by making status.o.o a CNAME to status1.o.o instead of using haproxy. Regards, Christian Boltz -- Yeah, my wife made serious trouble when I told her that I can't go to vacation because Kyrill needs his flash. And sorry, but she won that argument. [Stephan Kulow in opensuse-factory] -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org