[opensuse-buildservice] Post-mortem: Downtime on March 23, 2018
Good evening, as some of you might noticed, we had a downtime of ~25 minutes last Friday. For details what happened exactly, please have a look at our post mortem post: http://openbuildservice.org/2018/03/23/post-mortem-6/ Your Open Build Service Tea -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Monday 2018-03-26 17:59, Christian Bruckmayer wrote:
as some of you might noticed, we had a downtime of ~25 minutes last Friday. For details what happened exactly, please have a look at our post mortem post: http://openbuildservice.org/2018/03/23/post-mortem-6/
Certainly an entertaining read. Do you also have some notes on the failure of the notification system over the weekend, or is that, in fact, connected to that deployment problem?
Your Open Build Service Tea
This typo makes me want to suddenly have that kind of branded merchandise. It would be the next generation after the famous openSUSE beer. ;-) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Hi Jan, thanks for the feedback. The full crash of the application also caused a crash of the delayed jobs workers, which we use for asynchronous jobs. This was not discovered until Monday morning which caused e.g. that the notification system was not working over the weekend and delayed job needed to process several thousand jobs. We added the information to the postmortem report. http://openbuildservice.org/2018/03/23/post-mortem-6/ Christian On 03/26/2018 06:07 PM, Jan Engelhardt wrote:
On Monday 2018-03-26 17:59, Christian Bruckmayer wrote:
as some of you might noticed, we had a downtime of ~25 minutes last Friday. For details what happened exactly, please have a look at our post mortem post: http://openbuildservice.org/2018/03/23/post-mortem-6/ Certainly an entertaining read.
Do you also have some notes on the failure of the notification system over the weekend, or is that, in fact, connected to that deployment problem?
Your Open Build Service Tea This typo makes me want to suddenly have that kind of branded merchandise. It would be the next generation after the famous openSUSE beer. ;-)
-- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (2)
-
Christian Bruckmayer -
Jan Engelhardt