[heroes] Postmortem for mailing lists outage on 2018-10-30
Hello, on Monday 2018-10-29 around 18:00 UTC the mail delivery on the mailing lists started to be slow, resulting in total outage a couple of hours later. The issue identified to be on our vendor's DNS. The MX servers were unable to contact the openSUSE HAproxy, relay and mailing lists servers. We contacted our vendor via their ticketing system immediatelly as soon as we realized that this was the cause. While waiting for a response, we also thought of changing the configuration of our relay, MX and HAproxy servers in order to bypass the vendor's DNS, and instead use the openSUSE one only. Meanwhile the vendor fixed the issue, but while waiting for the change to propagate, we were able to verify that our configuration updates worked fine, which seems even better solution. Today Tuesday 2018-10-30 around 12:00 UTC everything was back to normal. Thorsten and Theo on behalf of the heroes team
participants (1)
-
Theo Chatzimichos