October 2018 - Heroes - openSUSE Mailing Lists

[heroes] openSUSE Forums issues
by Knurpht-openSUSE 19 Oct '18

19 Oct '18

Hi No report made before, since issues are inconsistent, change from time to time, are not the same for all users all the time. Please find below our issues and experiences Our experiences re. the openSUSE Forums: - lags in loading the forums pages - delays in replying after submit - "hangs" where the forums software is "Waiting for suse.com" - a "Too many redirects" page result - slow general performance compared to f.e. forums.suse.com The tough thing about reporting issues, is that they're not consistent. Some of the things we found: - on "hangs", clicking in the browser's address bar and hitting Enter does load the page ( workaround ) - Removing all involved cookies, i.e. opensuse, suse, microfocus, sometimes(!) seems to improve things for a while, but not consistent. Behaviour can change from one moment to the next. - from @malcolmlewis' research using the developer's Network tool in Firefox: "So looking at the timings between the SUSE and openSUSE forum, it's the openSUSE html code that's the issue IMHO. SUSE forum uses php to view at thread, around 600ms openSUSE uses html, around 2000ms 'waiting'.... This is on a HP dm4-3050us (intel core i5, 6GB ram, 750GB of rotating rust) openSUSE Leap 15 and Firefox 60.2.2 ESR. After my initial login there was a delay 'waiting' for around 11000ms for the first thread view...." - Kim Groneman suggested the culprit to be in the forums headers, since the forums.suse.com performs OK, and is on a same kind of machine. -- Gertjan Lettink a.k.a. Knurpht openSUSE Board Member openSUSE Forums Team -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 3

[heroes] Postmortem fow download.opensuse.org unplanned outage on 2018-10-03
by Theo Chatzimichos 11 Oct '18

11 Oct '18

Hello, on Tuesday 2018-10-02 around 22:00 UTC the download.opensuse.org mirrorbrain service became unavailable. Relevant ticket [1] Christian Boltz reacted almost immediatelly. As he reported on the heroes(a)opensuse.org mailing list [2], dmesg showed a filesystem corruption on the /srv partition on the machine that hosts download.opensuse.org, called pontifex2. In order to work around the issue, he started mirrorbrain on provo-mirror.opensuse.org and redirected all the traffic from pontifex2 to provo-mirror. This served well quite a big amount of the requests, but there were still a bunch of issues, both known and newly emerged: - the mirrorbrain database on provo-mirror was quite outdated - there was an outdated SSL certificate - there is no IPv6 on our opensuse infrastructure on Provo The next day 2018-10-03 Theo and Martin took over. The dmesg errors were agreeing with any attempt to access the /srv partition: pontifex2 (download.o.o):~ # ls /srv ls: cannot access '/srv': Input/output error Additionally, we realized that the disk was full, and after discussing the issue on #opensuse-admin on Freenode between various parties, we came to the conclusion that the full disk could even cause the IO errors. We couldn't find any indication though if this was an LVM, XFS or kernel issue. Even with the full disk issue, mirrorbrain should have still be operating and even self-heal. First step was to change the download.o.o and downloadcontent.o.o domains to point to provo-mirror immediatelly so that we can stop services on pontifex2. Then we stopped all the services that were trying to access /srv. Afterwards we increased the disk space to 2TB on the partition and also on the PV, but we gave only one TB extra to the LV, as a workaround for our abovementioned wild guess. We ran a filesystem check, that took some time as well but indicated no errors. After rebooting the machine, everything was back to normal. We switched the DNS records back and everything was back to normal around 13:00 UTC. Steps to be taken to avoid the issue in the future: - Try to reproduce the IO error on a similar setup using LVM, xfs and leap 42.3 or 15.x. Then it will be more clear why this weird IO error happens and how to fix it properly in production. - Figure out why the /srv went out of space so fast - Gather all the issues that prevented provo-mirror to act as a proper mirrorbrain replacement and fix them, in order to have a better failover for any future planned or unplanned outage of pontifex2. [1] https://progress.opensuse.org/issues/41933 [2] https://lists.opensuse.org/heroes/2018-10/msg00002.html Special thanks to: - Martin Caj - Christian Boltz - Per Jessen - Marcus Rueckert (for his precious consulting) Theo, on behalf of the openSUSE Heroes team

1 0

[heroes] Re: Löschung von Einträgen
by Per Jessen 05 Oct '18

05 Oct '18

Donald Duck wrote: > Sehr geehrte Damen und Herren, > > aus dem Jahre 2002 befinden sich noch Beiträge in der Mailingliste, die von > mir verfasst wurden. Ich möchte Sie darum bitten, die Beiträge zu löschen, > bzw. meinen Namen unkenntlich zu machen. > > Mit Freundlichen Grüßen > > Donald Duck Hi Board (I changed the name in the request above) I usually ignore these requests of "to be forgotten", but every now and then another one turns up. Not frequently, at most a couple of times a year. To put things in perspective, sofar I have not had anyone insist or repeat the request. In my role as the openSUSE mailing list admin, I could easily accomodate such requests, it's a bit of effort, that's all. I ignore these for the following reasons - a) fiddling with mailing list archives, no matter what was promised, is not to be done lightly. (in my opinion). b) I do not have any official/legal standing in our mostly-ethereal openSUSE organisation nor wrt SUSE. c) I am a citizen of the EU but I don't live in the EU, where applicable rules may or may not exist (IANAL, I simply do not know). Just for your information. I'm contemplating writing a wiki page to explain the situation, to give an easy reply to such requests. cheers Per -- Per Jessen, Zürich (25.2°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 2

[heroes] Reminder: Heroes meeting on Tuesday
by Christian Boltz 03 Oct '18

03 Oct '18

Hello, the next heroes meeting will be on Tuesday, 2018-10-02 18:00 UTC / 20:00 CEST in the #opensuse-admin IRC channel. I added the usual topics to https://progress.opensuse.org/issues/41825 - please add whatever you want to discuss. Regards, Christian Boltz -- > [SuSE vs. SUSE] A good question. Maybe the friend of ... had > a company which needed urgently some money? Sorry, I can't follow you there. Do you mean there's a company that sells capital U's? [> Thorsten Kukuk and Rasmus Plewe] -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 1

[heroes] B.O.O. broken
by Felix Miata 03 Oct '18

03 Oct '18

What happens when I try to search: Secure Connection Failed The document contains no data. The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. Please contact the website owners to inform them of this problem. My saved searches work, but all fresh searches fail. -- "Wisdom is supreme; therefore get wisdom. Whatever else you get, get wisdom." Proverbs 4:7 (New Living Translation) Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 1

[heroes] download.o.o down
by Christian Boltz 03 Oct '18

03 Oct '18

Hello, download.o.o is down, answers all requests with a 403 error, and dmesg indicates /srv has filesystem issues :-( See https://progress.opensuse.org/issues/41933 for details. I can imagine you now quickly checked and everything looks good. No, I'm not drunk, and I'm not inventing issues ;-) What you see is a workaround - I changed download.o.o to redirect everything to provo-mirror and started apache (instead of nginx) there. As a workaround, I changed the apache config so that *** everything gets redirected to provo-mirror *** I also stopped nginx and instead started apache on provo-mirror, which means the mirrorbrain setup there is active now Obviously the redirect to provo-mirror comes with some problems. The mirrorbrain database in Provo is outdated (I'd guess last update was the NBG power outage) - it doesn't know about Leap 15, which means provo- mirror will have to handle lots of load and traffic itsself. We'll see how good it will survive under the load. I know this solution is far from perfect, but it's still better than serving a 403 error for all of download.o.o. The original apache config is in /etc/apache2---before-workaround-2018-10-03 I copied it there before adding the redirect + doing some other changes needed to let apache load the updated config - for example, apache doesn't like a /srv/whatever DocumentRoot if /srv/ isn't available. After fixing the /srv partition, you can move the backed up apache config back to production, and everything should work again as if nothing had happened ;-) Regards, Christian Boltz -- These are the requirements for granted correct behavior. If you are not interested in that, I am happy to provide you with a list of data that could be removed from a general Suse installation to let Suse Linux behave similar and to permit booting from time to time only. [Jörg Schiling in https://bugzilla.novell.com/550021#c32] -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 1

[heroes] DETAIL: Could not write to file "pg_subtrans/166A" at offset 131072: No space left on device.
by Per Jessen 01 Oct '18

01 Oct '18

Looks like we've run out of space on mirrordb3 ? -- Per Jessen, Zürich (11.2°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 1