- Heroes

[heroes] Status update: fake iron machines
by Christian Boltz 27 Jan '20

27 Jan '20

Hello, You might have seen https://progress.opensuse.org/issues/62204 which is about VMs that appear as "physical" instead of "kvm" to salt, and therefore break applying the highstate. I had some fun (really!) debugging this yesterday - starting with reading the salt code that does the hardware detection to reading virt-what (used by salt if it's installed) and debugging it. If you are looking for an entertaining story how a small error was able to cause such a serious problem, read the ticket yourself (ideally while being logged in so that you also see the private comments). If you only want to hear about the actual reason and solution, scroll down - I'll add some whitespace (to prevent a spoiler) and then add what the actual fix was. However, reading the ticket is much more funny ;-) Regards, Christian Boltz PS: Scroll down for the end of the story ;-) scroll... scroll... scroll... scroll... Ok, enough scrolled ;-) In the end, it turned out that virt-what uses which virt-what-cpuid-helper to find one of its helper scripts. Unfortunately the virt-what package doesn't have "Requires: which" (only "Requires: util-linux", where which lived until 2013) - and without "which" installed, it's not a surprise that which $whatever fails ;-) The obvious workaround is to install "which". I added "which" to the package list in pillar/common.sls so that it gets installed on all 15.x machines, and opened a bugreport so that we don't need the same workaround in future releases. In case someone wonders why this only affected some VMs and not all with Leap 15.1: the condition to trigger this bug is that virt-what is installed and that which is not installed. If virt-what is not installed, salt uses systemd-detect-virt instead, which works without problems. -- >> MCSE: "Microsoft Certified Stupidity enclosed" [A. Spengler in dasr] > Ich dachte das heißt: > MCSE - Must Call Somebody Else [Markus Feilner in suse-linux] Na Na Na!!! Ihr könnt doch nicht so einfach über so ein Zertifikat herziehen! Das ist bestimmt der Neid der Besitzlosen! Wenn ich hier an die Wand sehe (da hängt das bei mir), dann lese ich ganz deutlich: Minsweeper Consultant and Solitaire Expert Und darauf lege ich doch grossen Wert!!!! [Konrad Neitzel in suse-linux] -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] Re: ** RECOVERY Service Alert: anna.infra.opensuse.org/Public SSL cert is OK **
by Karol Babioch 26 Jan '20

26 Jan '20

Hi all, the certificates for *.opensuse.org where about to expire in 28 days or so. Unfortunately the automatic issuance and deployment of new certificates (via Let's Encrypt) didn't work as intended - once again. Luckily we have some monitoring setup for this, so we have been notified of this problem. It has been complaing about this for some days now, but I was somehow hoping for a miracle and for the problem to "fix itself". (Un)fortunately computers are deterministic after all, so the problem didn't go away. Upon login on "crtmgr.infra.opensuse.org", I've noticed that there have been two failed services, namely: - dehydrated - sssd sssd fails with the following error messages: > Jan 26 22:03:57 crtmgr sssd[682]: SSSD couldn't load the configuration database [2]: No such file or directory. Not sure why sssd is enabled at all on this machine, since the sssd configuration file doesn't contain anything at all. Maybe this was the result of some (unfinished) Salt run in the past? (The machine is currently not managed in Salt intentionally). I've disabled the sssd for now and consider it to be (totally) unrelated to the actual issue. The journal said the following for dehydrated: > Jan 26 01:39:07 crtmgr systemd[1]: dehydrated.service: Main process exited, code=exited, status=1/FAILURE > Jan 26 01:39:07 crtmgr systemd[1]: Failed to start Certificate Update Runner for Dehydrated. > Jan 26 01:39:07 crtmgr systemd[1]: dehydrated.service: Unit entered failed state. > Jan 26 01:39:07 crtmgr systemd[1]: dehydrated.service: Failed with result 'exit-code'. While dehydrated is a great set of script(s) for Let's Encrypt, there seems to be no further logging. The man page [1] does not even mention debug and/or log at all :-/. We had similar issues in the past (broken dehydrated service), and had to debug them by running the script with "bash -x", etc. pp. This time all I did was to reboot the machine (due to pending kernel updates) and triggering the "dehydrated.service" manually afterwards. To my surprise, it successfully run, fetched new certificates and deployed them through the infrastructure, so the "problem is fixed" now. However, I still don't know why it didn't work when triggered by the systemd timer and I don't see a way to learn more about the issue. I'm not super familiar with dehydrated, and I was not the one who originally set up all of this. Are there any recommendations on how to deal with issues like this in the future? Have I missed some log/debug output that might be helpful in understanding what the root cause of the issue has been? Or is it really impossible to tell why a previous run of the script failed? This seems odd to me ... Best regards, Karol Babioch [1]: https://github.com/lukas2511/dehydrated/blob/master/docs/man/dehydrated.1

1 0

[heroes] pontifex - daily fetch of routeviews disabled?
by Per Jessen 26 Jan '20

26 Jan '20

I see the daily fetch from mirrorbrain was stopped, last one was 10/1 - I presume because of the certificate problem? I was just wondering. -- Per Jessen, Zürich (-0.1°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 7

[heroes] bit of a mirror tidy up
by Per Jessen 21 Jan '20

21 Jan '20

Now that Lars has been cleaning up the ticket queue, there is room for some more :-) We have more than a few mirrors that need some TLC. Some are just gone, others are misconfigured (not on our side). I've opened a few tickets for some today, there is more to come. Thanks for the hint about the marker filename Lars - it had not occurred to me they might simply be wrong. Many of the markers used a file called 'content' which is apparently long gone. I've changed that to 'CHECKSUMS' instead. You might not notice, but http://mirrors.opensuse.org now has a lot(!) more ticks than it used to have. More will appear as mirrors are being scanned. I have also added a marker for 'Tumbleweed history' as well as for Leap 15.2, and removed old ones for Leap 42.x. The complete page should now fit the full browser on a wide screen, without a horizontal scrollbar. It is really just a minor update, except for tumbleweed - we have a good mirror setup for TW, but because of the faulty markers, most of it wasn't being used , so e.g. people in the US ended up using European mirrors. -- Per Jessen, Zürich (3.2°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 1

[heroes] Weekends makes me wish I was using a different distro
by Felix Miata 18 Jan '20

18 Jan '20

I just waited 13.5 minutes for http://download.opensuse.org/distribution/leap/15.2/repo/oss/x86_64/ to completely load in SeaMonkey, 11.5 minutes before it even started to paint anything, 75,987,605 bytes of html. This is just after it took 45 minutes between submitting a new bug, and succeeding to add 3 additional attachments (in 4 parts: 3 15.2 initrds, one too large for acceptance which had to be split) to B.O.O.: http://bugzilla.opensuse.org/show_bug.cgi?id=1161238 http://ftp5.gwdg.de/pub/opensuse/distribution/leap/15.2/repo/oss/x86_64/ took 5 seconds to load html, 2,890,366 bytes. Something is very wrong that opensuse.org is setup to require such gargantuan HTML and 16000% longer load times. -- Evolution as taught in public schools is religion, not science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata *** http://fm.no-ip.com/ -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

6 13

[heroes] Re: [#59920] New DNS infrastructure for openSUSE domains
by Michael Ströder 18 Jan '20

18 Jan '20

Lars, thanks for driving this. On 1/13/20 11:56 PM, admin(a)opensuse.org wrote: > [openSUSE Tracker] > Issue #59920 has been updated by lrupp. > > % Done changed from 20 to 60 > > ns1.opensuse.org and ns2.opensuse.org are online and answer queries for the opensuse.org domain. > > left TODO: > * define a machine outside the Nuremberg network as DNS > * saltify the setup Do you still want some help? IIRC you've installed bind. At heroes meeting there was a tendency to go for PowerDNS. Backend was still to be decided. I have some preference for launch=ldap to have authc/authz integration to another LDAP server [1] and use native LDAP replication for HA. But maintaining DNSSEC RRs with pdns_control is probably not yet supported by PowerDNS' LDAP backend. I've tried to login to nue-ns1.infra.opensuse.org to have a look at the current setup but are probably not allowed to do so. Ciao, Michael. [1] https://www.ae-dir.com/apps.html#slapd-ldap -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

3 5

[heroes] Password Expiry in FreeIPA
by Michael Ströder 16 Jan '20

16 Jan '20

HI! from the last IRC meeting: ------------------- snip ------------------ [20:52:11] <cboltz> ("password expired" only means someone didn't login in FreeIPA for months - but you can still use the VPN and sudo with an expired password) [20:52:29] <kl_eisbaer> sure? (mean: did you try?) [20:52:48] <cboltz> yes, more than once [20:52:54] <kl_eisbaer> as in that case, I have the feeling that FreeIPA has a security problem :-( ------------------- snip ------------------ Known problem since many years. Password expiry is only enforced when authentication with Kerberos, not LDAP: https://bugzilla.redhat.com/show_bug.cgi?id=782917 They claim they have to keep it that way for resetting expired passwords. :-/ In opposite to that password expiry just works for LDAP simple bind in OpenLDAP and Æ-DIR provides a decent password reset web application which also works for expired passwords: https://www.ae-dir.com/pwd.html#reset-procedure This procedure is also used for setting the initial password (because both use-cases share the same security requirements and have the same goal). Ciao, Michael. -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] Leaving SUSE, staying with openSUSE & more changes
by Karol Babioch 14 Jan '20

14 Jan '20

Hi all, there have been quite a few changes within SUSE related to the Engineering Infrastructure team that I would like to share with you for reasons of transparency: - Kurt Garloff has left SUSE some weeks ago. We had a meeting discussing some of our current challenges & issues along with future options with him a few months ago. Notes have been taken [1]. We will try to stay on track with what has been discussed / agreed upon as much as possible. Feel free to remind us of anything we might have missed and/or fail to deliver! - The Engineering Infrastructure team, who is responsible for the openSUSE infrastructure within SUSE has finally a new team lead since late last year. Her name is Evženie Šujskaja (CC). - Obviously she is still trying to get a good understanding of the team and its responsibilities, but she is perfectly aware of the existence and necessity of the openSUSE infrastructure, the heroes and its responsibilities. - Feel free to contact her for any organization topics and whenever something related to the openSUSE infrastructure needs coordination with a representative from SUSE. - I will shortly be leaving SUSE. While I will try to stay as much around and connected within the openSUSE (heroes) community, I will definitely not have as much time and privileges to work on openSUSE as I've used to - also because I have quite a few challenges ahead in my personal life that will require a lot of time and attention ;-). - Please contact Ricardo Klein, Bernhard M. Wiedemann and/or Jens Daniel Schmidt (all of them CC'ed) for any technical questions/concerns that need attention from SUSE's side. At least one of them will try to be around in the monthly heroes meeting. - It is still a good idea to organize a yearly heroes meeting in person (in Nuremberg). Someone else will have to be the organizer / host next time(s), but please do not forget to think about it (e.g. in October / November / December). I'm still hoping to see and meet many of you by interacting in this and other Open Source communities, conferences, etc. In case someone wants to stay in touch more explicitly, I would like you to use my personal mail address (karol(a)babioch.de [also CC'ed]) from this point onward. Thank you so much for the good time and never forget to have a lot of fun keeping the systems up and running ;-). Best regards, Karol Babioch [1]: https://etherpad.opensuse.org/p/openSUSE_Infra

2 1

[heroes] static.o.o etc. outage - workaround
by Christian Boltz 09 Jan '20

09 Jan '20

Hello, (if you hear about this the first time, see status.o.o and the ticket system for some details.) I applied a workaround to get at least static.o.o back in a reliable way: since pages running via login2.o.o work without problems, I switched static.o.o over to use the login proxy. Changes done: - added vhost config for static.o.o on daffy1 - added haproxy config for static.o.o via login proxy on anna - changed DNS entry for static.o.o from "A 195.135.221.143" and "AAAA 2620:113:80c0:8::18" to "CNAME login2.o.o" Note that this only "fixes" static.o.o which causes the most visible problems (loosing all styling on several pages). Other services routed via anna/elsa (everything with CNAME proxy.o.o) are still affected by the network problems. (We could apply the same workaround for them, but I'd prefer to get the network issue fixed instead ;-) Regards, Christian Boltz -- [GUI vs. Command-Line] Einen ähnlichen Streit wird es in 20 Jahren auch geben, wenn die "2D-Screenfanatiker" auf die "VR Fans" losgehen und wieder ein Streit vom Zaun bricht der an Sinnfreiheit kaum zu überbieten ist. [Phillip Richdale in suse-linux] -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 1

[heroes] Status: Monitoring
by Lars Vogdt 08 Jan '20

08 Jan '20

Hi TL;DR: - all Apache Webservers report some statistics now - general monitoring cleanup done - monitor more Long version: One of the first jobs for me was to check the monitoring for changes or broken stuff. Meanwhile I'm happy that only the monitoring of remote logs from our machines is persistent enough to show up nearly all the time in monitoring. All other 1347 service checks from the 49 monitored hosts are either green (99.4% green services) or been acknowledged or worked on. I added more checks especially for gitlab and elasticsearch - and adjusted (or patched and upstreamed) some others. If you think I missed something, please let me know. Just one thing: I *love* monitoring and try to always keep an eye on it. If you miss a check or have an idea, please let me know. These points are open: * deploy monitoring config via Salt With kind regards, Lars -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 1