Heroes March 2018

heroes@lists.opensuse.org

5 participants
6 discussions

[heroes] [ddemaio@suse.de: [opensuse-project] Planned Outage To Affect openSUSE Services This Weekend]
by Theo Chatzimichos 09 Mar '18

09 Mar '18

Hello, we will need to prepare for this, at least for status2.o.o ----- Forwarded message from ddemaio <ddemaio(a)suse.de> ----- From: ddemaio <ddemaio(a)suse.de> To: Opensuse Project <opensuse-project(a)opensuse.org>, opensuse-factory(a)opensuse.org Date: Tue, 06 Mar 2018 14:59:15 +0100 Subject: [opensuse-project] Planned Outage To Affect openSUSE Services This Weekend Hi all, A quick FYI. A network outage this weekend will effect openSUSE services that have a Direction Connection to Nuremberg host infrastructure on Saturday between the hours of 1 and 6 p.m. UTC. The outage will last about 30 minutes and take place between the above listed hours. The outage effects only external connections due to maintenance in the datacenter. All services will run without any downtime. v/r Doug -- To unsubscribe, e-mail: opensuse-project+unsubscribe(a)opensuse.org To contact the owner, email: opensuse-project+owner(a)opensuse.org ----- End forwarded message ----- -- Theo Chatzimichos <tampakrap(a)opensuse.org> <tchatzimichos(a)suse.com> System Administrator SUSE Operations and Services Team

2 3

[heroes] minor mirroring update
by Per Jessen 08 Mar '18

08 Mar '18

We have not had any useful logging on pontifex (download.o.o) since Dec 8 2017. The apache logformat "combinedio_redirect" was not defined, so we have some four months of logfiles full of lines of "combinedio_redirect" :-) I have reinstated that logformat. I think I'll run some daily cronjob on the error-logs and start spamming admin-auto with some log excerpts. I found a German mirror being unused due to a network prefix being 0.0.0.0./0. -- Per Jessen, Zürich (6.5°C) Member, openSUSE Heroes -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] Piwik
by ddemaio 08 Mar '18

08 Mar '18

Hi all, I track the numbers on https://beans.opensuse.org/piwik/ every week and happened to notice that there was sharp drop about two weeks ago on the index opensuse.org. I was tracking about 100k visits a week and then all of the sudden it dropped to 3k. Do you happen to know if anyone worked on it or if something was broken. Also, this might be something someone can answer. Is there anyway we can track the numbers via Piwik for http://download.opensuse.org/ports/aarch64/ images? I was asked if I was tracking the amount of downloads we had with ARM images and I told him I wasn't tracking the numbers for that. The numbers might be good to know. v/r Doug -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

[heroes] updates on the repositories list on openSUSE:infrastructure
by Theo Chatzimichos 08 Mar '18

08 Mar '18

Hello, I performed the following actions on the repositories of openSUSE:infrastructure (only, not the subrepos) - added leap15 and sle15 repos - removed leap42.1 and sle12sp2 - disabled by default leap42.2 Let me know if there are any problems/objections -- Theo Chatzimichos <tampakrap(a)opensuse.org> <tchatzimichos(a)suse.com> System Administrator SUSE Operations and Services Team

1 0

[heroes] Results: openSUSE Heroes OffSite - 2018-03-03 - 2018-03-04
by Thorsten Bro 05 Mar '18

05 Mar '18

Hi all, please find the meeting minutes of our openSUSE Heroes OffSite 2018 below. A more readable (markdown-processed) version of the notes can be found here: * https://therealbro.github.io/openSUSE_Heroes/2018-03-03_openSUSE.html * https://therealbro.github.io/openSUSE_Heroes/2018-03-04_openSUSE.html # openSUSE Heroes offsite meeting minutes ## Where: SUSE Event Area, SUSE HQ, Nuremberg, Germany, Earth, Solar System, Milky Way, Universe ## When: 2018-03-03 / Day 1/2 ## Who: * all-time: * Christian Boltz * Christian Müller * Per Jessen * Sarah Julia Kriesch * Theo Chatzimichos * Thorsten Bro * part-time: * Bernhard Wiedemann * Richard Brown # Day 1 - Saturday - 2018-03-03 ## Coordinator * We need to organize ourselves a bit different, with the loss of Lars. * In general we don't want to have the role of "the Coordinator", as it is not really clear what the responsibilites should/would be * We need people who take-over responsibility for several tasks, but we want to organize ourselves managed by our ticket system * For special purposes - we can define single responsibilities for tasks, e.g.: * Organize off-site or other events [choose per event in Heroes Meeting] * Talk / coordinate with Release Management and Board [Per Jessen] * Hardware / Budget / Sponsoring [Thorsten Bro] * Interface-Function to SUSE-IT / MF-IT and SUSE-people in general [Theo Chatzimichos] * Heroes security (Sticking to coorporate SUSE / MF guidelines for security certification) [Christian Müller] * Onboarding / Guidance of new openSUSE Heroes (Help needed! Ask Theo!) [Theo Chatzimichos, Christian Müller] * We need an onboarding wiki page [Theo Chatzimichos] * In the Heroes meeting - Check the oldest tickets every month - to not loose track [everybody] ## openVPN / Bridging network - Provo / Nuremberg * PRV <---> NUE - ToDo: * Setup transfer network [rwawrig, cmueller] * Setup VPN in Provo [tchatzimichos] * HA-setup of scar + scarface [mcaj] * Add separate gateway machines in Provo/Nuremberg to do routing (not on scar which is facing external) [tchatzimichos, mcaj] * FreeIPA Master to Master setup (Provo / Nuremberg) [?] * Master-to-Master setup DB? [?] ## Board topics * Boards wants us to be more public - actually putting our SALT / Documentation on public pages * The onboarding process should be a starting point, to see which things we can document public and publish to the outside world * SALT states (w/o pillars) will go public * Documentation is up to everybody, to make it more public on its own extent * connect.o.o should be replaced in future - this is a long-term ToDo for the next year * check possibility of moving those extra DB-fields into eDirectory directly * Start communication with Heinlein regarding pushing out mailservice, based on results of connect/eDirectory project ## SLE 11 / Leap 15 - migration of old SLE 11 hosts / migration to Leap 15 of Leap 42.3 hosts ### SLE 11 hosts upgrade to 42.3 * (XX) progress.o.o [tchatzimichos] * (X) connect.o.o (boosters) - WAIT for more info * create disconnect.o.o to disconnect the openSUSE TSP web service from connect.o.o machine / dependencies * (~)old mysql * Wiki-DB [cboltz, tbro] - Move Wiki-DB, Add MySQL-User-Handling to Salt * (EASY)old postgresql [tchatzimichos] * (EASY)narwal - servers (static.o.o, studio express, more) [tbro] -> Move to Leap15 beta * (X)icc.o.o [tbro] (ask Kai about migration) * (X)conference.o.o [Henne and OSEM maintainers] * (X)community (irc-bot, some websites, maybe more stuff, etc.) [tchatzimichos] * tickets are there to be done: find out if stuff can be moved to static.o.o or needs a new machine * IRC-BOT (bugbot) - tchatzimichos will ask Henne about this * (EASY)osc-collab [mcaj] ### Leap 15 #### Testing Leap 15 BETA * Kiwi - Images for Leap 15 [tbro, cboltz] * Salt - Leap 15 Repos [cboltz, bmwiedemann] * Below tasks are blocked by the above: * static.o.o (narwal) move to Leap 15 beta [tbro, see above] * osc-collab to Leap 15 beta [mcaj, see above] * community static pages and bugbot to Leap 15 [tchatzimichos] #### After the Leap 15 GM Release * salt-master [tchatzimichos] * All other web services [all heroes] * Move mlmmj to mailman with testing first ;) [pjessen] ## Mirrors / Database clusters ### Mirrors * pontifex2.i.o.o in NUE * pontifex.i.o.o (rename the machine) [tchatzimichos] * shutdown pontifex3 and be on alert if something breaks (old SLE11) [tchatzimichos] * fix current mirroring setup * fix reverse-DNS for all pontifex machines [mcaj, pjessen, tchatzimichos] * do documentation of whole mirroring setup [mcaj, pjessen, tchatzimichos] * fix monitoring of mirroring [mcaj, pjessen, tchatzimichos] * work on mirrors as onboarding setup * add mirroring servers to salt * Widehat tasks * widehat maybe out of date * replace widehat with new sponsor [tbro] * put widehat to infra.o.o VPN (client) [tchatzimichos] * ask Ludwig about his publishing scripts * reply fast with a "stock message" to mirroring requests (generic for other simple customer requests) ## General advice ### Subscribe to admin-auto(a)o.o mailing list with your account ### make you machines / services (root-accounts) sending mails to the above mentioned list ## Salt / Automation * More configs / services * Formulas preferred * Static files / templates acceptable * salt-master vs. monitoring machine lists are deviating * monitoring client configs should go to the "base" role so that all machines get basic monitoring * We need the packages / configs / services / NRPE / check_mk / etc. * Check the "deploy CM" doc and see what can be automated * Webpage git repo -> push -> CI -> Reactor -> git pull on minion * GitHub repos cannot access gitlab-ci [bmwiedemann ask jdsn about mirroring git] * Encrypted GPG Pillars * documentation [tchatzichmichos] * Missing script to re-encrypt all GPG pillars * new services should always be configured with Salt! * master in Provo (syndic) # openSUSE Heroes offsite meeting minutes ## Where: SUSE Event Area, SUSE HQ, Nuremberg, Germany, Earth, Solar System, Milky Way, Universe ## When: 2018-03-04 / Day 2/2 ## Who: * all-time: * Christian Boltz * Christian Müller * Per Jessen * Sarah Julia Kriesch * Theo Chatzimichos * Thorsten Bro * Bernhard Wiedemann * part-time: * Ludwig Nussel # Day 2 - Sunday - 2018-03-04 ## monitoring.o.o / status.o.o ### status.o.o [tchatzimichos, tbro] * Users and admin access * Documenation / ReadMe needs to be done * Add new users to status.o.o while Onboarding * Script to check if ACL in tickets, status, FreeIPA agree * Replace with static page * No DB, only Git * No syncing * DNS round-robin will be possible * Git syncing: * GitLab as central * Minimal gitolite mirror on each status instance * Multiple remotes on admin checkouts * Problem: What about mail checks * Check how Gentoo, Fedora and others handle it * status page by Fedora: https://status.fedoraproject.org/ (maually updated) -> https://fedoraproject.org/wiki/Infrastructure#Infrastructure_health_status * status page by Gentoo: https://infra-status-na1.gentoo.org/ -> https://wiki.gentoo.org/wiki/Project:Infrastructure/Servers (server overview) * Current system * Update both: Incident and Service * Check documentation and update it [mcaj, AdaLovelace] ### monitoring.o.o [mcaj, AdaLovelace] * Automatic update of status.o.o through monitoring * Monitoring to send events to status * Status to check periodically if monitoring is accessible * Send events maybe through Salt exec modules * Send messages via Salt to Icinga with the status.$function exec module * collect status with nrpe / checkmk / salt -> send json to icinga * Add server to salt and add its configs / services * All clients need also basic setup via Salt * Monitor from external (Master and Satelite) / (per offered a machine) * NTP to chrony (check_mk offers it) * icinga1 to icinga2 migration * Monitoring in Provo * Thruk to combine Nue/Prv/External ## Sponsoring [tbro] * HW-Wishlist / Colocation-Space / Rootservers -> Write a wishlist [tbro] * cpanel sponsored us this year with new 10Gig converged network adapters - thanks a lot! * general sponsorship contact is Doug - who always should be the first contact * Silver / Gold / Platinum states can be found on: https://en.opensuse.org/Sponsors * We found out, that maybe not all sponsors are always up-to-date (search.o.o, build.o.o, opensuse.o, etc.) * We want to provide a service: sponsors.o.o where you can get * all sponsor logos via a URL in standard format / designs which fit in opensuse-design * deliver a random sponsor logo: http://nginx.org/en/docs/http/ngx_http_random_index_module.html * talk to Doug, designers and setup such a service [tbro] * We took part in the TK-Award from ThomasKrenn * Unfortunately, we were not in the winners' list :'( * Find a replacement with root-servers, rackspace or CDN-solution for widehat.o.o (talk to BuildOPS) [tbro] * Evaluate CDN77 possibilities [cmueller] * Heinlein -> proceed with mailsetup of opensuse.org and evaluate possibilities [cmueller] ## Release Management * progress.o.o - update without plugins * pontifex monitor bandwith - would be helpful to find bugs * openSUSE / publishing-scripts (on GitHub) * they are used for Leap15 only - at the moment * Tumbleweed should switch to them as well in future * Piwik should be split or moved - ask all relevant teams * Release (Leap15) will happen after oSC18 ## Lessons learned * This meeting was more focused on maintenance and filling up the ToDo list for the future, in our first meeting in 2016, we were more focused on learning about infrastructure * Well organized, no presentation missing, here are things to be done - and here are the people who do it * Presentations about major topics: Network, Cloud, Mirror infrastructure, Salt were not given, but were not really missing * We want at least one more in-person meeting during the year - one per year is not enough * Plans are after the oSC (summer and beyond) * On the next meeting, we should have reports of the splitted responsibilities, everybody who took over a responsibility should give a 10-15min overview as a status report about the "specialists topic" what the responsible was working on in the last months * We should maybe increase the communication on #openSUSE-admin @ Freenode - it is not really busy and we should maybe talk about more topics there ## Future topics * Do we want to have the "traditional presentation" at the oSC18 like every year? * We don't want to have the "traditional presentation" at oSC18 * Theo wants to give a presentation on Salt testing in the openSUSE infrastructure * Overview of the Heroes could be inside the last board talk * We want to do a short advertisement at the oSC18 Lightning talks - it will be 5-10 minutes talk - saying who we are and that we're hiring Best regards, -- Thorsten -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

2 1

[heroes] Heroes offsite (location)
by Thorsten Bro 03 Mar '18

03 Mar '18

Hi all, When you are wondering - where to go at 9.30 today? In the front of the building at Rollnerstrasse you find a SUSE geeko which looks like a bar sign... there is the SUSE Event Area - thats where we meet. See you there. Cheers, -- Thorsten Bro <tbro(a)opensuse.org> - Member of openSUSE Heroes - -- To unsubscribe, e-mail: heroes+unsubscribe(a)opensuse.org To contact the owner, e-mail: heroes+owner(a)opensuse.org

1 0

2022

2021

2020

2019

2018

2017

2016

Heroes March 2018