Hi all, please find the meeting minutes of our openSUSE Heroes OffSite 2018 below. A more readable (markdown-processed) version of the notes can be found here: * https://therealbro.github.io/openSUSE_Heroes/2018-03-03_openSUSE.html * https://therealbro.github.io/openSUSE_Heroes/2018-03-04_openSUSE.html # openSUSE Heroes offsite meeting minutes ## Where: SUSE Event Area, SUSE HQ, Nuremberg, Germany, Earth, Solar System, Milky Way, Universe ## When: 2018-03-03 / Day 1/2 ## Who: * all-time: * Christian Boltz * Christian Müller * Per Jessen * Sarah Julia Kriesch * Theo Chatzimichos * Thorsten Bro * part-time: * Bernhard Wiedemann * Richard Brown # Day 1 - Saturday - 2018-03-03 ## Coordinator * We need to organize ourselves a bit different, with the loss of Lars. * In general we don't want to have the role of "the Coordinator", as it is not really clear what the responsibilites should/would be * We need people who take-over responsibility for several tasks, but we want to organize ourselves managed by our ticket system * For special purposes - we can define single responsibilities for tasks, e.g.: * Organize off-site or other events [choose per event in Heroes Meeting] * Talk / coordinate with Release Management and Board [Per Jessen] * Hardware / Budget / Sponsoring [Thorsten Bro] * Interface-Function to SUSE-IT / MF-IT and SUSE-people in general [Theo Chatzimichos] * Heroes security (Sticking to coorporate SUSE / MF guidelines for security certification) [Christian Müller] * Onboarding / Guidance of new openSUSE Heroes (Help needed! Ask Theo!) [Theo Chatzimichos, Christian Müller] * We need an onboarding wiki page [Theo Chatzimichos] * In the Heroes meeting - Check the oldest tickets every month - to not loose track [everybody] ## openVPN / Bridging network - Provo / Nuremberg * PRV <---> NUE - ToDo: * Setup transfer network [rwawrig, cmueller] * Setup VPN in Provo [tchatzimichos] * HA-setup of scar + scarface [mcaj] * Add separate gateway machines in Provo/Nuremberg to do routing (not on scar which is facing external) [tchatzimichos, mcaj] * FreeIPA Master to Master setup (Provo / Nuremberg) [?] * Master-to-Master setup DB? [?] ## Board topics * Boards wants us to be more public - actually putting our SALT / Documentation on public pages * The onboarding process should be a starting point, to see which things we can document public and publish to the outside world * SALT states (w/o pillars) will go public * Documentation is up to everybody, to make it more public on its own extent * connect.o.o should be replaced in future - this is a long-term ToDo for the next year * check possibility of moving those extra DB-fields into eDirectory directly * Start communication with Heinlein regarding pushing out mailservice, based on results of connect/eDirectory project ## SLE 11 / Leap 15 - migration of old SLE 11 hosts / migration to Leap 15 of Leap 42.3 hosts ### SLE 11 hosts upgrade to 42.3 * (XX) progress.o.o [tchatzimichos] * (X) connect.o.o (boosters) - WAIT for more info * create disconnect.o.o to disconnect the openSUSE TSP web service from connect.o.o machine / dependencies * (~)old mysql * Wiki-DB [cboltz, tbro] - Move Wiki-DB, Add MySQL-User-Handling to Salt * (EASY)old postgresql [tchatzimichos] * (EASY)narwal - servers (static.o.o, studio express, more) [tbro] -> Move to Leap15 beta * (X)icc.o.o [tbro] (ask Kai about migration) * (X)conference.o.o [Henne and OSEM maintainers] * (X)community (irc-bot, some websites, maybe more stuff, etc.) [tchatzimichos] * tickets are there to be done: find out if stuff can be moved to static.o.o or needs a new machine * IRC-BOT (bugbot) - tchatzimichos will ask Henne about this * (EASY)osc-collab [mcaj] ### Leap 15 #### Testing Leap 15 BETA * Kiwi - Images for Leap 15 [tbro, cboltz] * Salt - Leap 15 Repos [cboltz, bmwiedemann] * Below tasks are blocked by the above: * static.o.o (narwal) move to Leap 15 beta [tbro, see above] * osc-collab to Leap 15 beta [mcaj, see above] * community static pages and bugbot to Leap 15 [tchatzimichos] #### After the Leap 15 GM Release * salt-master [tchatzimichos] * All other web services [all heroes] * Move mlmmj to mailman with testing first ;) [pjessen] ## Mirrors / Database clusters ### Mirrors * pontifex2.i.o.o in NUE * pontifex.i.o.o (rename the machine) [tchatzimichos] * shutdown pontifex3 and be on alert if something breaks (old SLE11) [tchatzimichos] * fix current mirroring setup * fix reverse-DNS for all pontifex machines [mcaj, pjessen, tchatzimichos] * do documentation of whole mirroring setup [mcaj, pjessen, tchatzimichos] * fix monitoring of mirroring [mcaj, pjessen, tchatzimichos] * work on mirrors as onboarding setup * add mirroring servers to salt * Widehat tasks * widehat maybe out of date * replace widehat with new sponsor [tbro] * put widehat to infra.o.o VPN (client) [tchatzimichos] * ask Ludwig about his publishing scripts * reply fast with a "stock message" to mirroring requests (generic for other simple customer requests) ## General advice ### Subscribe to admin-auto@o.o mailing list with your account ### make you machines / services (root-accounts) sending mails to the above mentioned list ## Salt / Automation * More configs / services * Formulas preferred * Static files / templates acceptable * salt-master vs. monitoring machine lists are deviating * monitoring client configs should go to the "base" role so that all machines get basic monitoring * We need the packages / configs / services / NRPE / check_mk / etc. * Check the "deploy CM" doc and see what can be automated * Webpage git repo -> push -> CI -> Reactor -> git pull on minion * GitHub repos cannot access gitlab-ci [bmwiedemann ask jdsn about mirroring git] * Encrypted GPG Pillars * documentation [tchatzichmichos] * Missing script to re-encrypt all GPG pillars * new services should always be configured with Salt! * master in Provo (syndic) # openSUSE Heroes offsite meeting minutes ## Where: SUSE Event Area, SUSE HQ, Nuremberg, Germany, Earth, Solar System, Milky Way, Universe ## When: 2018-03-04 / Day 2/2 ## Who: * all-time: * Christian Boltz * Christian Müller * Per Jessen * Sarah Julia Kriesch * Theo Chatzimichos * Thorsten Bro * Bernhard Wiedemann * part-time: * Ludwig Nussel # Day 2 - Sunday - 2018-03-04 ## monitoring.o.o / status.o.o ### status.o.o [tchatzimichos, tbro] * Users and admin access * Documenation / ReadMe needs to be done * Add new users to status.o.o while Onboarding * Script to check if ACL in tickets, status, FreeIPA agree * Replace with static page * No DB, only Git * No syncing * DNS round-robin will be possible * Git syncing: * GitLab as central * Minimal gitolite mirror on each status instance * Multiple remotes on admin checkouts * Problem: What about mail checks * Check how Gentoo, Fedora and others handle it * status page by Fedora: https://status.fedoraproject.org/ (maually updated) -> https://fedoraproject.org/wiki/Infrastructure#Infrastructure_health_status * status page by Gentoo: https://infra-status-na1.gentoo.org/ -> https://wiki.gentoo.org/wiki/Project:Infrastructure/Servers (server overview) * Current system * Update both: Incident and Service * Check documentation and update it [mcaj, AdaLovelace] ### monitoring.o.o [mcaj, AdaLovelace] * Automatic update of status.o.o through monitoring * Monitoring to send events to status * Status to check periodically if monitoring is accessible * Send events maybe through Salt exec modules * Send messages via Salt to Icinga with the status.$function exec module * collect status with nrpe / checkmk / salt -> send json to icinga * Add server to salt and add its configs / services * All clients need also basic setup via Salt * Monitor from external (Master and Satelite) / (per offered a machine) * NTP to chrony (check_mk offers it) * icinga1 to icinga2 migration * Monitoring in Provo * Thruk to combine Nue/Prv/External ## Sponsoring [tbro] * HW-Wishlist / Colocation-Space / Rootservers -> Write a wishlist [tbro] * cpanel sponsored us this year with new 10Gig converged network adapters - thanks a lot! * general sponsorship contact is Doug - who always should be the first contact * Silver / Gold / Platinum states can be found on: https://en.opensuse.org/Sponsors * We found out, that maybe not all sponsors are always up-to-date (search.o.o, build.o.o, opensuse.o, etc.) * We want to provide a service: sponsors.o.o where you can get * all sponsor logos via a URL in standard format / designs which fit in opensuse-design * deliver a random sponsor logo: http://nginx.org/en/docs/http/ngx_http_random_index_module.html * talk to Doug, designers and setup such a service [tbro] * We took part in the TK-Award from ThomasKrenn * Unfortunately, we were not in the winners' list :'( * Find a replacement with root-servers, rackspace or CDN-solution for widehat.o.o (talk to BuildOPS) [tbro] * Evaluate CDN77 possibilities [cmueller] * Heinlein -> proceed with mailsetup of opensuse.org and evaluate possibilities [cmueller] ## Release Management * progress.o.o - update without plugins * pontifex monitor bandwith - would be helpful to find bugs * openSUSE / publishing-scripts (on GitHub) * they are used for Leap15 only - at the moment * Tumbleweed should switch to them as well in future * Piwik should be split or moved - ask all relevant teams * Release (Leap15) will happen after oSC18 ## Lessons learned * This meeting was more focused on maintenance and filling up the ToDo list for the future, in our first meeting in 2016, we were more focused on learning about infrastructure * Well organized, no presentation missing, here are things to be done - and here are the people who do it * Presentations about major topics: Network, Cloud, Mirror infrastructure, Salt were not given, but were not really missing * We want at least one more in-person meeting during the year - one per year is not enough * Plans are after the oSC (summer and beyond) * On the next meeting, we should have reports of the splitted responsibilities, everybody who took over a responsibility should give a 10-15min overview as a status report about the "specialists topic" what the responsible was working on in the last months * We should maybe increase the communication on #openSUSE-admin @ Freenode - it is not really busy and we should maybe talk about more topics there ## Future topics * Do we want to have the "traditional presentation" at the oSC18 like every year? * We don't want to have the "traditional presentation" at oSC18 * Theo wants to give a presentation on Salt testing in the openSUSE infrastructure * Overview of the Heroes could be inside the last board talk * We want to do a short advertisement at the oSC18 Lightning talks - it will be 5-10 minutes talk - saying who we are and that we're hiring Best regards, -- Thorsten -- To unsubscribe, e-mail: heroes+unsubscribe@opensuse.org To contact the owner, e-mail: heroes+owner@opensuse.org