Mailinglist Archive: heroes (10 mails)

< Previous Next >
[heroes] Results: openSUSE Heroes OffSite - 2018-03-03 - 2018-03-04
Hi all,

please find the meeting minutes of our openSUSE Heroes OffSite 2018 below.

A more readable (markdown-processed) version of the notes can be found here:

* https://therealbro.github.io/openSUSE_Heroes/2018-03-03_openSUSE.html

* https://therealbro.github.io/openSUSE_Heroes/2018-03-04_openSUSE.html

# openSUSE Heroes offsite meeting minutes

## Where: SUSE Event Area, SUSE HQ, Nuremberg, Germany, Earth, Solar
System, Milky Way, Universe

## When: 2018-03-03 / Day 1/2

## Who:

* all-time:
* Christian Boltz
* Christian Müller
* Per Jessen
* Sarah Julia Kriesch
* Theo Chatzimichos
* Thorsten Bro

* part-time:
* Bernhard Wiedemann
* Richard Brown

# Day 1 - Saturday - 2018-03-03

## Coordinator

* We need to organize ourselves a bit different, with the loss of Lars.
* In general we don't want to have the role of "the Coordinator", as it
is not really clear what the responsibilites should/would be
* We need people who take-over responsibility for several tasks, but we
want to organize ourselves managed by our ticket system
* For special purposes - we can define single responsibilities for
tasks, e.g.:
* Organize off-site or other events [choose per event in Heroes Meeting]
* Talk / coordinate with Release Management and Board [Per Jessen]
* Hardware / Budget / Sponsoring [Thorsten Bro]
* Interface-Function to SUSE-IT / MF-IT and SUSE-people in general
[Theo Chatzimichos]
* Heroes security (Sticking to coorporate SUSE / MF guidelines for
security certification) [Christian Müller]
* Onboarding / Guidance of new openSUSE Heroes (Help needed! Ask
Theo!) [Theo Chatzimichos, Christian Müller]
* We need an onboarding wiki page [Theo Chatzimichos]
* In the Heroes meeting - Check the oldest tickets every month - to
not loose track [everybody]

## openVPN / Bridging network - Provo / Nuremberg

* PRV <---> NUE - ToDo:
* Setup transfer network [rwawrig, cmueller]
* Setup VPN in Provo [tchatzimichos]
* HA-setup of scar + scarface [mcaj]
* Add separate gateway machines in Provo/Nuremberg to do routing
(not on scar which is facing external) [tchatzimichos, mcaj]
* FreeIPA Master to Master setup (Provo / Nuremberg) [?]
* Master-to-Master setup DB? [?]

## Board topics

* Boards wants us to be more public - actually putting our SALT /
Documentation on public pages
* The onboarding process should be a starting point, to see which
things we can document public and publish to the outside world
* SALT states (w/o pillars) will go public
* Documentation is up to everybody, to make it more public on its own
extent
* connect.o.o should be replaced in future - this is a long-term ToDo
for the next year
* check possibility of moving those extra DB-fields into eDirectory
directly
* Start communication with Heinlein regarding pushing out mailservice,
based on results of connect/eDirectory project

## SLE 11 / Leap 15 - migration of old SLE 11 hosts / migration to Leap
15 of Leap 42.3 hosts

### SLE 11 hosts upgrade to 42.3

* (XX) progress.o.o [tchatzimichos]
* (X) connect.o.o (boosters) - WAIT for more info
* create disconnect.o.o to disconnect the openSUSE TSP web service
from connect.o.o machine / dependencies
* (~)old mysql
* Wiki-DB [cboltz, tbro] - Move Wiki-DB, Add MySQL-User-Handling to Salt
* (EASY)old postgresql [tchatzimichos]
* (EASY)narwal - servers (static.o.o, studio express, more) [tbro] ->
Move to Leap15 beta
* (X)icc.o.o [tbro] (ask Kai about migration)
* (X)conference.o.o [Henne and OSEM maintainers]
* (X)community (irc-bot, some websites, maybe more stuff, etc.)
[tchatzimichos]
* tickets are there to be done: find out if stuff can be moved to
static.o.o or needs a new machine
* IRC-BOT (bugbot) - tchatzimichos will ask Henne about this
* (EASY)osc-collab [mcaj]

### Leap 15

#### Testing Leap 15 BETA

* Kiwi - Images for Leap 15 [tbro, cboltz]
* Salt - Leap 15 Repos [cboltz, bmwiedemann]
* Below tasks are blocked by the above:
* static.o.o (narwal) move to Leap 15 beta [tbro, see above]
* osc-collab to Leap 15 beta [mcaj, see above]
* community static pages and bugbot to Leap 15 [tchatzimichos]

#### After the Leap 15 GM Release

* salt-master [tchatzimichos]
* All other web services [all heroes]
* Move mlmmj to mailman with testing first ;) [pjessen]

## Mirrors / Database clusters

### Mirrors

* pontifex2.i.o.o in NUE
* pontifex.i.o.o (rename the machine) [tchatzimichos]
* shutdown pontifex3 and be on alert if something breaks (old SLE11)
[tchatzimichos]
* fix current mirroring setup
* fix reverse-DNS for all pontifex machines [mcaj, pjessen,
tchatzimichos]
* do documentation of whole mirroring setup [mcaj, pjessen,
tchatzimichos]
* fix monitoring of mirroring [mcaj, pjessen, tchatzimichos]
* work on mirrors as onboarding setup
* add mirroring servers to salt
* Widehat tasks
* widehat maybe out of date
* replace widehat with new sponsor [tbro]
* put widehat to infra.o.o VPN (client) [tchatzimichos]
* ask Ludwig about his publishing scripts
* reply fast with a "stock message" to mirroring requests (generic for
other simple customer requests)

## General advice

### Subscribe to admin-auto@o.o mailing list with your account
### make you machines / services (root-accounts) sending mails to the
above mentioned list

## Salt / Automation

* More configs / services
* Formulas preferred
* Static files / templates acceptable
* salt-master vs. monitoring machine lists are deviating
* monitoring client configs should go to the "base" role so that all
machines get basic monitoring
* We need the packages / configs / services / NRPE / check_mk / etc.
* Check the "deploy CM" doc and see what can be automated
* Webpage git repo -> push -> CI -> Reactor -> git pull on minion
* GitHub repos cannot access gitlab-ci [bmwiedemann ask jdsn about
mirroring git]
* Encrypted GPG Pillars
* documentation [tchatzichmichos]
* Missing script to re-encrypt all GPG pillars
* new services should always be configured with Salt!
* master in Provo (syndic)

# openSUSE Heroes offsite meeting minutes

## Where: SUSE Event Area, SUSE HQ, Nuremberg, Germany, Earth, Solar
System, Milky Way, Universe

## When: 2018-03-04 / Day 2/2

## Who:

* all-time:
* Christian Boltz
* Christian Müller
* Per Jessen
* Sarah Julia Kriesch
* Theo Chatzimichos
* Thorsten Bro
* Bernhard Wiedemann

* part-time:
* Ludwig Nussel

# Day 2 - Sunday - 2018-03-04

## monitoring.o.o / status.o.o

### status.o.o [tchatzimichos, tbro]

* Users and admin access
* Documenation / ReadMe needs to be done
* Add new users to status.o.o while Onboarding
* Script to check if ACL in tickets, status, FreeIPA agree
* Replace with static page
* No DB, only Git
* No syncing
* DNS round-robin will be possible
* Git syncing:
* GitLab as central
* Minimal gitolite mirror on each status instance
* Multiple remotes on admin checkouts
* Problem: What about mail checks
* Check how Gentoo, Fedora and others handle it
* status page by Fedora: https://status.fedoraproject.org/ (maually
updated) ->
https://fedoraproject.org/wiki/Infrastructure#Infrastructure_health_status
* status page by Gentoo: https://infra-status-na1.gentoo.org/ ->
https://wiki.gentoo.org/wiki/Project:Infrastructure/Servers (server
overview)

* Current system
* Update both: Incident and Service
* Check documentation and update it [mcaj, AdaLovelace]

### monitoring.o.o [mcaj, AdaLovelace]

* Automatic update of status.o.o through monitoring
* Monitoring to send events to status
* Status to check periodically if monitoring is accessible
* Send events maybe through Salt exec modules

* Send messages via Salt to Icinga with the status.$function exec module
* collect status with nrpe / checkmk / salt -> send json to icinga
* Add server to salt and add its configs / services
* All clients need also basic setup via Salt
* Monitor from external (Master and Satelite) / (per offered a machine)
* NTP to chrony (check_mk offers it)
* icinga1 to icinga2 migration
* Monitoring in Provo
* Thruk to combine Nue/Prv/External

## Sponsoring [tbro]

* HW-Wishlist / Colocation-Space / Rootservers -> Write a wishlist [tbro]
* cpanel sponsored us this year with new 10Gig converged network
adapters - thanks a lot!
* general sponsorship contact is Doug - who always should be the first
contact
* Silver / Gold / Platinum states can be found on:
https://en.opensuse.org/Sponsors
* We found out, that maybe not all sponsors are always up-to-date
(search.o.o, build.o.o, opensuse.o, etc.)
* We want to provide a service: sponsors.o.o where you can get
* all sponsor logos via a URL in standard format / designs which fit
in opensuse-design
* deliver a random sponsor logo:
http://nginx.org/en/docs/http/ngx_http_random_index_module.html
* talk to Doug, designers and setup such a service [tbro]
* We took part in the TK-Award from ThomasKrenn
* Unfortunately, we were not in the winners' list :'(
* Find a replacement with root-servers, rackspace or CDN-solution for
widehat.o.o (talk to BuildOPS) [tbro]
* Evaluate CDN77 possibilities [cmueller]
* Heinlein -> proceed with mailsetup of opensuse.org and evaluate
possibilities [cmueller]

## Release Management

* progress.o.o - update without plugins
* pontifex monitor bandwith - would be helpful to find bugs
* openSUSE / publishing-scripts (on GitHub)
* they are used for Leap15 only - at the moment
* Tumbleweed should switch to them as well in future
* Piwik should be split or moved - ask all relevant teams
* Release (Leap15) will happen after oSC18

## Lessons learned

* This meeting was more focused on maintenance and filling up the ToDo
list for the future, in our first meeting in 2016, we were more focused
on learning about infrastructure
* Well organized, no presentation missing, here are things to be done -
and here are the people who do it
* Presentations about major topics: Network, Cloud, Mirror
infrastructure, Salt were not given, but were not really missing
* We want at least one more in-person meeting during the year - one per
year is not enough
* Plans are after the oSC (summer and beyond)
* On the next meeting, we should have reports of the splitted
responsibilities, everybody who took over a responsibility should give a
10-15min overview as a status report about the "specialists topic" what
the responsible was working on in the last months
* We should maybe increase the communication on #openSUSE-admin @
Freenode - it is not really busy and we should maybe talk about more
topics there

## Future topics

* Do we want to have the "traditional presentation" at the oSC18 like
every year?
* We don't want to have the "traditional presentation" at oSC18
* Theo wants to give a presentation on Salt testing in the openSUSE
infrastructure
* Overview of the Heroes could be inside the last board talk
* We want to do a short advertisement at the oSC18 Lightning talks -
it will be 5-10 minutes talk - saying who we are and that we're hiring


Best regards,

--
Thorsten
--
To unsubscribe, e-mail: heroes+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: heroes+owner@xxxxxxxxxxxx

< Previous Next >
Follow Ups