Hi
Sorry that I missed the meeting. I just forgot it...
Here are my short notes (sorry, long day, just trying to make it short):
* DNSSec is enabled for opensuse.org domain since last weekend. Meanwhile also the caches should be purged and all DNS servers should provide valid responses. Please check, if we missed something...
* mirrordb2 is currently the one and only PostgreSQL DB. I am not 100% sure if this is really wanted, but... ...on the other side, it forced me to think about a backup of the databases. Currently there is a NFS mount (mirrordb2:/backup) from backup.infra.opensuse.org, allowing the postgresql-backupscript to dump a backup of each database (individually) every day around midnight.
* The new (Hey, nearly...) widehat is ready and syncing. I just did neither find the time to prepare some more VMs (like MX, static, DB-Server, more?) nor to exchange it with the current one. Sorry for that. I hope to find some time during the next two weeks for this. As result, our old widehat aka rsync.opensuse.org will not be available for a day.
* There is still some old stuff up and running, that we should shut down, if we follow our old policy (or should I better say: idea?): + users.o.o -> old ELGG instance + wiki.o.o -> old mediawiki instance + elections.o.o -> old helios instance Any objections?
* We could simply upgrade Redmine (aka progress.o.o) to the latest version, if we would not use these old, self-backed plugins. Some of them might be obsolete already, some not used - but others? Anyway: there is progress-test.o.o - a new machine running the new Redmine on an old DB dump. Authentication does not work (yet), as we might simply switch to one of the supported authentication mechanisms (and get rid of another, old plugin). Anyone interested to drive this?
* We plan to move more and more traffic to the MirrorCache instances during the next days, to test them under more load. For this, ~50% of the traffic on download.o.o will be redirected to the MirrorCache instance. This should be transparent for the users. For those interested: https://mirrorcache-eu.opensuse.org/ looks more and more like http://download.opensuse.org/ - but still needs some styling to be "openSUSE".
* DNS management is meanwhile on chip: https://chip.infra.opensuse.org/ If you like to get access, please ping me. LDAP authentication works, but we need to put you in the right group in the WebUI, so you can edit DNS settings. Note 1: infra.opensuse.org is currently still on FreeIPA. Can be moved at any time. I'm just waiting for *your* go here. After that, everything DNS related is on chip. Note 2: `pdnutil edit-zone opensuse.org` on the command line might be much easier. Don't worry, if you get asked if you want to increase the serial number of the zone once you saved your changes. Please answer 'y', so the DNS servers get aware of your changes (they just compare the serial to know if something has changed).
* There is a backup.infra.opensuse.org machine with 3T space, waiting to get filled by clients. Anyone who wants to suggest a good, open source, backup tool?
* I like to build some redundancy of machines running in Nuermberg in Provo and at our CoLo, to survive outages better in the future. Anyone, who wants to join this "project business continuity"?
With kind regards, Lars
Lars Vogdt wrote:
- mirrordb2 is currently the one and only PostgreSQL DB. I am not 100% sure if this is really wanted, but...
What happened to db1 ?
- DNS management is meanwhile on chip: https://chip.infra.opensuse.org/ If you like to get access, please ping me. LDAP authentication works, but we need to put you in the right group in the WebUI, so you can edit DNS settings.
Done.
Note 1: infra.opensuse.org is currently still on FreeIPA. Can be moved at any time. I'm just waiting for *your* go here. After that, everything DNS related is on chip.
From my pov, let's move it.
- There is a backup.infra.opensuse.org machine with 3T space, waiting to get filled by clients. Anyone who wants to suggest a good, open source, backup tool?
differential rsync ?
What happened to db1 ?
It is probably in corrupted state since last outage. I wasn't fully sure about the plan to fight the outage and decided to try restoring logical backup in pg 12 on mirrordb2, leaving mirrordb1 as is just in case (if restoration fails or problem is somehow mirrordb1-specific and for eventual further troubleshooting of root cause of failures).
Since root cause of the problems is more or less known - the action plan may be to let mirrordb1 replicate mirrordb2 or make mirrordb1 primary again with pg 13, then upgrade mirrordb2 to pg13 and let it replicate mirrordb1.
-- Andrii Nikitin andrii.nikitin@suse.com DevOPS Automation and Build Service Engineer SUSE Software Solutions Germany GmbH Maxfeldstr. 5 90409 Nuremberg Germany
(HRB 247165, AG München) Managing Director: Felix Imendörffer
Andrii Nikitin wrote:
Since root cause of the problems is more or less known -
Oh, that's good, I was not aware.
the action plan may be to let mirrordb1 replicate mirrordb2 or make mirrordb1 primary again with pg 13, then upgrade mirrordb2 to pg13 and let it replicate mirrordb1.
Yeah, something like that.
Since root cause of the problems is more or less known -
Oh, that's good, I was not aware.
Sorry - somehow the feedback got lost. Last communication from Reinhard Max was that Postgres has nasty bug related to glibc upgrade, which e.g. happens during Leap upgrade 15.2 -> 15.3 It affects both pg 12 and pg 13 and proper workaround is to rebuild all tables with unique string indexes. Since all tables were freshly created at mirrordb2 when restoring logical backup - mirrordb2 should be good. But it doesn't require to stick to pg 12 - it should be safe to upgrade it to pg 13 now.
Am Wed, 7 Jul 2021 12:38:06 +0000 schrieb Andrii Nikitin andrii.nikitin@suse.com:
But it doesn't require to stick to pg 12 - it should be safe to upgrade it to pg 13 now.
https://news.opensuse.org/2021/04/06/upgrading-to-the-next-postgresql-versio...
There will be a downtime. But it should be less than if we do a full dump - reload cycle.
Regards, Lars
Forgot: * Upgraded Gitlab to 13.12.8 - please report back, if something is not working for you.
Hello,
Am Dienstag, 6. Juli 2021, 23:27:55 CEST schrieb Lars Vogdt:
Sorry that I missed the meeting. I just forgot it...
You'll get an extra invitation next month ;-)
Here are my short notes (sorry, long day, just trying to make it short):
- DNSSec is enabled for opensuse.org domain since last weekend. Meanwhile also the caches should be purged and all DNS servers
should provide valid responses. Please check, if we missed something...
I didn't do DNS queries to check, but... Nobody complained, and everything "just works", therefore you probably did a good job :-)
[...]
- There is still some old stuff up and running, that we should shut down, if we follow our old policy (or should I better say: idea?):
- users.o.o -> old ELGG instance
- wiki.o.o -> old mediawiki instance
- elections.o.o -> old helios instance
Any objections?
Yes ;-) (and I also know that two of these servers are on my TODO list for too long :-/ - I won't complain if someone wants to help upgrading them.)
- We could simply upgrade Redmine (aka progress.o.o) to the latest version, if we would not use these old, self-backed plugins. Some of
them might be obsolete already, some not used - but others? Anyway: there is progress-test.o.o - a new machine running the new Redmine on an old DB dump. Authentication does not work (yet), as we might simply switch to one of the supported authentication mechanisms (and get rid of another, old plugin).
If a supported plugin works for us, I see no reason not to use it. Especially if it replaces something we'd have to maintain on our own.
Anyone interested to drive this?
Sorry, my TODO list is already full ;-) (Also, my Ruby knownledge is basically non-existing - I'm still surprised that my very first one-line patch fixed a problem instead of creating new ones.)
However, if you are unsure if a specific plugin is still needed, I can probably answer that.
[...]
- DNS management is meanwhile on chip: https://chip.infra.opensuse.org/
[...]
Note 1: infra.opensuse.org is currently still on FreeIPA. Can be moved at any time. I'm just waiting for *your* go here.
IMHO you can move it so that we have everything in one place.
After that, everything DNS related is on chip.
Well, nearly ;-)
The *.vpn.infra.o.o zones are still living on freeipa.i.o.o, and not available on chip. (Note: you'll probably need to update your script to create VPN users when moving these zones.)
[...]
- There is a backup.infra.opensuse.org machine with 3T space, waiting to get filled by clients. Anyone who wants to suggest a good, open source, backup tool?
A backup tool? I guess the hard part is to suggest _one_ tool ;-) I'll try nevertheless...
For mostly static content (like files uploaded to the wikis) I'd recommend rsnapshot. It's quite simple, "just works" and makes recovery easy.
Ideally backup.i.o.o should _fetch_ the files from the (for example) wiki server so that old backups can't be damaged from/by the wiki server. (Technically, this can be a done with a ssh key which is restricted to reading files [1] or a read-only NFS export.)
The major problem of rsnapshot is that it will need quite some disk space for content that changes every day (worst case: database dumps) because the only de-duplication it does are hardlinks to the previous backup if a file is unchanged.
Speaking about backup - do we have enough disk space in Provo (and a good enough connection) to do backups there? That would be even better than having the backup in the same datacenter.
- I like to build some redundancy of machines running in Nuermberg in Provo and at our CoLo, to survive outages better in the future. Anyone, who wants to join this "project business continuity"?
I'll happily help with doing this for narwal* aka static.o.o - in theory, it will only need a few lines in salt so that the deploy script syncs to another server :-)
IIRC the scripting behind the jekyll-based pages (news.o.o etc.) is prepared to deploy to multiple servers, so that also shouldn't be too hard.
Regards,
Christian Boltz
[1] I have a read-only ssh root access on my laptop so that my backup server can login and fetch everything. This is done with a combination of ssh and AppArmor, details on request.
Am July 8, 2021 6:07:59 PM UTC schrieb Christian Boltz opensuse@cboltz.de:
- DNS management is meanwhile on chip: https://chip.infra.opensuse.org/
[...]
Note 1: infra.opensuse.org is currently still on FreeIPA. Can be moved at any time. I'm just waiting for *your* go here.
IMHO you can move it so that we have everything in one place.
Ok.
After that, everything DNS related is on chip.
Well, nearly ;-)
The *.vpn.infra.o.o zones are still living on freeipa.i.o.o, and not available on chip. (Note: you'll probably need to update your script to create VPN users when moving these zones.)
Right and wrong at the same time ;-)
*vpn.infra.o.o domains are "maintained" at scar. There is already a script, that creates the DNS entries (tcp,udp,general & reverse) every day from the VPN setup. So far, all other DNS servers just forward requests for these domains to scar. I would not touch this (unless someone takes over), as it "just works". :-)
Regards, Lars
Hello,
Am Donnerstag, 8. Juli 2021, 22:44:02 CEST schrieb Lars Vogdt:
Am July 8, 2021 6:07:59 PM UTC schrieb Christian Boltz:
Note 1: infra.opensuse.org is currently still on FreeIPA. Can be moved at any time. I'm just waiting for *your* go here.
IMHO you can move it so that we have everything in one place.
Ok.
Just looked at FreeIPA again - there are a few more zones and reverse zones left. Feel free to move all of them to chip.
The *.vpn.infra.o.o zones are still living on freeipa.i.o.o, and not available on chip. (Note: you'll probably need to update your script to create VPN users when moving these zones.)
Right and wrong at the same time ;-)
*vpn.infra.o.o domains are "maintained" at scar. There is already a script, that creates the DNS entries (tcp,udp,general & reverse) every day from the VPN setup. So far, all other DNS servers just forward requests for these domains to scar. I would not touch this (unless someone takes over), as it "just works". :-)
Oh, I wasn't aware of this detail ;-) and had guessed that you somehow automagically update these zones in FreeIPA. Since you dropped the (probably outdated) *.vpn.i.o.o zones from FreeIPA in the meantime and it still works, my guess was obviously wrong.
Anyway, feel free to keep vpn.i.o.o on scar.
Regards,
Christian Boltz
-
Andrii Nikitin -
Christian Boltz -
Lars Vogdt -
Per Jessen