- Heroes

Reminder: Heroes meeting on Thursday
by Christian Boltz 05 Dec '23

05 Dec '23

Hello, the next heroes meeting will be on Thursday (2023-12-07) at 19:00 UTC / 20:00 CET in https://meet.opensuse.org/heroes Note: Currently meet.o.o is still down. If it's still down on Thursday, we'll move to https://meet-test.opensuse.org/heroes If in doubt, check #opensuse-admin for the latest information shortly before the meeting. The usual topics are already listed on https://progress.opensuse.org/issues/152113 This time we even have an additional topic Georg wants to discuss: Infrastructure policy, see https://en.opensuse.org/openSUSE:Infrastructure_policy Regards, Christian Boltz -- The PURPOSE of the list is to give people support. The FUNCTION of the list is to give greybeards a place to hang out. The JOB of management is to keep the place pleasant, clean and tidy, and keep the trolls and yobs at bay. That way, everyone wins :-) [Wols Lists in opensuse]

1 0

Re: Let the voting begin!!!
by Luna Jernberg 25 Nov '23

25 Nov '23

its back up now Den fre 24 nov. 2023 kl 10:58 skrev Luna Jernberg <droidbittin(a)gmail.com>: > > Hey! > > I wanted to vote but the voting site is currently down > > Den tors 23 nov. 2023 kl 22:33 skrev Douglas DeMaio <douglas.demaio(a)suse.com>: > > > > Hi all, > > Read the article about the logos contest at https://news.opensuse.org/2023/11/23/selecting-the-new-face-of-os-is-underw… or go to survey.opensuse.org to vote. > > v/r > > Doug

1 1

Infrastructure policy
by Georg Pfuetzenreuter 17 Nov '23

17 Nov '23

Hi, I plan to discuss some changes to our infrastructure policy in the next Heroes meeting, however felt the need to already introduce few non-intrusive changes in order to reflect the new infrastructure until then. The full policy can be found here: https://en.opensuse.org/openSUSE:Infrastructure_policy A diff with my changes as of today can be found here: https://en.opensuse.org/index.php?title=openSUSE:Infrastructure_policy&diff… In case of any objections please let me know. Thanks, Georg

1 0

Re: New infrastructure
by Georg Pfuetzenreuter 14 Nov '23

14 Nov '23

On 11/14/23 15:39, Michael Matz wrote: > Hello Georg, > > On Sat, 11 Nov 2023, Georg Pfuetzenreuter wrote: > >> 5. Proxying >> >> We try to keep the amount of machines with direct exposure to the internet to >> a minimum in the new infrastructure. Hence all traffic for most public >> services will need to pass the atlas{1,2}.infra.opensuse.org reverse proxy >> servers. This was already implemented for all existing services, but should be >> noted when designing new ones. > > So, I gather from this and remarks in Slack that gate.opensuse.org > deliberately switched off ssh port forwarding. Ergo our method to push > data from inside the SUSE network to gcc-stats (via ssh -p 2271 > gcc(a)gate.opensuse.org) doesn't work anymore. > > What's the alternative to that? openVPN doesn't seem viable as we need to > rsync-push data from different machines (again, all inside the SUSE > engineering networks) to gcc-stats. openVPN only normally supports one > source machine, and additionally the credentials should probably not just > lie around on random devel machines. In effect I don't really _want_ any > of these devel machines to be part of the heroes VPN network. Maybe our > own sshd on gcc-stats open to the SUSE network? But that requires routing > from SUSE networks directly to gcc-stats. The port forwarding was > basically the most efficient and secure mechanism for this. > > So, yeah, what's the alternative? Thanks for any insight. Hi Michael, thanks for reaching out. A colleague of yours (?) had the same request regarding gcc-stats.i.o.o and opened a ticket regarding it, in which I already proposed an alternative two days ago: https://progress.opensuse.org/issues/139244 Please follow-up in the ticket and coordinate with other users of gcc-stats.i.o.o in order for all of you to have a common solution. Best, Georg > > > Ciao, > Micha.

2 1

opensuse-mirror.datakeepers.co.za
by Bernhard M. Wiedemann 13 Nov '23

13 Nov '23

Hi Ish, I noticed strange behaviour of your opensuse-mirror.datakeepers.co.za on stage.o.o rsync. It seems as if multiple parallel rsync sessions try to sync the same files. This could happen, if there is no locking on your side to prevent concurrent sync runs. Additionally, you could use rsync://stage-main-repos.opensuse.org as source. It has copies of the most interesting 4TB on fast SSDs on a better network connection. Ciao Bernhard M.

2 1

New infrastructure
by Georg Pfuetzenreuter 11 Nov '23

11 Nov '23

Hi everyone, with our migration from Nuremberg to Prague, several changes in the infrastructure were introduced. I yet have to update our admin wiki, but want to highlight some differences you will need to consider when connecting to the VPN and navigating around machines. 1. VPN gateway There is a new VPN gateway in place. If you were using an IP address as the VPN "remote", please update it to "gate.opensuse.org", or, if you have specific reasons to not use the domain name, the new addresses gate.opensuse.org resolves to now. 2. DNS forwarders The new gateway announces new DNS servers as well, but if you use a custom forwarding setup (for example using dnsmasq), please update your forwarding nameservers for infra.opensuse.org e.7.2.b.0.4.e.d.7.0.a.2.ip6.arpa. to 2a07:de40:b27e:1203::11 and 2a07:de40:b27e:1203::12 3. Network segments and shell access The new infrastructure uses multiple network segments (VLANs) to separate services. Most existing VMs were migrated to the "openSUSE-internal" network, which is accessible directly from your VPN client, hence you should be able to reach such machines as before. Same applies for connectivity to VMs hosted in Provo. New, non migrated, machines, might be installed in more restricted segments. Those are accessible through the thor.infra.opensuse.org bastion host. To find which network segments are implemented at a given time, check the file "pillar/infra/networks.yaml" in our Salt repository. To find which network segment a machine is residing in, check the respective "source" field in the file "pillar/infra/hosts.yaml". https://code.opensuse.org/heroes/salt/blob/production/f/pillar/infra 4. Firewalling The new network segments implement strict firewalling to ensure both separation between the segments, but also to the internet. This firewalling is based on a whitelist-only principle, meaning _all_ traffic is blocked by default, and legitimate connectivity needs to be allowed on a case-by-case basis. We tried to track down and implement all necessary connectivity as part of the server migrations, but if you notice some traffic which is required for the operation of your service to still be blocked, please reach out to discuss the options. 5. Proxying We try to keep the amount of machines with direct exposure to the internet to a minimum in the new infrastructure. Hence all traffic for most public services will need to pass the atlas{1,2}.infra.opensuse.org reverse proxy servers. This was already implemented for all existing services, but should be noted when designing new ones. 6. IPv6 We believe in the future and decided to run our new internal infrastructure in a "single-stack", IPv6-only, fashion. Whilst it might be unexpected for some users to not find an IPv4 address on a network interface, it was ensured that this new design functions as transparently as possible. Connectivity to IPv4-only internet services (for example github.com) is still possible, and so is connectivity to machines in Provo which have not yet been changed to IPv6 - but note that the firewall restrictions described in 4. apply here as well. Of course, we still implemented IPv4 on servers which need to be reachable by the wide internet audience, such as our proxy servers (5.). We tried our best to mitigate all possible issues arising with the firewalling and IPv6, but it is possible that some slipped through. Please let us know if you encounter any issues, either in #opensuse-admin or by opening a ticket in https://progress.opensuse.org/projects/opensuse-admin/issues. To find which VMs have already been migrated and which are still pending, please reference the lists at the bottom of https://etherpad.opensuse.org/p/move-nue-prg. Have a lot of fun with the new setup ... Georg

1 0

Reminder: Heroes meeting on Thursday
by Christian Boltz 31 Oct '23

31 Oct '23

Hello, the next heroes meeting will be on Thursday (2023-11-02) at 19:00 UTC / 20:00 CET in https://meet.opensuse.org/heroes Note that Europe switched to winter time, therefore the UTC time changed. So far, https://progress.opensuse.org/issues/137501 lists the usual topics - as usual, feel free to add things you want to discuss. Oh, and I'm quite sure that the status reports will include a big report about how the migration to Prague progresses ;-) Regards, Christian Boltz -- > [viele kleine Dateien löschen] Bei ext3 kann man getrost ein > (oder zwei, oder drei, oder...) Kaffee trinken gehen. Also bei mir war es offenbar ein sehr unperformantes Dateisystem... oder Du trinkst sehr langsam Kaffee. [> Thomas Hertweck und Andre Tann in suse-linux]

1 0

openSUSE infrastructure migration
by Georg Pfuetzenreuter 30 Oct '23

30 Oct '23

Hi everyone, this week are starting to migrate various machines to our new location in Prague. This will cause intermittent outage of affected services. I will be updating https://status.opensuse.org/ where possible. If there are any issues not correlating to a report on status.opensuse.org, please reach out in #opensuse-admin (on ircs://irc.libera.chat) or open a ticket [1] in order for us to investigate. Note that the status and ticket systems are going to be temporarily affected as well. Cheers, Georg [1] https://progress.opensuse.org/projects/opensuse-admin/issues

1 0

[heroes] no list mail today
by Felix Miata 13 Oct '23

13 Oct '23

It's been at least 22 hours since I last received email from opensuse.org. Best I can recall I'm subscribed to at least: buildservice (maybe not) factory heroes kde kde3 kernel mirror opensuse-factory-mozilla (if it still exists?) project support user web (if it still exists?) yast-devel And I'm subscribed to several hundred bugs. Only one bugmail arrived since yesterday: 1216070. Anyone who responds may need to CC me for me to know you did. -- Evolution as taught in public schools is, like religion, based on faith, not based on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata

1 0

Meeting minutes
by Christian Boltz 05 Oct '23

05 Oct '23

Hello, here are the minutes of today's^Wyesterday's heroes meeting: 2023-10-05 heroes meeting Start Time - 2000 CEST Attendees: Bill, Neal, Georg, Carlos, "Captain Tofu", Christian Boltz, Luciano S Status reports: - Lots of salt activity to prepare the move to Prague datacenter - Proposal to add attendees to meeting notes -> ok - Positive feedback from the new CDN with US users Infra notes - Determine whether to redeploy from scratch or reconfig existing Servers (relocation to Prague DC, working from Georg's spreadsheet) * Crtmgr * Separate crtmgr between employee and infra * Leave old crtmgr to build test * Verify salt on the crtmgr setup * Elsa * Redeploy HAProxy (public services) with Salt * Internal mail relay (include in existing mx*.i.o.o * Deploy new NTP (chronyd) with Salt * Drop turnserver (only used by Jitsi) * Deploy new internal DB HProxy + PgBouncer with Salt (ex Ipsilon) * Mickey * Copy 1:1 * Jekyll * Redeploy with Salt (Check SSH keys for redeployment) * Anna * Same as Elsa * Backup * Copy 1:1 * Int * Copy 1:1 check with mjambor(a)suse.cz * Chip * Redeploy Salt with Leap * Finish deployment of PowerDNS (dnsdist in front, maybe Unbound for internal DNS) * Goal is to decommisssion BIND and DNSmasq * Man * Copy 1:1 ? * Runs RPM to docsrv * SUSE to check with tkukuk * Community * Copy 1:1 unless @lkocman provides a package * Hosts docs * Community 2 * Factory dashboard * gets two html pages rsynced (rsyncd running), check with dimstar/fvogt for alternative ways * Other services unknown? (a few more hosts configured in nginx, but not reachable via haproxy) * Who owns the system ? * Do we need to move this to something else ? * for now, copy 1:1 * discourse01 * copy 1:1 * mirrorcache* * check with anikitin * dale * copy 1:1, coordinate with henne * discourse01 * copy 1:1 * elections2 * redeploy with salt, keep copy as fallback * etherpad * copy 1:1 * is data in mysql? -> check * freeipa * copy 1:1 * freeipa2 * copy 1:1 * galera* * setup new VMs, copy over databases while migrating services * ideally salt the setup (Georg has SUSE salt states we could use as base, but they need cleanup) - volunteer needed * obsreview * copy 1:1 * check with henne * identification2 * unused, drop * jenkins-agent * idle VM, re-create * opi-proxy * copy 1:1 * jenkins * idle VM, re-create * kali * check with Lars * limesurvey * try to re-deploy with salt, keep VM copy as fallback * malman3 * re-deploy with salt, keep VM copy as fallback * matomo * copy 1:1 * matrix * maintainer to re-deploy with salt, otherwise copy 1:1 * metrics * copy 1:1 * minnie * drop (new salt master will be deployed) * mirrordb* * actually "just" a pgsql cluster * try to re-deploy with salt (repmgr part might need to be manual) * choose a better hostname ;-) * monitor * copy 1:1 * mx* and test-mx * copy 1:1 (check with pjessen, salt available) * narwal* * re-deploy with salt (needds updated ssh host keys in salt) * nue-ns* * manually re-deploy or copy 1:1, update config (chip.i.o.o config might need updated) * consider to replace with proper PowerDNS setup later * nuka * copy 1:1, clarify with Antonio/Marketa * olaf * copy 1:1 with less resources * remainders of MirrorBrain - consolidate into another machine? * osc-collab * copy 1:1, clarify with dimstar * pagure01 * re-deploy with salt, check with Neal * pinot * re-deploy with salt * pontifex2 * check with build team / Andrii * progressoo * copy 1:1, re-deploy with salt * riesling * copy 1:1 * riesling3 * copy 1:1 * rpmlint * copy 1:1 * scar * SUSE to deploy new OpenVPN server and migrate configurations and CA (easyrsa) * Firewall parts replaced by asgard* * svn * svn.o.o is broken (404) * also hosts kernel.o.o - candidate for move to narwal* - check with kernel team * tsp * copy 1:1 * water* * copy 1:1 Long-term Goals - Explore Leap Micro for redeployments (and create SOP) - How can we turn packages into containers for expanding workload scalability State of Jitsi * Packages are out of date and need refreshing, help welcome * Package sources: https://build.opensuse.org/project/show/openSUSE:infrastructure:jitsi Regards, Christian Boltz -- "Does your computer ever crash?" "Oh definitely, believe me. We want to make a tool that we can use ourselves and we know from our own use we can make it a lot better and a lot more reliable." [Bill Gates in a BBC interview]

1 0