Hello,
the next heroes meeting will be on Thursday (2023-12-07) at 19:00 UTC /
20:00 CET in https://meet.opensuse.org/heroes
Note: Currently meet.o.o is still down. If it's still down on Thursday,
we'll move to https://meet-test.opensuse.org/heroes
If in doubt, check #opensuse-admin for the latest information shortly
before the meeting.
The usual topics are already listed on
https://progress.opensuse.org/issues/152113
This time we even have an additional topic Georg wants to discuss:
Infrastructure policy, see
https://en.opensuse.org/openSUSE:Infrastructure_policy
Regards,
Christian Boltz
--
The PURPOSE of the list is to give people support. The FUNCTION of the
list is to give greybeards a place to hang out. The JOB of management is
to keep the place pleasant, clean and tidy, and keep the trolls and yobs
at bay. That way, everyone wins :-) [Wols Lists in opensuse]
its back up now
Den fre 24 nov. 2023 kl 10:58 skrev Luna Jernberg <droidbittin(a)gmail.com>:
>
> Hey!
>
> I wanted to vote but the voting site is currently down
>
> Den tors 23 nov. 2023 kl 22:33 skrev Douglas DeMaio <douglas.demaio(a)suse.com>:
> >
> > Hi all,
> > Read the article about the logos contest at https://news.opensuse.org/2023/11/23/selecting-the-new-face-of-os-is-underw… or go to survey.opensuse.org to vote.
> > v/r
> > Doug
On 11/14/23 15:39, Michael Matz wrote:
> Hello Georg,
>
> On Sat, 11 Nov 2023, Georg Pfuetzenreuter wrote:
>
>> 5. Proxying
>>
>> We try to keep the amount of machines with direct exposure to the internet to
>> a minimum in the new infrastructure. Hence all traffic for most public
>> services will need to pass the atlas{1,2}.infra.opensuse.org reverse proxy
>> servers. This was already implemented for all existing services, but should be
>> noted when designing new ones.
>
> So, I gather from this and remarks in Slack that gate.opensuse.org
> deliberately switched off ssh port forwarding. Ergo our method to push
> data from inside the SUSE network to gcc-stats (via ssh -p 2271
> gcc(a)gate.opensuse.org) doesn't work anymore.
>
> What's the alternative to that? openVPN doesn't seem viable as we need to
> rsync-push data from different machines (again, all inside the SUSE
> engineering networks) to gcc-stats. openVPN only normally supports one
> source machine, and additionally the credentials should probably not just
> lie around on random devel machines. In effect I don't really _want_ any
> of these devel machines to be part of the heroes VPN network. Maybe our
> own sshd on gcc-stats open to the SUSE network? But that requires routing
> from SUSE networks directly to gcc-stats. The port forwarding was
> basically the most efficient and secure mechanism for this.
>
> So, yeah, what's the alternative? Thanks for any insight.
Hi Michael,
thanks for reaching out.
A colleague of yours (?) had the same request regarding gcc-stats.i.o.o
and opened a ticket regarding it, in which I already proposed an
alternative two days ago:
https://progress.opensuse.org/issues/139244
Please follow-up in the ticket and coordinate with other users of
gcc-stats.i.o.o in order for all of you to have a common solution.
Best,
Georg
>
>
> Ciao,
> Micha.
Hi Ish,
I noticed strange behaviour of your opensuse-mirror.datakeepers.co.za on
stage.o.o rsync.
It seems as if multiple parallel rsync sessions try to sync the same
files. This could happen, if there is no locking on your side to prevent
concurrent sync runs.
Additionally, you could use
rsync://stage-main-repos.opensuse.org
as source. It has copies of the most interesting 4TB on fast SSDs on a
better network connection.
Ciao
Bernhard M.
Hi everyone,
with our migration from Nuremberg to Prague, several changes in the
infrastructure were introduced.
I yet have to update our admin wiki, but want to highlight some
differences you will need to consider when connecting to the VPN and
navigating around machines.
1. VPN gateway
There is a new VPN gateway in place. If you were using an IP address as
the VPN "remote", please update it to "gate.opensuse.org", or, if you
have specific reasons to not use the domain name, the new addresses
gate.opensuse.org resolves to now.
2. DNS forwarders
The new gateway announces new DNS servers as well, but if you use a
custom forwarding setup (for example using dnsmasq), please update your
forwarding nameservers for
infra.opensuse.org
e.7.2.b.0.4.e.d.7.0.a.2.ip6.arpa.
to
2a07:de40:b27e:1203::11 and 2a07:de40:b27e:1203::12
3. Network segments and shell access
The new infrastructure uses multiple network segments (VLANs) to
separate services. Most existing VMs were migrated to the
"openSUSE-internal" network, which is accessible directly from your VPN
client, hence you should be able to reach such machines as before. Same
applies for connectivity to VMs hosted in Provo.
New, non migrated, machines, might be installed in more restricted
segments. Those are accessible through the thor.infra.opensuse.org
bastion host.
To find which network segments are implemented at a given time, check
the file "pillar/infra/networks.yaml" in our Salt repository.
To find which network segment a machine is residing in, check the
respective "source" field in the file "pillar/infra/hosts.yaml".
https://code.opensuse.org/heroes/salt/blob/production/f/pillar/infra
4. Firewalling
The new network segments implement strict firewalling to ensure both
separation between the segments, but also to the internet. This
firewalling is based on a whitelist-only principle, meaning _all_
traffic is blocked by default, and legitimate connectivity needs to be
allowed on a case-by-case basis.
We tried to track down and implement all necessary connectivity as part
of the server migrations, but if you notice some traffic which is
required for the operation of your service to still be blocked, please
reach out to discuss the options.
5. Proxying
We try to keep the amount of machines with direct exposure to the
internet to a minimum in the new infrastructure. Hence all traffic for
most public services will need to pass the atlas{1,2}.infra.opensuse.org
reverse proxy servers. This was already implemented for all existing
services, but should be noted when designing new ones.
6. IPv6
We believe in the future and decided to run our new internal
infrastructure in a "single-stack", IPv6-only, fashion. Whilst it might
be unexpected for some users to not find an IPv4 address on a network
interface, it was ensured that this new design functions as
transparently as possible. Connectivity to IPv4-only internet services
(for example github.com) is still possible, and so is connectivity to
machines in Provo which have not yet been changed to IPv6 - but note
that the firewall restrictions described in 4. apply here as well.
Of course, we still implemented IPv4 on servers which need to be
reachable by the wide internet audience, such as our proxy servers (5.).
We tried our best to mitigate all possible issues arising with the
firewalling and IPv6, but it is possible that some slipped through.
Please let us know if you encounter any issues, either in
#opensuse-admin or by opening a ticket in
https://progress.opensuse.org/projects/opensuse-admin/issues.
To find which VMs have already been migrated and which are still
pending, please reference the lists at the bottom of
https://etherpad.opensuse.org/p/move-nue-prg.
Have a lot of fun with the new setup ...
Georg
Hello,
the next heroes meeting will be on Thursday (2023-11-02) at 19:00 UTC /
20:00 CET in https://meet.opensuse.org/heroes
Note that Europe switched to winter time, therefore the UTC time
changed.
So far, https://progress.opensuse.org/issues/137501 lists the usual
topics - as usual, feel free to add things you want to discuss.
Oh, and I'm quite sure that the status reports will include a big report
about how the migration to Prague progresses ;-)
Regards,
Christian Boltz
--
> [viele kleine Dateien löschen] Bei ext3 kann man getrost ein
> (oder zwei, oder drei, oder...) Kaffee trinken gehen.
Also bei mir war es offenbar ein sehr unperformantes Dateisystem...
oder Du trinkst sehr langsam Kaffee.
[> Thomas Hertweck und Andre Tann in suse-linux]
Hi everyone,
this week are starting to migrate various machines to our new location
in Prague. This will cause intermittent outage of affected services. I
will be updating https://status.opensuse.org/ where possible.
If there are any issues not correlating to a report on
status.opensuse.org, please reach out in #opensuse-admin (on
ircs://irc.libera.chat) or open a ticket [1] in order for us to investigate.
Note that the status and ticket systems are going to be temporarily
affected as well.
Cheers,
Georg
[1] https://progress.opensuse.org/projects/opensuse-admin/issues
It's been at least 22 hours since I last received email from opensuse.org. Best I
can recall I'm subscribed to at least:
buildservice (maybe not)
factory
heroes
kde
kde3
kernel
mirror
opensuse-factory-mozilla (if it still exists?)
project
support
user
web (if it still exists?)
yast-devel
And I'm subscribed to several hundred bugs. Only one bugmail arrived since
yesterday: 1216070.
Anyone who responds may need to CC me for me to know you did.
--
Evolution as taught in public schools is, like religion,
based on faith, not based on science.
Team OS/2 ** Reg. Linux User #211409 ** a11y rocks!
Felix Miata
Hello,
here are the minutes of today's^Wyesterday's heroes meeting:
2023-10-05 heroes meeting
Start Time - 2000 CEST
Attendees: Bill, Neal, Georg, Carlos, "Captain Tofu", Christian Boltz, Luciano S
Status reports:
- Lots of salt activity to prepare the move to Prague datacenter
- Proposal to add attendees to meeting notes -> ok
- Positive feedback from the new CDN with US users
Infra notes
- Determine whether to redeploy from scratch or reconfig existing
Servers (relocation to Prague DC, working from Georg's spreadsheet)
* Crtmgr
* Separate crtmgr between employee and infra
* Leave old crtmgr to build test
* Verify salt on the crtmgr setup
* Elsa
* Redeploy HAProxy (public services) with Salt
* Internal mail relay (include in existing mx*.i.o.o
* Deploy new NTP (chronyd) with Salt
* Drop turnserver (only used by Jitsi)
* Deploy new internal DB HProxy + PgBouncer with Salt (ex Ipsilon)
* Mickey
* Copy 1:1
* Jekyll
* Redeploy with Salt (Check SSH keys for redeployment)
* Anna
* Same as Elsa
* Backup
* Copy 1:1
* Int
* Copy 1:1 check with mjambor(a)suse.cz
* Chip
* Redeploy Salt with Leap
* Finish deployment of PowerDNS (dnsdist in front, maybe Unbound for internal DNS)
* Goal is to decommisssion BIND and DNSmasq
* Man
* Copy 1:1 ?
* Runs RPM to docsrv
* SUSE to check with tkukuk
* Community
* Copy 1:1 unless @lkocman provides a package
* Hosts docs
* Community 2
* Factory dashboard
* gets two html pages rsynced (rsyncd running), check with dimstar/fvogt for alternative ways
* Other services unknown? (a few more hosts configured in nginx, but not reachable via haproxy)
* Who owns the system ?
* Do we need to move this to something else ?
* for now, copy 1:1
* discourse01
* copy 1:1
* mirrorcache*
* check with anikitin
* dale
* copy 1:1, coordinate with henne
* discourse01
* copy 1:1
* elections2
* redeploy with salt, keep copy as fallback
* etherpad
* copy 1:1
* is data in mysql? -> check
* freeipa
* copy 1:1
* freeipa2
* copy 1:1
* galera*
* setup new VMs, copy over databases while migrating services
* ideally salt the setup (Georg has SUSE salt states we could use as base, but they need cleanup) - volunteer needed
* obsreview
* copy 1:1
* check with henne
* identification2
* unused, drop
* jenkins-agent
* idle VM, re-create
* opi-proxy
* copy 1:1
* jenkins
* idle VM, re-create
* kali
* check with Lars
* limesurvey
* try to re-deploy with salt, keep VM copy as fallback
* malman3
* re-deploy with salt, keep VM copy as fallback
* matomo
* copy 1:1
* matrix
* maintainer to re-deploy with salt, otherwise copy 1:1
* metrics
* copy 1:1
* minnie
* drop (new salt master will be deployed)
* mirrordb*
* actually "just" a pgsql cluster
* try to re-deploy with salt (repmgr part might need to be manual)
* choose a better hostname ;-)
* monitor
* copy 1:1
* mx* and test-mx
* copy 1:1 (check with pjessen, salt available)
* narwal*
* re-deploy with salt (needds updated ssh host keys in salt)
* nue-ns*
* manually re-deploy or copy 1:1, update config (chip.i.o.o config might need updated)
* consider to replace with proper PowerDNS setup later
* nuka
* copy 1:1, clarify with Antonio/Marketa
* olaf
* copy 1:1 with less resources
* remainders of MirrorBrain - consolidate into another machine?
* osc-collab
* copy 1:1, clarify with dimstar
* pagure01
* re-deploy with salt, check with Neal
* pinot
* re-deploy with salt
* pontifex2
* check with build team / Andrii
* progressoo
* copy 1:1, re-deploy with salt
* riesling
* copy 1:1
* riesling3
* copy 1:1
* rpmlint
* copy 1:1
* scar
* SUSE to deploy new OpenVPN server and migrate configurations and CA (easyrsa)
* Firewall parts replaced by asgard*
* svn
* svn.o.o is broken (404)
* also hosts kernel.o.o - candidate for move to narwal* - check with kernel team
* tsp
* copy 1:1
* water*
* copy 1:1
Long-term Goals
- Explore Leap Micro for redeployments (and create SOP)
- How can we turn packages into containers for expanding workload scalability
State of Jitsi
* Packages are out of date and need refreshing, help welcome
* Package sources: https://build.opensuse.org/project/show/openSUSE:infrastructure:jitsi
Regards,
Christian Boltz
--
"Does your computer ever crash?"
"Oh definitely, believe me. We want to make a tool that we can use
ourselves and we know from our own use we can make it a lot better
and a lot more reliable." [Bill Gates in a BBC interview]