- Heroes

openSUSE infrastructure migration
by Georg Pfuetzenreuter 30 Oct '23

30 Oct '23

Hi everyone, this week are starting to migrate various machines to our new location in Prague. This will cause intermittent outage of affected services. I will be updating https://status.opensuse.org/ where possible. If there are any issues not correlating to a report on status.opensuse.org, please reach out in #opensuse-admin (on ircs://irc.libera.chat) or open a ticket [1] in order for us to investigate. Note that the status and ticket systems are going to be temporarily affected as well. Cheers, Georg [1] https://progress.opensuse.org/projects/opensuse-admin/issues

1 0

[heroes] no list mail today
by Felix Miata 13 Oct '23

13 Oct '23

It's been at least 22 hours since I last received email from opensuse.org. Best I can recall I'm subscribed to at least: buildservice (maybe not) factory heroes kde kde3 kernel mirror opensuse-factory-mozilla (if it still exists?) project support user web (if it still exists?) yast-devel And I'm subscribed to several hundred bugs. Only one bugmail arrived since yesterday: 1216070. Anyone who responds may need to CC me for me to know you did. -- Evolution as taught in public schools is, like religion, based on faith, not based on science. Team OS/2 ** Reg. Linux User #211409 ** a11y rocks! Felix Miata

1 0

Meeting minutes
by Christian Boltz 05 Oct '23

05 Oct '23

Hello, here are the minutes of today's^Wyesterday's heroes meeting: 2023-10-05 heroes meeting Start Time - 2000 CEST Attendees: Bill, Neal, Georg, Carlos, "Captain Tofu", Christian Boltz, Luciano S Status reports: - Lots of salt activity to prepare the move to Prague datacenter - Proposal to add attendees to meeting notes -> ok - Positive feedback from the new CDN with US users Infra notes - Determine whether to redeploy from scratch or reconfig existing Servers (relocation to Prague DC, working from Georg's spreadsheet) * Crtmgr * Separate crtmgr between employee and infra * Leave old crtmgr to build test * Verify salt on the crtmgr setup * Elsa * Redeploy HAProxy (public services) with Salt * Internal mail relay (include in existing mx*.i.o.o * Deploy new NTP (chronyd) with Salt * Drop turnserver (only used by Jitsi) * Deploy new internal DB HProxy + PgBouncer with Salt (ex Ipsilon) * Mickey * Copy 1:1 * Jekyll * Redeploy with Salt (Check SSH keys for redeployment) * Anna * Same as Elsa * Backup * Copy 1:1 * Int * Copy 1:1 check with mjambor(a)suse.cz * Chip * Redeploy Salt with Leap * Finish deployment of PowerDNS (dnsdist in front, maybe Unbound for internal DNS) * Goal is to decommisssion BIND and DNSmasq * Man * Copy 1:1 ? * Runs RPM to docsrv * SUSE to check with tkukuk * Community * Copy 1:1 unless @lkocman provides a package * Hosts docs * Community 2 * Factory dashboard * gets two html pages rsynced (rsyncd running), check with dimstar/fvogt for alternative ways * Other services unknown? (a few more hosts configured in nginx, but not reachable via haproxy) * Who owns the system ? * Do we need to move this to something else ? * for now, copy 1:1 * discourse01 * copy 1:1 * mirrorcache* * check with anikitin * dale * copy 1:1, coordinate with henne * discourse01 * copy 1:1 * elections2 * redeploy with salt, keep copy as fallback * etherpad * copy 1:1 * is data in mysql? -> check * freeipa * copy 1:1 * freeipa2 * copy 1:1 * galera* * setup new VMs, copy over databases while migrating services * ideally salt the setup (Georg has SUSE salt states we could use as base, but they need cleanup) - volunteer needed * obsreview * copy 1:1 * check with henne * identification2 * unused, drop * jenkins-agent * idle VM, re-create * opi-proxy * copy 1:1 * jenkins * idle VM, re-create * kali * check with Lars * limesurvey * try to re-deploy with salt, keep VM copy as fallback * malman3 * re-deploy with salt, keep VM copy as fallback * matomo * copy 1:1 * matrix * maintainer to re-deploy with salt, otherwise copy 1:1 * metrics * copy 1:1 * minnie * drop (new salt master will be deployed) * mirrordb* * actually "just" a pgsql cluster * try to re-deploy with salt (repmgr part might need to be manual) * choose a better hostname ;-) * monitor * copy 1:1 * mx* and test-mx * copy 1:1 (check with pjessen, salt available) * narwal* * re-deploy with salt (needds updated ssh host keys in salt) * nue-ns* * manually re-deploy or copy 1:1, update config (chip.i.o.o config might need updated) * consider to replace with proper PowerDNS setup later * nuka * copy 1:1, clarify with Antonio/Marketa * olaf * copy 1:1 with less resources * remainders of MirrorBrain - consolidate into another machine? * osc-collab * copy 1:1, clarify with dimstar * pagure01 * re-deploy with salt, check with Neal * pinot * re-deploy with salt * pontifex2 * check with build team / Andrii * progressoo * copy 1:1, re-deploy with salt * riesling * copy 1:1 * riesling3 * copy 1:1 * rpmlint * copy 1:1 * scar * SUSE to deploy new OpenVPN server and migrate configurations and CA (easyrsa) * Firewall parts replaced by asgard* * svn * svn.o.o is broken (404) * also hosts kernel.o.o - candidate for move to narwal* - check with kernel team * tsp * copy 1:1 * water* * copy 1:1 Long-term Goals - Explore Leap Micro for redeployments (and create SOP) - How can we turn packages into containers for expanding workload scalability State of Jitsi * Packages are out of date and need refreshing, help welcome * Package sources: https://build.opensuse.org/project/show/openSUSE:infrastructure:jitsi Regards, Christian Boltz -- "Does your computer ever crash?" "Oh definitely, believe me. We want to make a tool that we can use ourselves and we know from our own use we can make it a lot better and a lot more reliable." [Bill Gates in a BBC interview]

1 0

Reminder: Heroes meeting today
by Christian Boltz 05 Oct '23

05 Oct '23

Hello, we'll have a heroes meeting today at 18:00 UTC / 20:00 CEST in https://meet.opensuse.org/heroes So far, we have the usual topics on our agenda. Feel free to add topics to https://progress.opensuse.org/issues/135386, or just bring them up in the meeting. Regards, Christian Boltz PS: Sorry for sending out the reminder later than usual! -- [package naming] Another funny example is libreoffice, it actually collides with our shared library policy ;-) [Sascha Peilicke in opensuse-packaging]

2 1

Hiperkitty under moderation
by Carlos E. R. 27 Sep '23

27 Sep '23

Hi, On the Spanish mail list, I added a header check. All mail which have user-agent=HyperKitty on https://lists.opensuse.org/ are now moderated. Reason is that the list has got recently (September) a dozen or more English language spam, all coming via that interface, from "non members" apparently. Somehow spammers found a hole and they are exploiting it. Interesting thing is, the first hit provoked a reply from the spammer, sent to users-es-owner(a)lists.opensuse.org. It is possible that they interpret the rejection post as a "reply to our spam" that triggers a "reply to customer" automatically. If that is so, I may eventually have to silently reject or delete all email from hiperkitty. It would then be preferable to simply close the page, in order to not confuse possible good faith posters. Curio: https://lists.opensuse.org/manage/lists/users-es.lists.opensuse.org/members… has 9491 entries including this last attempt. -- Cheers / Saludos, Carlos E. R. (from openSUSE 15.5 (Laicolasse))

3 7

Please ban Ayhem Kahri <Ayhem.kahri@skiff.com>
by Knurpht-openSUSE 19 Sep '23

19 Sep '23

Hi all, This person has been banned for serious misbehaviour on our reddit, was banned indefinitely, and tries to circumvent this by going to our MLs. Recently with a clearly AI generated spam message. The Board was informed and decided that Ayhem Kahri <Ayhem.kahri(a)skiff.com> should be banned from our MLs, i.e. their emails going straight to /dev/null. Thanks in advance. please keep us posted on status. Gertjan Lettink a.k.a. Knurpht openSUSE Board openSUSE Forums Team

1 0

Question regarding Heinlein sponsoring
by Lars Vogdt 14 Sep '23

14 Sep '23

Hi there! Heinlein[1] is listed as Sponsor for openSUSE since a long time now. And from my old memory, Per offered in the past to provide Email accounts for openSUSE members for us (via mailbox.org). As there is an ongoing move of the openSUSE infrastructure planned during the next weeks, I'm thinking to myself: what about using the momentum and re-thinking the current setup??? 1) keep the own mx*.opensuse.org servers as they are They are already somehow redundant (with mx1 & mx2 in the SUSE DC in Nuremberg, mx3 in a Colocation and mx4 in the SUSE DC in Provo - but I don't know the status of the last two) and do their work well from my point of view. 2) allow openSUSE members to request an own, real @opensuse.org mailbox With this "opt-in", everyone who wants to read (and write!) Emails as $member(a)opensuse.org would be able to do so. -> No more problems with some providers who do not accept Emails sent as $member(a)opensuse.org ... 3) Maybe extend the sponsorship for a while? While meet.opensuse.org runs fine and without problems (thanks!) at the moment, Heinlein started just recently with an additional service: OpenTalk[2]. As Jitsi might be not available during the migration into the new SUSE DC, what about asking if openSUSE can (temporary) use (test) the new Videoconferencing solution? With kind regards, Lars -- [1]: https://en.opensuse.org/Sponsors#Heinlein_Support [2]: https://opentalk.eu/en/product

1 0

Meeting minutes
by Christian Boltz 07 Sep '23

07 Sep '23

Hello, here are the minutes of today's meeting: 2023-09-07 heroes meeting Bernhard: cdn.o.o debugging: purge broken -> reduced TTL for repomd.xml -> low cache-hit-rate ; will work on a fix with fastly+Dirk openSUSE server move to Prague * install new physical machines * plan network layout + firewall * use port-whitelists * also filter outgoing traffic 15.5 upgrades * most machines upgraded * Henne will do 2 himself * pontifex todo (andrii to clean up) * l10n to be done by sbrabec/team * riesling/wiki to be investigated by cboltz * status/cachet missing package repairs Updates * galera cluster nodes should not all be rebooted at the same time; try Salt solution We also worked on some issues live during the meeting - see the merge requests in our salt repo that were created and merged today. Regards, Christian Boltz -- > [viele kleine Dateien löschen] Bei ext3 kann man getrost ein > (oder zwei, oder drei, oder...) Kaffee trinken gehen. Also bei mir war es offenbar ein sehr unperformantes Dateisystem... oder Du trinkst sehr langsam Kaffee. [> Thomas Hertweck und Andre Tann in suse-linux]

1 0

Re: Question regarding cdn and mirror
by Bernhard M. Wiedemann 06 Sep '23

06 Sep '23

On 06/09/2023 07.07, medwinz wrote: > Hi mirror list, > > As we already know, openSUSE implements cdn now. Could someone explain > to me what the impact of the implementation is to our mirror? AFAIK > cdn will maintain the newest copy from one legitimate source, I > imagine this would be the mirror in openSUSE infrastructure. If it is > the case anyone will use the closest cdn rather than the mirror. My > understanding might be wrong. > > Looking at the statistics in the mirror that I manage the traffic is > still high, most likely the TX amount is more than 100GB daily, > sometimes more than 200GB. Thousand daily hits and hundred visitors. > > Anyone see the significant traffic difference before and after the cdn > implementation? It is expected that traffic remains: 1. The CDN is only partially used atm - either via opt-in or automatic for some regions. We need to improve the config some more before it can become the default for everyone. 2. The CDN contract comes with limited bandwidth, so the current plan is to serve only small files (<100kB) directly via CDN (because there latency matters most) and for larger files cache the redirect or metalink pointing to mirrors. Plus we serve files through CDN that are not yet found on any mirrors. In one random sample, we found that download.o.o redirected 1.5 TB/h worth of traffic to our 100+ mirrors - it would be expensive (around 6kUSD/mon) to replace all of that. And abandoning our mirrors would mean, we have no good fallback in case of CDN outages. 3. The CDN is a caching proxy. It is good because we only need to transfer files that are really requested. It is bad because it adds latency when caches are cold or expired. Using a mirror directly is still the fastest for many users. Ciao Bernhard M.

1 0

Reminder: Heroes meeting on thursday
by Christian Boltz 05 Sep '23

05 Sep '23

Hello, the next Heroes meeting will be on Thursday (2023-09-07) at 18:00 UTC / 20:00 CEST in https://meet.opensuse.org/heroes So far, https://progress.opensuse.org/issues/133778 lists the usual topics. Feel free to add whatever you want to discuss, or just bring it up in the meeting. Regards, Christian Boltz -- > Was ist ein "umbrella bug"? Eine Regenschirm-Wanze ;-) [> Al Bogner und Andreas Winkelmann in suse-linux]

1 0