September 2022 - Heroes - openSUSE Mailing Lists

improving download.o.o
by Bernhard M. Wiedemann 12 Oct '22

12 Oct '22

Hi, as you might know, pontifex2 aka download.o.o serves 4 major roles atm. 1. download.opensuse.org redirector for zypper repos 2. downloadcontent.opensuse.org fallback mirror 3. stage.opensuse.org rsync source server for major mirrors 4. repopusher to 6 mirrors During August, my nagios monitoring has logged at least 20+10+20+19+22+70+30+10+20+10+10 minutes of high packet loss on download.o.o - that is 241m or 4h When I captured a tcpdump during such a troublesome moment, I found that 89% of it was rsync traffic and the total traffic sent was between 2.5 and 3 GBit/s on average, with 60 different mirror IPs. We only have a 4GBit/s link there shared for all of the Nuremberg SUSE Maxtorhof stuff. That indicates that 3. and 4. above saturated the link and that contibuted to high packet loss that makes for a very poor user-experience of download.o.o users. With 60 mirrors rsyncing simultaneously at 3GBit, that leaves around 6 MByte/s for each mirror and that means, syncing just 122GB of Factory takes over 5h. [Math 122000/(3000.0/8/60)/60/60 => 5.4 ] To solve this, we should try to separate these roles so that they conflict less with each other. The first thing we should try is to move out and split stage.o.o . Ideally, the data should be transferred only once to each continent via repopusher and then other mirrors pull from these primary mirrors. Another easy improvement might be to serve download.o.o via mirrorcache from another network. This might not even need much bandwidth as it mostly serves redirects. I'd like to have Andrii's opinion on this, once he is back from vacation. As this is a major change to our download infra, I don't want to decide on that alone, so I'd appreciate your feedback if and how we should change things. Ciao Bernhard M.

4 7

Spam is increasing
by Carlos E. R. 02 Oct '22

02 Oct '22

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On users-es(a)lists.opensuse.org and users(a)lists.opensuse.org. Also factory(a)lists.opensuse.org. - -- Cheers Carlos E. R. (from openSUSE Leap 15.3 at Telcontar) -----BEGIN PGP SIGNATURE----- iHYEARECADYWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCYzbUsRgcY2FybG9zLmUu ckBvcGVuc3VzZS5vcmcACgkQtTMYHG2NR9UiYgCfRqCvxMGa2iV2y+lAacwvpDhB PxYAn1Ywc3gs+fzUnHqUuI5VXVqTUWE6 =NFTS -----END PGP SIGNATURE-----

5 16

thanks for all your efforts tidying up the list of issues, but .....
by Per Jessen 24 Sep '22

24 Sep '22

crameleon, avicenzi - maybe others too: when working on our list of issues - a) spam - don't reject, just delete. Anything else is a waste of time and effort. b) if you reject an issue, please add a courteous explanation. People get really pissed off if you just reject. c) changing priority usually means nothing. -- Per Jessen, Zürich (12.8°C) Member, openSUSE Heroes (2016 - present) We're hiring - https://en.opensuse.org/openSUSE:Heroes

2 3

improving the repopush
by Per Jessen 13 Sep '22

13 Sep '22

I guess not everyone may be aware (?), but we offer a push-service for the build service repositories, such that they can be made available on mirrors as soon as possible. It is up to the mirror admin "on the other end" to ask for this service to be set up. Currently we have six active mirrors with repositories being pushed out to them. There are also five that have stopped for some or other reason. We are pushing out a lot of changes every day, and the original scheme dates back to a different era. The rate of change keeps going up and up and the setup was growing increasingly slow. Well, for the last couple of weeks, I have been looking at how to improve this - I think I have come up with an improved repopush setup - see attached. I put my scheme into motion on 28Aug. Initially I had some teething problems, but as of 2 September, it seems that things have been improving steadily. A fast mirror like ftp.gwdg.de can now complete a push in about three hours, although it depends on how busy build service has been. Our own rsync.o.o is also doing quite well. The new scheme is to generate a Makefile based on the contents of /srv/bs/arrived/ and /srv/bs/pushed/ - 'make' is very good at figuring out what needs to be pushed and 'make' is also good at running things in parallel. pseudo-code: do forever generate makefile.repos run pushes in parallel, one per mirror when one push finishes, cancel the rest [1] done [1] to avoid starving a fast mirror. There is some work left to do - the server push scripts need to return a proper return code to indicate if a push was successful, for instance. They also still do locking, which is no longer necessary. Key code elements - /srv/bs/bin/push_repositories /srv/bs/bin/repopush.genmake I also need to create a systemd service unit, but that's no big deal. -- Per Jessen, Zürich (18.8°C) Member, openSUSE Heroes

1 0

Re: improving download.o.o
by Alexandre Vicenzi 08 Sep '22

08 Sep '22

Hi, The rsync problem could be solved if something else is implemented. Zypper fetches packages from the closest place or where it is available. What if the mirrors did a similar thing? Let's say, there's a small mirror app that queries the official openSUSE servers and get the latest metadata. With this metadata, the mirror app could fetch the RPM packages from another mirror, like Zypper does. This could even be P2P, or just a dump app the queries an API to get the lastest medatata, and queries mirrorcache to get where the package is found. I believe SLES RMT has a similar behavior. Best, Alexandre ________________________________ From: Bernhard M. Wiedemann Sent: Wednesday, September 7, 2022 6:52 PM To: heroes(a)lists.opensuse.org Subject: Re: improving download.o.o I gathered some more data points: /var/log/apache2/download.opensuse.org/2022/09/download.opensuse.org-20220904-access_log.xz contained 200 GB worth of HTTP 200 responses - counted with perl -ne 'm/ 200 (\d+) / and $s+=$1; END{print $s}' /var/log/nginx/downloadcontent/2022/09/2022-09-04-access.log-20220905.xz contained 720 GB worth of HTTP 200 and another 100 GB worth of 206 (Partial) responses for that day. /proc/net/dev showed an average of 21TB/d sent over the 6d uptime - equivalent of a continuous 2GBit/s - but of course it is not steady. I used grep -o "GET [^ ]* HTTP/1.* 200 [^ ]* " \ 2022-09-04-access.log-20220905 | sort|uniq -c|sort -n | tail -200 > most-requested-direct-downloads-size To get the list of most requested (unmirrored) files (attached) and found that a caching squid frontend could provide nice bandwidth savings for tumbleweed and update repos - but probably less than those 720 GB/d served. I still think that the main problem is the rsync traffic. You can use tcpdump -nr with the /tmp/dump*.pcap files and look for rsync port 873, which made up 89% of it. Apart from the practical tcpdump approach, there is also the math, that says, dividing 3000 MBit/s by 70 mirrors leaves only 43 MBit/s to each of them. I think, the best approach would be to find or create a few well-connected rsync mirrors that get pushed /tumbleweed and /update first and from there, the other mirrors can sync. This could increase latency of updates, because it needs 2 copies to reach most mirrors, but OTOH, the copies should happen much faster, because more total bandwidth will be available. Ciao Bernhard M.

2 1

Discourse
by Sasi Olin 07 Sep '22

07 Sep '22

Hi, I am intending to restart work on salting discourse, which has been blocked for years by two things: * Migration to vb5 * Fixing the database in vb4 I know there has been some progress on vb4 database fix, but vb5 migration seems to be blocked on auth which would likely be easier to fix by just migrating to discourse right away, since it uses a system we have integrated in other apps in the project already. The problem I am facing is that since the vb4 fix isn't public right now, any attempt at database migration testing I could perform on discourse is not gonna be done on the same state of the database as we would get after applying the fix. How should we proceed then? From my personal pov, setting up a duplicate vb4 setup and fixing a database, so that we can properly test it and apply those fixes back into the main instance is the best way to solve this issue. From there we can just work on migrating to discourse, with testing that actually tests the correct state of the database. Alternatively I'm sure we could find a Nextcloud plugin to replace forums with Nextcloud too, I'm sure that would solve all the problems ;) LCP [Sasi] https://lcp.world/

6 12

Meeting minutes
by Christian Boltz 01 Sep '22

01 Sep '22

Hello, here are the minutes of today's meeting. Q&A from the community * Alex asked for access to the ticket system to help with handling tickets Status reports: * lots of status updates from Per on the mailinglist: https://lists.opensuse.org/archives/list/heroes@lists.opensuse.org/message/… * mailman_webui often eats 100% CPU and is slow, needs debugging, python-debugging-foo wanted * download.o.o trouble - rsync eats too much bandwidth -> move stage.o.o to a different host with 10Gbit link? * can we handle GDPR requests via progress.o.o or do we have to redirect people to the official privacy@ process? AI: Bernhard to check in SUSE and report in October Regards, Christian Boltz -- Ich sags ja immer: RfCs in Stein meisseln (5cm starke Granitplatten, grosser Font) und diese Idioten solange damit schlagen, bis sie es entweder kapiert haben, oder der Genpool bereinigt wurde. [J. P. Meier in dasr]

1 0

Reminder: Heroes meeting on Thursday
by Christian Boltz 01 Sep '22

01 Sep '22

Hello, the next heroes meeting will be on Thursday (2022-09-01) at 18:00 UTC / 20:00 CEST in https://meet.opensuse.org/heroes https://progress.opensuse.org/issues/116030 already lists the usual topics - feel free to add things you want to discuss. Regards, Christian Boltz -- <tampakrap> DimStar: here? <DimStar> tampakrap: I'm not sure - do you have good news for me ? :) [from #opensuse-admin]

4 3