Hello,
here are the minutes of today's heroes meeting:
* downloadcontent2.o.o (a cloud VM) improved the stability of
download.o.o a lot - documented at
https://progress.opensuse.org/projects/opensuse-admin-wiki/wiki/
Downloadcontent2
* /update/leap/15.3/sle/repodata/*-primary.xml.gz (120 MB) is a
"hot" file causing traffic spikes
* not pushing Tumbleweed images in home repos can save traffic -
they get rebuilt more often than downloaded
* there's desire for a new server (3 TB SSD disk, 10 GBit network)
that can host the most important repos and will serve as a rsync
master for most mirrors (not certain yet)
* mirrorcache-br.o.o
* was cache.opensuse.net.br
* https://progress.opensuse.org/issues/116710 - "collect root passwords"
(or not)
* Georg: root password hashes to be set to * in /etc/shadow
* (which also means we'll enforce using sudo or ssh-key)
* Georg: close GDPR related tickets with reference to privacy at
suse.com
* Georg: close email alias related tickets with reference to
https://code.opensuse.org/project/membership/new_issue
Regards,
Christian Boltz
--
Es wird also sehr wohl gemountet, wie soll denn yast sonst auf die
CD zugreifen? Per dd erstmal alles auf die HD kopieren? Oder lässt
es sich die Pakete vielleicht als .wav vorsingen?
[Jan Trippler in suse-linux]
Hello,
the next heroes meeting will be on Thursday (2022-10-06) at 18:00 UTC /
20:00 CEST in https://meet.opensuse.org/heroes
So far the meeting ticket https://progress.opensuse.org/issues/116140
lists the usual topics. Feel free to add whatever you want to discuss,
or just bring it up during the meeting.
Regards,
Christian Boltz
--
Die Borg sind einfach eine Allegorie auf M$: gross, toll und voller
endloser Featuritis - aber wenn es ernst wird, sterben sie an einer
Schutzverletzung. [Andreas Pohlke in drsst]
Hi,
as you might know, pontifex2 aka download.o.o serves 4 major roles atm.
1. download.opensuse.org redirector for zypper repos
2. downloadcontent.opensuse.org fallback mirror
3. stage.opensuse.org rsync source server for major mirrors
4. repopusher to 6 mirrors
During August, my nagios monitoring has logged at least
20+10+20+19+22+70+30+10+20+10+10 minutes of high packet loss on
download.o.o - that is 241m or 4h
When I captured a tcpdump during such a troublesome moment, I found that
89% of it was rsync traffic and the total traffic sent was between 2.5
and 3 GBit/s on average, with 60 different mirror IPs. We only have a
4GBit/s link there shared for all of the Nuremberg SUSE Maxtorhof stuff.
That indicates that 3. and 4. above saturated the link and that
contibuted to high packet loss that makes for a very poor
user-experience of download.o.o users.
With 60 mirrors rsyncing simultaneously at 3GBit, that leaves around 6
MByte/s for each mirror and that means, syncing just 122GB of Factory
takes over 5h.
[Math
122000/(3000.0/8/60)/60/60
=> 5.4
]
To solve this, we should try to separate these roles so that they
conflict less with each other.
The first thing we should try is to move out and split stage.o.o .
Ideally, the data should be transferred only once to each continent via
repopusher and then other mirrors pull from these primary mirrors.
Another easy improvement might be to serve download.o.o via mirrorcache
from another network. This might not even need much bandwidth as it
mostly serves redirects.
I'd like to have Andrii's opinion on this, once he is back from vacation.
As this is a major change to our download infra, I don't want to decide
on that alone, so I'd appreciate your feedback if and how we should
change things.
Ciao
Bernhard M.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On users-es(a)lists.opensuse.org and users(a)lists.opensuse.org. Also
factory(a)lists.opensuse.org.
- --
Cheers
Carlos E. R.
(from openSUSE Leap 15.3 at Telcontar)
-----BEGIN PGP SIGNATURE-----
iHYEARECADYWIQQZEb51mJKK1KpcU/W1MxgcbY1H1QUCYzbUsRgcY2FybG9zLmUu
ckBvcGVuc3VzZS5vcmcACgkQtTMYHG2NR9UiYgCfRqCvxMGa2iV2y+lAacwvpDhB
PxYAn1Ywc3gs+fzUnHqUuI5VXVqTUWE6
=NFTS
-----END PGP SIGNATURE-----
crameleon, avicenzi - maybe others too:
when working on our list of issues -
a) spam - don't reject, just delete. Anything else is a waste of time
and effort.
b) if you reject an issue, please add a courteous explanation. People
get really pissed off if you just reject.
c) changing priority usually means nothing.
--
Per Jessen, Zürich (12.8°C)
Member, openSUSE Heroes (2016 - present)
We're hiring - https://en.opensuse.org/openSUSE:Heroes
I guess not everyone may be aware (?), but we offer a push-service for
the build service repositories, such that they can be made available on
mirrors as soon as possible. It is up to the mirror admin "on the
other end" to ask for this service to be set up.
Currently we have six active mirrors with repositories being pushed out
to them. There are also five that have stopped for some or other
reason.
We are pushing out a lot of changes every day, and the original scheme
dates back to a different era. The rate of change keeps going up and
up and the setup was growing increasingly slow.
Well, for the last couple of weeks, I have been looking at how to
improve this -
I think I have come up with an improved repopush setup - see attached.
I put my scheme into motion on 28Aug. Initially I had some teething
problems, but as of 2 September, it seems that things have been
improving steadily. A fast mirror like ftp.gwdg.de can now complete a
push in about three hours, although it depends on how busy build
service has been. Our own rsync.o.o is also doing quite well.
The new scheme is to generate a Makefile based on the contents
of /srv/bs/arrived/ and /srv/bs/pushed/ - 'make' is very good at
figuring out what needs to be pushed and 'make' is also good at running
things in parallel.
pseudo-code:
do forever
generate makefile.repos
run pushes in parallel, one per mirror
when one push finishes, cancel the rest [1]
done
[1] to avoid starving a fast mirror.
There is some work left to do - the server push scripts need to return a
proper return code to indicate if a push was successful, for instance.
They also still do locking, which is no longer necessary.
Key code elements -
/srv/bs/bin/push_repositories
/srv/bs/bin/repopush.genmake
I also need to create a systemd service unit, but that's no big deal.
--
Per Jessen, Zürich (18.8°C)
Member, openSUSE Heroes
Hi,
The rsync problem could be solved if something else is implemented.
Zypper fetches packages from the closest place or where it is available.
What if the mirrors did a similar thing?
Let's say, there's a small mirror app that queries the official openSUSE servers and get the latest metadata.
With this metadata, the mirror app could fetch the RPM packages from another mirror, like Zypper does.
This could even be P2P, or just a dump app the queries an API to get the lastest medatata, and queries mirrorcache to get where the package is found.
I believe SLES RMT has a similar behavior.
Best,
Alexandre
________________________________
From: Bernhard M. Wiedemann
Sent: Wednesday, September 7, 2022 6:52 PM
To: heroes(a)lists.opensuse.org
Subject: Re: improving download.o.o
I gathered some more data points:
/var/log/apache2/download.opensuse.org/2022/09/download.opensuse.org-20220904-access_log.xz
contained 200 GB worth of HTTP 200 responses
- counted with
perl -ne 'm/ 200 (\d+) / and $s+=$1; END{print $s}'
/var/log/nginx/downloadcontent/2022/09/2022-09-04-access.log-20220905.xz
contained 720 GB worth of HTTP 200 and another 100 GB worth of 206
(Partial) responses for that day.
/proc/net/dev showed an average of 21TB/d sent over the 6d uptime -
equivalent of a continuous 2GBit/s - but of course it is not steady.
I used
grep -o "GET [^ ]* HTTP/1.* 200 [^ ]* " \
2022-09-04-access.log-20220905 |
sort|uniq -c|sort -n |
tail -200 > most-requested-direct-downloads-size
To get the list of most requested (unmirrored) files (attached) and
found that a caching squid frontend could provide nice bandwidth savings
for tumbleweed and update repos - but probably less than those 720 GB/d
served.
I still think that the main problem is the rsync traffic.
You can use tcpdump -nr with the /tmp/dump*.pcap files and look for
rsync port 873, which made up 89% of it.
Apart from the practical tcpdump approach, there is also the math, that
says, dividing 3000 MBit/s by 70 mirrors leaves only 43 MBit/s to each
of them.
I think, the best approach would be to find or create a few
well-connected rsync mirrors that get pushed /tumbleweed and /update
first and from there, the other mirrors can sync.
This could increase latency of updates, because it needs 2 copies to
reach most mirrors, but OTOH, the copies should happen much faster,
because more total bandwidth will be available.
Ciao
Bernhard M.
Hi,
I am intending to restart work on salting discourse, which has been
blocked for years by two things:
* Migration to vb5
* Fixing the database in vb4
I know there has been some progress on vb4 database fix, but vb5
migration seems to be blocked on auth which would likely be easier to
fix by just migrating to discourse right away, since it uses a system
we have integrated in other apps in the project already.
The problem I am facing is that since the vb4 fix isn't public right
now, any attempt at database migration testing I could perform on
discourse is not gonna be done on the same state of the database as we
would get after applying the fix.
How should we proceed then?
From my personal pov, setting up a duplicate vb4 setup and fixing a
database, so that we can properly test it and apply those fixes back
into the main instance is the best way to solve this issue. From there
we can just work on migrating to discourse, with testing that actually
tests the correct state of the database.
Alternatively I'm sure we could find a Nextcloud plugin to replace
forums with Nextcloud too, I'm sure that would solve all the
problems ;)
LCP [Sasi]
https://lcp.world/
Hello,
here are the minutes of today's meeting.
Q&A from the community
* Alex asked for access to the ticket system to help with handling
tickets
Status reports:
* lots of status updates from Per on the mailinglist:
https://lists.opensuse.org/archives/list/heroes@lists.opensuse.org/message/…
* mailman_webui often eats 100% CPU and is slow, needs debugging,
python-debugging-foo wanted
* download.o.o trouble - rsync eats too much bandwidth -> move stage.o.o
to a different host with 10Gbit link?
* can we handle GDPR requests via progress.o.o or do we have to redirect
people to the official privacy@ process? AI: Bernhard to check in SUSE
and report in October
Regards,
Christian Boltz
--
Ich sags ja immer: RfCs in Stein meisseln (5cm starke Granitplatten,
grosser Font) und diese Idioten solange damit schlagen, bis sie es
entweder kapiert haben, oder der Genpool bereinigt wurde.
[J. P. Meier in dasr]
Hello,
the next heroes meeting will be on Thursday (2022-09-01) at 18:00 UTC /
20:00 CEST in https://meet.opensuse.org/heroeshttps://progress.opensuse.org/issues/116030 already lists the usual
topics - feel free to add things you want to discuss.
Regards,
Christian Boltz
--
<tampakrap> DimStar: here?
<DimStar> tampakrap: I'm not sure - do you have good news for me ? :)
[from #opensuse-admin]
