Hi Neal, Correct me if I am wrong - MirrorManager expects all mirrors have the same content, while MirrorBrain and MirrorCache tolerate divergents and are able to use from them whatever is available. Actually MirrorCache does support metalinks, just the core problem is that it doesn't have access to exact file sizes yet. You can try e.g. : curl -H 'Accept: */*, application/metalink+xml' -s https://mirrorcache-eu.opensuse.org/download/tumbleweed/repo/oss/x86_64/2048... Regards, Andrii Nikitin ________________________________ From: Neal Gompa This looks very cool, but I'm confused why we would do this instead of using MirrorManager2[1] and collaborating on that project? The documentation showing the differences between MirrorCache and MirrorBrain seem to mirror the differences between MirrorManager2 and MirrorBrain based on my memory of discussing this at oSC two years ago. Moreover, MirrorManager2 already has self-service management of mirrors. The main difference between MirrorManager2 and MirrorCache seems to be that MirrorCache lacks metalink support, which I think we would actually *want* to have. Are there any other specific features that MirrorCache has that I'm missing? [1]: https://github.com/fedora-infra/mirrormanager2/ -- 真実はいつも一つ！/ Always, there's only one truth!

Neal Gompa wrote:

This looks very cool, but I'm confused why we would do this instead of using MirrorManager2[1] and collaborating on that project?

I agree it sounds cool, but I remain confused about the purpose of mirrorcache, as well as why the effort wasn't directed at mirrorbrain instead. MirrorBrain currently has only two real shortcomings - IPv6 and https support. Fixing these as well as other more or less imagined flaws with a bolt-on just sounds like a kludge to me. The issue of clients in North America being redirected to other continents really only applies to the OBS repositories for which we simply don't have enough mirrors in the Americas. (one was recently been added in Uruguay though). -- Per Jessen, Zürich (5.0°C) Member, openSUSE Heroes

Neal Gompa wrote:

On Fri, Mar 5, 2021 at 8:05 AM Per Jessen wrote:

...

Neal Gompa wrote:

...

This looks very cool, but I'm confused why we would do this instead of using MirrorManager2[1] and collaborating on that project?

I agree it sounds cool, but I remain confused about the purpose of mirrorcache, as well as why the effort wasn't directed at mirrorbrain instead. MirrorBrain currently has only two real shortcomings - IPv6 and https support. Fixing these as well as other more or less imagined flaws with a bolt-on just sounds like a kludge to me.

The issue of clients in North America being redirected to other continents really only applies to the OBS repositories for which we simply don't have enough mirrors in the Americas. (one was recently been added in Uruguay though).

It definitely applies to openSUSE as a whole. I *rarely* get directed to an American mirror, I usually get something in Eastern Europe or Asia, which is frustrating because they often time out.

I guess we could do with a ticket for that - we have plenty of mirrors in the US, thirty of them or thereabouts. If you are frequently directed to another continent for non-OBS downloads, something is wrong with the geo-location. Seriously, open a ticket and include your typical IP-address. Or maybe just post a mirrorlist where this can be seen? -- Per Jessen, Zürich (5.1°C) Member, openSUSE Heroes

Andrii Nikitin wrote:

From: Per Jessen

...

I agree it sounds cool, but I remain confused about the purpose of mirrorcache, as well as why the effort wasn't directed at mirrorbrain instead. MirrorBrain currently has only two real shortcomings - IPv6 and https support. Fixing these as well as other more or less imagined flaws with a bolt-on just sounds like a kludge to me.

I did try to do it in MirrorBrain e.g. see https://github.com/openSUSE/mirrorbrain/pull/43 But it didn't come through, I'd say because nobody had enough drive and resources and good understanding how it can be achieved.

That is certainly a shame.

In my understanding MirrorBrain is lacking proper job queue and adding it + WebUI would result in managing two different projects inside single github repository. This is why we have two projects. I would be glad to do full help if somebody had a good vision how to fix it in mirrorbrain.

Thinking out loud - the two key shortcomings, https and ipv6 - are simply two more criteria for the mirror selection, nothing else. That sounds to me like some database changes for storing the mirror attributes, plus the appropriate changes to the selection algorithm.

...

The issue of clients in North America being redirected to other continents really only applies to the OBS repositories for which we simply don't have enough mirrors in the Americas. (one was recently been added in Uruguay though).

We may have geo database outdated,

The Maxmind database is refreshed every Monday morning, I did the latest fix to the new maxmind interface myself, in 2019. It is worth checking up on though. -- Per Jessen, Zürich (5.8°C) Member, openSUSE Heroes

Andrii Nikitin wrote:

...

...

In my understanding MirrorBrain is lacking proper job queue and adding it + WebUI would result in managing two different projects inside single github repository. This is why we have two projects. I would be glad to do full help if somebody had a good vision how to fix it in mirrorbrain.

...

Thinking out loud - the two key shortcomings, https and ipv6 - are simply two more criteria for the mirror selection, nothing else. That sounds to me like some database changes for storing the mirror attributes, plus the appropriate changes to the selection algorithm.

If we implement it properly, we must regularly probe every mirror for each attribute: http, https, ipv4 and ipv6 (because certificates are added, expire, network changes, etc).

Without having thought it through - isn't that pretty much what the scanner does today? it checks a mirror to see what is available, if an address is bad or a certificate doesn't work, the availability is updated. A poorly configured mirror should not interfere with the proper working of the rest of mirrorbrain. It would however be nice to be a little more fine-grained and record which parts of a mirror are working: ipv4, http ipv4, https ipv6, http ipv6, https In comparison to the current setup, this is just like having four mirrors instead of one. Maybe that is the way to look at the design.

And then we must track that properly and consider when matching client request. This is actually what the PR does.

The scanning, the status keeping and the selection algorithm?

But MirrorBrain also has locking problems when two scanners scan the same folder on the same or differet mirrors, which is hard to track and troubleshoot.

We scan regularly on olaf, and pontifex scans an individual mirror after each push of OBS repos. New mirrors are scanned before we enable them. I have not looked at the scanning setup in any detail, but is there any significant risk of two scanners colliding ? If these locking problems occur frequently, we have to look at that, agreed. -- Per Jessen, Zürich (4.8°C) Member, openSUSE Heroes

Per Jessen wrote:

Without having thought it through - isn't that pretty much what the scanner does today? it checks a mirror to see what is available, if an address is bad or a certificate doesn't work, the availability is updated. A poorly configured mirror should not interfere with the proper working of the rest of mirrorbrain.

It would however be nice to be a little more fine-grained and record which parts of a mirror are working:

ipv4, http ipv4, https ipv6, http ipv6, https

In comparison to the current setup, this is just like having four mirrors instead of one. Maybe that is the way to look at the design.

Correction - no, of course not, we already keep track of different access ways for each mirror - rsync, ftp, http. Maybe just add some more variations? -- Per Jessen, Zürich (4.2°C) Member, openSUSE Heroes

From: Arjen de Korte

...

We may have geo database outdated, MirrorCache has a route to check that: http://mirrorcache.opensuse.org/rest/myip

That puts my (static) IP in Moldova instead of The Netherlands, so I assume the GeoIP database used is somewhat outdated. Which brings me

I've replied privately to address this.

to the request that it would be really nice to be able to override the GeoIP location by setting an environment variable in case the location is really far off.

That is probably question for zypper team, I will try to follow up. E.g. for direct MirrorCache (or MirrorBrain) requests you can overwrite IP with X-Forwarded-For header, e.g.: curl -H 'X-Forwarded-For: 142.250.185.206' http://mirrorcache.opensuse.org/rest/myip {"country":"US","ip":"142.250.185.206","region":"NA"} Regards, Andrii Nikitin

Citeren Per Jessen :

Arjen de Korte wrote:

...

Citeren Andrii Nikitin :

...

We may have geo database outdated, MirrorCache has a route to check that: http://mirrorcache.opensuse.org/rest/myip

That puts my (static) IP in Moldova instead of The Netherlands, so I assume the GeoIP database used is somewhat outdated. Which brings me to the request that it would be really nice to be able to override the GeoIP location by setting an environment variable in case the location is really far off.

In fact you can, but in the Netherlands you should hardly have an issue?

I wish that was true. A few years ago, 'zypper' would consistently locate my IPv6 address in the US. I only noticed this after a while because of the horrendous latency and terrible speed. I had to disable IPv6 before runnning updates.

zypper probably won't support this though, but never mind:

wget --header='X-Forwarded-For: 1.2.3.4,your.own.ip.addr" \ http://download.o.o/something/something/item.mirrorlist.

1.2.3.4 is the IP you wish to use instead of your own.

As far as I know, 'zypper' defaults to using '(lib)curl', so I doubt this will work. But it has been fixed in the meantime by loading a newer GeoIP database. Which is the problem with many GeoIP services: stale databases.

On 05/03/2021 14.05, Per Jessen wrote:

The issue of clients in North America being redirected to other continents really only applies to the OBS repositories for which we simply don't have enough mirrors in the Americas. (one was recently been added in Uruguay though).

The issue is also with latency. US clients always go to download.o.o that is in NUE and then get redirected to the actual mirror. If we wanted to have a mirrorbrain server in the US, we would need another 20TB disk space to mirror all of published OBS - and would need the bandwidth to keep it updated. Also, mirrorbrain is written in C as an apache module and the original author is not coding anymore. So not easy to maintain.