[openSUSE admin - tickets #125615] Problem with our mirror
[openSUSE Tracker] Issue #125615 has been updated by avicenzi. Assignee set to andriinikitin Hi Ariel, My theory is that we now have many mirrors in South America, more precisely, in Brazil. The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily. The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay. This leaves your mirror for most likely Uruguayan users and depending on the freshness of the data, users can still be sent somewhere else (RS, SC, or SP in Brazil due to distance) in case the files are not found in your mirror. I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh. To solve that you should sync more frequently, maybe every 2h or even a shorter period. This is just a theory for now, but Andrii may be able to dig some metrics from our servers. Best, Alex ---------------------------------------- tickets #125615: Problem with our mirror https://progress.opensuse.org/issues/125615#change-610511 * Author: asabigue@fder.edu.uy * Status: New * Priority: Normal * Assignee: andriinikitin * Category: Mirrors * Target version: ---------------------------------------- Hello! We are receiving almost no traffic to our mirror, but we don't see any trouble with the data, update scripts or other working details. I checked at https://mirrors.opensuse.org/ where we are still listed, yet, few things on green. Could you please let us know if there is something not working as expected? Our mirror is espejito.fder.edu.uy regards ariel -- You have received this notification because you have either subscribed to it, or are involved in it. To change your notification preferences, please click here: http://progress.opensuse.org/my/account
Thanks for your answer! We are refreshing every 4 hours, now we will check hourly (let's say, less than one hour after the previous rsync finishes). I know that our big brother Brasil is there, but even traffic from Uruguay is routed to Brasil... that is what I found weird. (as a matter of fact, I realized it downloading from home and being redirected to Brasil, not our mirror). Formerly we rsynced from: rsync://stage.opensuse.org/opensuse-full-really-everything-including-repositories/opensuse and found that after big updates in one branch, it took ages for other branches to become updated, thus, we broke the initial script into * rsync_opensuse_ports * rsync_opensuse_debug * rsync_opensuse_repositories * rsync_opensuse_distribution * rsync_opensuse_tumbleweed * rsync_opensuse_history * rsync_opensuse_update so each of these branches update in parallel.... but we are not syncing the root folder anymore. Do you think that it could make any trouble? thanks again ariel On 8/3/23 8:45, redmine@opensuse.org wrote:
[openSUSE Tracker] Issue #125615 has been updated by avicenzi.
Assignee set to andriinikitin
Hi Ariel,
My theory is that we now have many mirrors in South America, more precisely, in Brazil. The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily. The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay.
This leaves your mirror for most likely Uruguayan users and depending on the freshness of the data, users can still be sent somewhere else (RS, SC, or SP in Brazil due to distance) in case the files are not found in your mirror.
I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh. To solve that you should sync more frequently, maybe every 2h or even a shorter period.
This is just a theory for now, but Andrii may be able to dig some metrics from our servers.
Best,
Alex
---------------------------------------- tickets #125615: Problem with our mirror https://progress.opensuse.org/issues/125615#change-610511
* Author:asabigue@fder.edu.uy * Status: New * Priority: Normal * Assignee: andriinikitin * Category: Mirrors * Target version: ---------------------------------------- Hello!
We are receiving almost no traffic to our mirror, but we don't see any trouble with the data, update scripts or other working details. I checked athttps://mirrors.opensuse.org/ where we are still listed, yet, few things on green.
Could you please let us know if there is something not working as expected?
Our mirror is espejito.fder.edu.uy
regards
ariel
Hi Ariel, Yes, it is weird but there are some reasons for it. MirrorCache is responsible for figuring out where a user should be sent to, and MirrorCache returns a list of multiple mirrors ranked by a rule (freshness, distance, and other things). Even in Uruguay, for a given package, you may not end up in your mirror because there's a better-ranked mirror somewhere else. If the file is not found in your mirror, you need to be sent somewhere else, if there's a mirror closer to you that is not your own mirror, you also may end up in another one. There are some other cases, like how up to date, your uptime, and other things that count towards this scoring algorithm. We are also aware that even with mirrors in certain countries, like Chile, for example, people are still sent to Brazil for some requests. Now I'm also aware that Uruguay has the same problems as Chile. I've seen similar problems even in Europe. Brazil has big mirrors namely UFPR (C3SL) and now Locaweb, that have a lot of what users want and they are very much in sync with our main openSUSE servers, which contributes a lot to users from other SA countries ending up in Brazil. There are some limitations on what we can do about this, and because of such problems, Mirror Sorcerer was born [1] (community project). We have been looking into ways to improve this, but I'm not sure how far we are with the changes. We have a new rsync server, with less data but a faster network, it is stage-main-repos.opensuse.org. Last time I checked, to fully copy state-main-repos, you had to have around 5 TB of disk space, and this will cover pretty much all common needs of users. Maybe you want to switch to it, as you can sync more frequently and faster from it. I have been working on improving the quality of our infrastructure in South America, mainly in Brazil, but also in a few other countries and it is not easy to solve everything, to be honest. Brazil is a huge country, South America has terrible internet backbones, and many countries have better links to the US than other countries in SA itself, which is a shame. I don't think there's anything wrong with your setup, it is just that there are better-ranked mirrors in SA. I'll talk to Andrii and check if there's anything that we can do to improve this situation a bit. I hope that this clarifies things a bit more, and I'll try to keep you updated on any changes we introduce to improve this behavior. Best, Alex [1]: https://github.com/Firstyear/mirrorsorcerer (https://github.com/Firstyear/mirrorsorcerer) March 8, 2023 12:59 PM, "ariel sabiguero yawelak" <asabigue@fing.edu.uy (mailto:asabigue@fing.edu.uy?to=%22ariel%20sabiguero%20yawelak%22%20<asabigue@fing.edu.uy>)> wrote: Thanks for your answer! We are refreshing every 4 hours, now we will check hourly (let's say, less than one hour after the previous rsync finishes). I know that our big brother Brasil is there, but even traffic from Uruguay is routed to Brasil... that is what I found weird. (as a matter of fact, I realized it downloading from home and being redirected to Brasil, not our mirror). Formerly we rsynced from: rsync://stage.opensuse.org/opensuse-full-really-everything-including-repositories/opensuse and found that after big updates in one branch, it took ages for other branches to become updated, thus, we broke the initial script into * rsync_opensuse_ports * rsync_opensuse_debug * rsync_opensuse_repositories * rsync_opensuse_distribution * rsync_opensuse_tumbleweed * rsync_opensuse_history * rsync_opensuse_update so each of these branches update in parallel.... but we are not syncing the root folder anymore. Do you think that it could make any trouble? thanks again ariel On 8/3/23 8:45, redmine@opensuse.org (mailto:redmine@opensuse.org) wrote: [openSUSE Tracker] Issue #125615 has been updated by avicenzi. Assignee set to andriinikitin Hi Ariel, My theory is that we now have many mirrors in South America, more precisely, in Brazil. The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily. The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay. This leaves your mirror for most likely Uruguayan users and depending on the freshness of the data, users can still be sent somewhere else (RS, SC, or SP in Brazil due to distance) in case the files are not found in your mirror. I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh. To solve that you should sync more frequently, maybe every 2h or even a shorter period. This is just a theory for now, but Andrii may be able to dig some metrics from our servers. Best, Alex ---------------------------------------- tickets #125615: Problem with our mirror https://progress.opensuse.org/issues/125615#change-610511 (https://progress.opensuse.org/issues/125615#change-610511) * Author: asabigue@fder.edu.uy (mailto:asabigue@fder.edu.uy) * Status: New * Priority: Normal * Assignee: andriinikitin * Category: Mirrors * Target version: ---------------------------------------- Hello! We are receiving almost no traffic to our mirror, but we don't see any trouble with the data, update scripts or other working details. I checked at https://mirrors.opensuse.org/ (https://mirrors.opensuse.org/) where we are still listed, yet, few things on green. Could you please let us know if there is something not working as expected? Our mirror is espejito.fder.edu.uy regards ariel
I -almost- completely agree with your description... yet, it seems strange that not even traffic from the same AS is redirected there. I even understand that different carriers from Uruguay may get a different mirror. obrigado ariel On 8/3/23 13:47, alexandre@opensuse.net.br wrote:
Hi Ariel,
Yes, it is weird but there are some reasons for it.
MirrorCache is responsible for figuring out where a user should be sent to, and MirrorCache returns a list of multiple mirrors ranked by a rule (freshness, distance, and other things). Even in Uruguay, for a given package, you may not end up in your mirror because there's a better-ranked mirror somewhere else.
If the file is not found in your mirror, you need to be sent somewhere else, if there's a mirror closer to you that is not your own mirror, you also may end up in another one. There are some other cases, like how up to date, your uptime, and other things that count towards this scoring algorithm.
We are also aware that even with mirrors in certain countries, like Chile, for example, people are still sent to Brazil for some requests. Now I'm also aware that Uruguay has the same problems as Chile. I've seen similar problems even in Europe.
Brazil has big mirrors namely UFPR (C3SL) and now Locaweb, that have a lot of what users want and they are very much in sync with our main openSUSE servers, which contributes a lot to users from other SA countries ending up in Brazil.
There are some limitations on what we can do about this, and because of such problems, Mirror Sorcerer was born [1] (community project). We have been looking into ways to improve this, but I'm not sure how far we are with the changes.
We have a new rsync server, with less data but a faster network, it is stage-main-repos.opensuse.org. Last time I checked, to fully copy state-main-repos, you had to have around 5 TB of disk space, and this will cover pretty much all common needs of users. Maybe you want to switch to it, as you can sync more frequently and faster from it.
I have been working on improving the quality of our infrastructure in South America, mainly in Brazil, but also in a few other countries and it is not easy to solve everything, to be honest. Brazil is a huge country, South America has terrible internet backbones, and many countries have better links to the US than other countries in SA itself, which is a shame.
I don't think there's anything wrong with your setup, it is just that there are better-ranked mirrors in SA. I'll talk to Andrii and check if there's anything that we can do to improve this situation a bit.
I hope that this clarifies things a bit more, and I'll try to keep you updated on any changes we introduce to improve this behavior.
Best,
Alex
[1]: https://github.com/Firstyear/mirrorsorcerer
March 8, 2023 12:59 PM, "ariel sabiguero yawelak" <asabigue@fing.edu.uy <mailto:asabigue@fing.edu.uy?to=%22ariel%20sabiguero%20yawelak%22%20<asabigue@fing.edu.uy>>> wrote:
Thanks for your answer!
We are refreshing every 4 hours, now we will check hourly (let's say, less than one hour after the previous rsync finishes).
I know that our big brother Brasil is there, but even traffic from Uruguay is routed to Brasil... that is what I found weird. (as a matter of fact, I realized it downloading from home and being redirected to Brasil, not our mirror).
Formerly we rsynced from:
rsync://stage.opensuse.org/opensuse-full-really-everything-including-repositories/opensuse
and found that after big updates in one branch, it took ages for other branches to become updated, thus, we broke the initial script into
* rsync_opensuse_ports * rsync_opensuse_debug * rsync_opensuse_repositories * rsync_opensuse_distribution * rsync_opensuse_tumbleweed * rsync_opensuse_history * rsync_opensuse_update
so each of these branches update in parallel.... but we are not syncing the root folder anymore. Do you think that it could make any trouble?
thanks again
ariel
On 8/3/23 8:45, redmine@opensuse.org wrote:
[openSUSE Tracker] Issue #125615 has been updated by avicenzi. Assignee set to andriinikitin Hi Ariel, My theory is that we now have many mirrors in South America, more precisely, in Brazil. The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily. The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay. This leaves your mirror for most likely Uruguayan users and depending on the freshness of the data, users can still be sent somewhere else (RS, SC, or SP in Brazil due to distance) in case the files are not found in your mirror. I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh. To solve that you should sync more frequently, maybe every 2h or even a shorter period. This is just a theory for now, but Andrii may be able to dig some metrics from our servers. Best, Alex ---------------------------------------- tickets #125615: Problem with our mirrorhttps://progress.opensuse.org/issues/125615#change-610511 * Author:asabigue@fder.edu.uy * Status: New * Priority: Normal * Assignee: andriinikitin * Category: Mirrors * Target version: ---------------------------------------- Hello! We are receiving almost no traffic to our mirror, but we don't see any trouble with the data, update scripts or other working details. I checked athttps://mirrors.opensuse.org/ where we are still listed, yet, few things on green. Could you please let us know if there is something not working as expected? Our mirror is espejito.fder.edu.uy regards ariel
Hi, There could be an issue, can you test this for me: curl -L https://download.opensuse.org/tumbleweed/iso/README.FIRST.metalink curl -L https://mirrorcache-br.opensuse.org/tumbleweed/iso/README.FIRST.metalink curl -L https://mirrorcache-br-2.opensuse.org/tumbleweed/iso/README.FIRST.metalink curl -L https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-... curl -L https://mirrorcache-br.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap... curl -L https://mirrorcache-br-2.opensuse.org/distribution/leap/15.4/iso/openSUSE-Le... You can replace the part in the URL with any files on your server. Your mirror should appear below: <!-- Mirrors which handle this country (UY): --> I tried here, and it does not return any mirrors for Uruguay. In this case, it could mean that MirrorCache is not able to scan your mirror. I don't have any other information for now, but we will look into it to identify what can be the cause. Best, Alex March 8, 2023 6:14 PM, "ariel sabiguero yawelak" <asabigue@fing.edu.uy (mailto:asabigue@fing.edu.uy?to=%22ariel%20sabiguero%20yawelak%22%20<asabigue@fing.edu.uy>)> wrote: I -almost- completely agree with your description... yet, it seems strange that not even traffic from the same AS is redirected there. I even understand that different carriers from Uruguay may get a different mirror. obrigado ariel On 8/3/23 13:47, alexandre@opensuse.net.br (mailto:alexandre@opensuse.net.br) wrote: Hi Ariel, Yes, it is weird but there are some reasons for it. MirrorCache is responsible for figuring out where a user should be sent to, and MirrorCache returns a list of multiple mirrors ranked by a rule (freshness, distance, and other things). Even in Uruguay, for a given package, you may not end up in your mirror because there's a better-ranked mirror somewhere else. If the file is not found in your mirror, you need to be sent somewhere else, if there's a mirror closer to you that is not your own mirror, you also may end up in another one. There are some other cases, like how up to date, your uptime, and other things that count towards this scoring algorithm. We are also aware that even with mirrors in certain countries, like Chile, for example, people are still sent to Brazil for some requests. Now I'm also aware that Uruguay has the same problems as Chile. I've seen similar problems even in Europe. Brazil has big mirrors namely UFPR (C3SL) and now Locaweb, that have a lot of what users want and they are very much in sync with our main openSUSE servers, which contributes a lot to users from other SA countries ending up in Brazil. There are some limitations on what we can do about this, and because of such problems, Mirror Sorcerer was born [1] (community project). We have been looking into ways to improve this, but I'm not sure how far we are with the changes. We have a new rsync server, with less data but a faster network, it is stage-main-repos.opensuse.org. Last time I checked, to fully copy state-main-repos, you had to have around 5 TB of disk space, and this will cover pretty much all common needs of users. Maybe you want to switch to it, as you can sync more frequently and faster from it. I have been working on improving the quality of our infrastructure in South America, mainly in Brazil, but also in a few other countries and it is not easy to solve everything, to be honest. Brazil is a huge country, South America has terrible internet backbones, and many countries have better links to the US than other countries in SA itself, which is a shame. I don't think there's anything wrong with your setup, it is just that there are better-ranked mirrors in SA. I'll talk to Andrii and check if there's anything that we can do to improve this situation a bit. I hope that this clarifies things a bit more, and I'll try to keep you updated on any changes we introduce to improve this behavior. Best, Alex [1]: https://github.com/Firstyear/mirrorsorcerer (https://github.com/Firstyear/mirrorsorcerer) March 8, 2023 12:59 PM, "ariel sabiguero yawelak" <asabigue@fing.edu.uy (mailto:asabigue@fing.edu.uy?to=%22ariel%20sabiguero%20yawelak%22%20<asabigue@fing.edu.uy>)> wrote: Thanks for your answer! We are refreshing every 4 hours, now we will check hourly (let's say, less than one hour after the previous rsync finishes). I know that our big brother Brasil is there, but even traffic from Uruguay is routed to Brasil... that is what I found weird. (as a matter of fact, I realized it downloading from home and being redirected to Brasil, not our mirror). Formerly we rsynced from: rsync://stage.opensuse.org/opensuse-full-really-everything-including-repositories/opensuse and found that after big updates in one branch, it took ages for other branches to become updated, thus, we broke the initial script into * rsync_opensuse_ports * rsync_opensuse_debug * rsync_opensuse_repositories * rsync_opensuse_distribution * rsync_opensuse_tumbleweed * rsync_opensuse_history * rsync_opensuse_update so each of these branches update in parallel.... but we are not syncing the root folder anymore. Do you think that it could make any trouble? thanks again ariel On 8/3/23 8:45, redmine@opensuse.org (mailto:redmine@opensuse.org) wrote: [openSUSE Tracker] Issue #125615 has been updated by avicenzi. Assignee set to andriinikitin Hi Ariel, My theory is that we now have many mirrors in South America, more precisely, in Brazil. The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily. The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay. This leaves your mirror for most likely Uruguayan users and depending on the freshness of the data, users can still be sent somewhere else (RS, SC, or SP in Brazil due to distance) in case the files are not found in your mirror. I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh. To solve that you should sync more frequently, maybe every 2h or even a shorter period. This is just a theory for now, but Andrii may be able to dig some metrics from our servers. Best, Alex ---------------------------------------- tickets #125615: Problem with our mirror https://progress.opensuse.org/issues/125615#change-610511 (https://progress.opensuse.org/issues/125615#change-610511) * Author: asabigue@fder.edu.uy (mailto:asabigue@fder.edu.uy) * Status: New * Priority: Normal * Assignee: andriinikitin * Category: Mirrors * Target version: ---------------------------------------- Hello! We are receiving almost no traffic to our mirror, but we don't see any trouble with the data, update scripts or other working details. I checked at https://mirrors.opensuse.org/ (https://mirrors.opensuse.org/) where we are still listed, yet, few things on green. Could you please let us know if there is something not working as expected? Our mirror is espejito.fder.edu.uy regards ariel
No metalink files in our server..... espejito:/zp0/mirrors/opensuse/opensuse/opensuse # find | grep metalink$ espejito:/zp0/mirrors/opensuse/opensuse/opensuse # and I am up-to date: espejito:~ # cd /zp0/mirrors/opensuse espejito:/zp0/mirrors/opensuse # sudo -u mirror rsync -4rlpt rsync://stage.opensuse.org/opensuse-full-really-everything-including-repositories/opensuse/distribution/ opensuse/opensuse/distribution --delete-after --delete-excluded --timeout=1800 -hi This is stage.opensuse.org, restricted rsync server of openSUSE.org. If you run a public mirror, please get in contact so we can give you access to the this rsync server. You'll find conditions for access and further information at http://en.opensuse.org/Mirror_Infrastructure Thanks! admin@opensuse.org espejito:/zp0/mirrors/opensuse # am I rsync-ing from a wrong source? On 8/3/23 15:51, alexandre@opensuse.net.br wrote:
Hi,
There could be an issue, can you test this for me:
curl -L https://download.opensuse.org/tumbleweed/iso/README.FIRST.metalink curl -L https://mirrorcache-br.opensuse.org/tumbleweed/iso/README.FIRST.metalink curl -L https://mirrorcache-br-2.opensuse.org/tumbleweed/iso/README.FIRST.metalink
curl -L https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-... curl -L https://mirrorcache-br.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap... curl -L https://mirrorcache-br-2.opensuse.org/distribution/leap/15.4/iso/openSUSE-Le...
You can replace the part in the URL with any files on your server.
Your mirror should appear below: <!-- Mirrors which handle this country (UY): -->
I tried here, and it does not return any mirrors for Uruguay. In this case, it could mean that MirrorCache is not able to scan your mirror. I don't have any other information for now, but we will look into it to identify what can be the cause.
Best,
Alex
March 8, 2023 6:14 PM, "ariel sabiguero yawelak" <asabigue@fing.edu.uy <mailto:asabigue@fing.edu.uy?to=%22ariel%20sabiguero%20yawelak%22%20<asabigue@fing.edu.uy>>> wrote:
I -almost- completely agree with your description... yet, it seems strange that not even traffic from the same AS is redirected there. I even understand that different carriers from Uruguay may get a different mirror.
obrigado
ariel
On 8/3/23 13:47, alexandre@opensuse.net.br wrote:
Hi Ariel,
Yes, it is weird but there are some reasons for it.
MirrorCache is responsible for figuring out where a user should be sent to, and MirrorCache returns a list of multiple mirrors ranked by a rule (freshness, distance, and other things). Even in Uruguay, for a given package, you may not end up in your mirror because there's a better-ranked mirror somewhere else.
If the file is not found in your mirror, you need to be sent somewhere else, if there's a mirror closer to you that is not your own mirror, you also may end up in another one. There are some other cases, like how up to date, your uptime, and other things that count towards this scoring algorithm.
We are also aware that even with mirrors in certain countries, like Chile, for example, people are still sent to Brazil for some requests. Now I'm also aware that Uruguay has the same problems as Chile. I've seen similar problems even in Europe.
Brazil has big mirrors namely UFPR (C3SL) and now Locaweb, that have a lot of what users want and they are very much in sync with our main openSUSE servers, which contributes a lot to users from other SA countries ending up in Brazil.
There are some limitations on what we can do about this, and because of such problems, Mirror Sorcerer was born [1] (community project). We have been looking into ways to improve this, but I'm not sure how far we are with the changes.
We have a new rsync server, with less data but a faster network, it is stage-main-repos.opensuse.org. Last time I checked, to fully copy state-main-repos, you had to have around 5 TB of disk space, and this will cover pretty much all common needs of users. Maybe you want to switch to it, as you can sync more frequently and faster from it.
I have been working on improving the quality of our infrastructure in South America, mainly in Brazil, but also in a few other countries and it is not easy to solve everything, to be honest. Brazil is a huge country, South America has terrible internet backbones, and many countries have better links to the US than other countries in SA itself, which is a shame.
I don't think there's anything wrong with your setup, it is just that there are better-ranked mirrors in SA. I'll talk to Andrii and check if there's anything that we can do to improve this situation a bit.
I hope that this clarifies things a bit more, and I'll try to keep you updated on any changes we introduce to improve this behavior.
Best,
Alex
[1]: https://github.com/Firstyear/mirrorsorcerer
March 8, 2023 12:59 PM, "ariel sabiguero yawelak" <asabigue@fing.edu.uy <mailto:asabigue@fing.edu.uy?to=%22ariel%20sabiguero%20yawelak%22%20<asabigue@fing.edu.uy>>> wrote:
Thanks for your answer!
We are refreshing every 4 hours, now we will check hourly (let's say, less than one hour after the previous rsync finishes).
I know that our big brother Brasil is there, but even traffic from Uruguay is routed to Brasil... that is what I found weird. (as a matter of fact, I realized it downloading from home and being redirected to Brasil, not our mirror).
Formerly we rsynced from:
rsync://stage.opensuse.org/opensuse-full-really-everything-including-repositories/opensuse
and found that after big updates in one branch, it took ages for other branches to become updated, thus, we broke the initial script into
* rsync_opensuse_ports * rsync_opensuse_debug * rsync_opensuse_repositories * rsync_opensuse_distribution * rsync_opensuse_tumbleweed * rsync_opensuse_history * rsync_opensuse_update
so each of these branches update in parallel.... but we are not syncing the root folder anymore. Do you think that it could make any trouble?
thanks again
ariel
On 8/3/23 8:45, redmine@opensuse.org wrote:
[openSUSE Tracker] Issue #125615 has been updated by avicenzi. Assignee set to andriinikitin Hi Ariel, My theory is that we now have many mirrors in South America, more precisely, in Brazil. The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily. The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay. This leaves your mirror for most likely Uruguayan users and depending on the freshness of the data, users can still be sent somewhere else (RS, SC, or SP in Brazil due to distance) in case the files are not found in your mirror. I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh. To solve that you should sync more frequently, maybe every 2h or even a shorter period. This is just a theory for now, but Andrii may be able to dig some metrics from our servers. Best, Alex ---------------------------------------- tickets #125615: Problem with our mirrorhttps://progress.opensuse.org/issues/125615#change-610511 * Author:asabigue@fder.edu.uy * Status: New * Priority: Normal * Assignee: andriinikitin * Category: Mirrors * Target version: ---------------------------------------- Hello! We are receiving almost no traffic to our mirror, but we don't see any trouble with the data, update scripts or other working details. I checked athttps://mirrors.opensuse.org/ where we are still listed, yet, few things on green. Could you please let us know if there is something not working as expected? Our mirror is espejito.fder.edu.uy regards ariel
Hola Ariel, Metalink is not a file in your server, but a URL on MirrorCache (our mirror scanner and metadata server). Metalink returns the metadata from a given file, and this metadata returns the list of mirrors available. Run the command: curl -L https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-... (https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-...) You can replace distribution/leap/15.4/iso/openSUSE-Leap-15.4-DVD-aarch64-Media.iso with any files on your server. I made a few tests, and today, a few files are returning from your mirror when I ask for Uruguay. So, it seems that our scanner is picking up your mirror, but it may take some time to find all files on your server. About the rsync error, stage.opensuse.org requires registration. We need to release your IP address. If you want access to it, send an email to admin [at] opensuse [dot] org. More details here: https://en.opensuse.org/openSUSE:Mirror_infrastructure#Registering_your_mirr... (https://en.opensuse.org/openSUSE:Mirror_infrastructure#Registering_your_mirr...) You can switch stage.opensuse.org with stage-main-repos.opensuse.org stage-main-repos.opensuse.org does not require registration. Best, Alex March 8, 2023 9:24 PM, "ariel sabiguero yawelak" <asabigue@fing.edu.uy (mailto:asabigue@fing.edu.uy?to=%22ariel%20sabiguero%20yawelak%22%20<asabigue@fing.edu.uy>)> wrote: No metalink files in our server..... espejito:/zp0/mirrors/opensuse/opensuse/opensuse # find | grep metalink$ espejito:/zp0/mirrors/opensuse/opensuse/opensuse # and I am up-to date: espejito:~ # cd /zp0/mirrors/opensuse espejito:/zp0/mirrors/opensuse # sudo -u mirror rsync -4rlpt rsync://stage.opensuse.org/opensuse-full-really-everything-including-repositories/opensuse/distribution/ opensuse/opensuse/distribution --delete-after --delete-excluded --timeout=1800 -hi This is stage.opensuse.org, restricted rsync server of openSUSE.org. If you run a public mirror, please get in contact so we can give you access to the this rsync server. You'll find conditions for access and further information at http://en.opensuse.org/Mirror_Infrastructure (http://en.opensuse.org/Mirror_Infrastructure) Thanks! admin@opensuse.org (mailto:admin@opensuse.org) espejito:/zp0/mirrors/opensuse # am I rsync-ing from a wrong source? On 8/3/23 15:51, alexandre@opensuse.net.br (mailto:alexandre@opensuse.net.br) wrote: Hi, There could be an issue, can you test this for me: curl -L https://download.opensuse.org/tumbleweed/iso/README.FIRST.metalink (https://download.opensuse.org/tumbleweed/iso/README.FIRST.metalink) curl -L https://mirrorcache-br.opensuse.org/tumbleweed/iso/README.FIRST.metalink (https://mirrorcache-br.opensuse.org/tumbleweed/iso/README.FIRST.metalink) curl -L https://mirrorcache-br-2.opensuse.org/tumbleweed/iso/README.FIRST.metalink (https://mirrorcache-br-2.opensuse.org/tumbleweed/iso/README.FIRST.metalink) curl -L https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-... (https://download.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap-15.4-...) curl -L https://mirrorcache-br.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap... (https://mirrorcache-br.opensuse.org/distribution/leap/15.4/iso/openSUSE-Leap...) curl -L https://mirrorcache-br-2.opensuse.org/distribution/leap/15.4/iso/openSUSE-Le... (https://mirrorcache-br-2.opensuse.org/distribution/leap/15.4/iso/openSUSE-Le...) You can replace the part in the URL with any files on your server. Your mirror should appear below: <!-- Mirrors which handle this country (UY): --> I tried here, and it does not return any mirrors for Uruguay. In this case, it could mean that MirrorCache is not able to scan your mirror. I don't have any other information for now, but we will look into it to identify what can be the cause. Best, Alex March 8, 2023 6:14 PM, "ariel sabiguero yawelak" <asabigue@fing.edu.uy (mailto:asabigue@fing.edu.uy?to=%22ariel%20sabiguero%20yawelak%22%20<asabigue@fing.edu.uy>)> wrote: I -almost- completely agree with your description... yet, it seems strange that not even traffic from the same AS is redirected there. I even understand that different carriers from Uruguay may get a different mirror. obrigado ariel On 8/3/23 13:47, alexandre@opensuse.net.br (mailto:alexandre@opensuse.net.br) wrote: Hi Ariel, Yes, it is weird but there are some reasons for it. MirrorCache is responsible for figuring out where a user should be sent to, and MirrorCache returns a list of multiple mirrors ranked by a rule (freshness, distance, and other things). Even in Uruguay, for a given package, you may not end up in your mirror because there's a better-ranked mirror somewhere else. If the file is not found in your mirror, you need to be sent somewhere else, if there's a mirror closer to you that is not your own mirror, you also may end up in another one. There are some other cases, like how up to date, your uptime, and other things that count towards this scoring algorithm. We are also aware that even with mirrors in certain countries, like Chile, for example, people are still sent to Brazil for some requests. Now I'm also aware that Uruguay has the same problems as Chile. I've seen similar problems even in Europe. Brazil has big mirrors namely UFPR (C3SL) and now Locaweb, that have a lot of what users want and they are very much in sync with our main openSUSE servers, which contributes a lot to users from other SA countries ending up in Brazil. There are some limitations on what we can do about this, and because of such problems, Mirror Sorcerer was born [1] (community project). We have been looking into ways to improve this, but I'm not sure how far we are with the changes. We have a new rsync server, with less data but a faster network, it is stage-main-repos.opensuse.org. Last time I checked, to fully copy state-main-repos, you had to have around 5 TB of disk space, and this will cover pretty much all common needs of users. Maybe you want to switch to it, as you can sync more frequently and faster from it. I have been working on improving the quality of our infrastructure in South America, mainly in Brazil, but also in a few other countries and it is not easy to solve everything, to be honest. Brazil is a huge country, South America has terrible internet backbones, and many countries have better links to the US than other countries in SA itself, which is a shame. I don't think there's anything wrong with your setup, it is just that there are better-ranked mirrors in SA. I'll talk to Andrii and check if there's anything that we can do to improve this situation a bit. I hope that this clarifies things a bit more, and I'll try to keep you updated on any changes we introduce to improve this behavior. Best, Alex [1]: https://github.com/Firstyear/mirrorsorcerer (https://github.com/Firstyear/mirrorsorcerer) March 8, 2023 12:59 PM, "ariel sabiguero yawelak" <asabigue@fing.edu.uy (mailto:asabigue@fing.edu.uy?to=%22ariel%20sabiguero%20yawelak%22%20<asabigue@fing.edu.uy>)> wrote: Thanks for your answer! We are refreshing every 4 hours, now we will check hourly (let's say, less than one hour after the previous rsync finishes). I know that our big brother Brasil is there, but even traffic from Uruguay is routed to Brasil... that is what I found weird. (as a matter of fact, I realized it downloading from home and being redirected to Brasil, not our mirror). Formerly we rsynced from: rsync://stage.opensuse.org/opensuse-full-really-everything-including-repositories/opensuse and found that after big updates in one branch, it took ages for other branches to become updated, thus, we broke the initial script into * rsync_opensuse_ports * rsync_opensuse_debug * rsync_opensuse_repositories * rsync_opensuse_distribution * rsync_opensuse_tumbleweed * rsync_opensuse_history * rsync_opensuse_update so each of these branches update in parallel.... but we are not syncing the root folder anymore. Do you think that it could make any trouble? thanks again ariel On 8/3/23 8:45, redmine@opensuse.org (mailto:redmine@opensuse.org) wrote: [openSUSE Tracker] Issue #125615 has been updated by avicenzi. Assignee set to andriinikitin Hi Ariel, My theory is that we now have many mirrors in South America, more precisely, in Brazil. The requests are sent to the closest mirror based on distance (lat/long) from IPs, so, as Brazil is the biggest country and has the most openSUSE users, requests are not leaving the country that easily. The south of Brazil is now covered by a mirror in Porto Alegre (RS) and Florianópolis (SC) before these users could go to São Paulo or even Uruguay. This leaves your mirror for most likely Uruguayan users and depending on the freshness of the data, users can still be sent somewhere else (RS, SC, or SP in Brazil due to distance) in case the files are not found in your mirror. I also noticed that many users are on Tumbleweed in SA, and your mirror is red because the data is not fresh. To solve that you should sync more frequently, maybe every 2h or even a shorter period. This is just a theory for now, but Andrii may be able to dig some metrics from our servers. Best, Alex ---------------------------------------- tickets #125615: Problem with our mirror https://progress.opensuse.org/issues/125615#change-610511 (https://progress.opensuse.org/issues/125615#change-610511) * Author: asabigue@fder.edu.uy (mailto:asabigue@fder.edu.uy) * Status: New * Priority: Normal * Assignee: andriinikitin * Category: Mirrors * Target version: ---------------------------------------- Hello! We are receiving almost no traffic to our mirror, but we don't see any trouble with the data, update scripts or other working details. I checked at https://mirrors.opensuse.org/ (https://mirrors.opensuse.org/) where we are still listed, yet, few things on green. Could you please let us know if there is something not working as expected? Our mirror is espejito.fder.edu.uy regards ariel
participants (3)
-
alexandre＠opensuse.net.br -
ariel sabiguero yawelak -
redmine＠opensuse.org