Re: if some automated p2p method of rsync existed
Carlos E. R. wrote:
This could be improved if some automated p2p method of rsync existed to propagate the updates also from mirror to mirror. It has to be invented, AFAIK.
Somewhat offtopic, but never mind. It's been too long without a good rant :-) Nothing needs to be "invented", there are various technologies/solutions that could be applied. Distributed filesystems, some sort of internet multicast, maybe super sparrow, content distribution networks. The main hurdle - virtually regardless of which solution - is the mirror operator. She is generally happy to volunteer some Terabytes of disk space, some network bandwidth and 20mins for setting up an rsync cron-job. Some mirror operators (if you are listening here, you know who you are) are certainly very cooperative and more than happy to help out, but to most, running an openSUSE mirror is not a priority. Insignificant hurdles, in comparison - openSUSE Heroes. -- Per Jessen, Zürich (16.8°C) Member, openSUSE Heroes (2016 - present) We're hiring - https://en.opensuse.org/openSUSE:Heroes
On 2023-03-23 18:51, Per Jessen wrote:
Carlos E. R. wrote:
This could be improved if some automated p2p method of rsync existed to propagate the updates also from mirror to mirror. It has to be invented, AFAIK.
Somewhat offtopic, but never mind. It's been too long without a good rant :-)
Huh, I had not switched rant mode on today :-p
Nothing needs to be "invented", there are various technologies/solutions that could be applied. Distributed filesystems, some sort of internet multicast, maybe super sparrow, content distribution networks.
Maybe they exist, but I don't know about them. Not my league.
The main hurdle - virtually regardless of which solution - is the mirror operator. She is generally happy to volunteer some Terabytes of disk space, some network bandwidth and 20mins for setting up an rsync cron-job. Some mirror operators (if you are listening here, you know who you are) are certainly very cooperative and more than happy to help out, but to most, running an openSUSE mirror is not a priority.
But of course. That's why I said "automated". Little configuration, no maintenance.
Insignificant hurdles, in comparison - openSUSE Heroes.
The only way I know to set this up is manually. Define a few master rsync mirrors which are directly updated by "us", and the rest must be updated indirectly from those 1st level mirrors. That's too much work. -- Cheers / Saludos, Carlos E. R. (from Elesar, using openSUSE Leap 15.4)
Carlos E. R. wrote:
On 2023-03-23 18:51, Per Jessen wrote:
Carlos E. R. wrote:
This could be improved if some automated p2p method of rsync existed to propagate the updates also from mirror to mirror. It has to be invented, AFAIK.
Somewhat offtopic, but never mind. It's been too long without a good rant :-)
Huh, I had not switched rant mode on today :-p
My turn :-)
The main hurdle - virtually regardless of which solution - is the mirror operator. She is generally happy to volunteer some Terabytes of disk space, some network bandwidth and 20mins for setting up an rsync cron-job. Some mirror operators (if you are listening here, you know who you are) are certainly very cooperative and more than happy to help out, but to most, running an openSUSE mirror is not a priority.
But of course. That's why I said "automated". Little configuration, no maintenance.
That is the status quo - little configuration, minimal maintenance. I read your "automated" to mean the running (which is obvious), but I'm talking about the setting up, the configuration. I don't think that is possible. Even if we design a brand new mirroring system and package everything into a neat container, how many sysadmins are going to run stuff they don't know? Will they permit us to have access?
Insignificant hurdles, in comparison - openSUSE Heroes.
The only way I know to set this up is manually. Define a few master rsync mirrors which are directly updated by "us",
We already have that.
and the rest must be updated indirectly from those 1st level mirrors.
That is the situation today. Maybe depending on what you mean by "indirectly". -- Per Jessen, Zürich (15.3°C) Member, openSUSE Heroes (2016 - present) We're hiring - https://en.opensuse.org/openSUSE:Heroes
On 2023-03-23 20:08, Per Jessen wrote:
Carlos E. R. wrote:
On 2023-03-23 18:51, Per Jessen wrote:
Carlos E. R. wrote:
This could be improved if some automated p2p method of rsync existed to propagate the updates also from mirror to mirror. It has to be invented, AFAIK.
Somewhat offtopic, but never mind. It's been too long without a good rant :-)
Huh, I had not switched rant mode on today :-p
My turn :-)
Ah, ok :-)
The main hurdle - virtually regardless of which solution - is the mirror operator. She is generally happy to volunteer some Terabytes of disk space, some network bandwidth and 20mins for setting up an rsync cron-job. Some mirror operators (if you are listening here, you know who you are) are certainly very cooperative and more than happy to help out, but to most, running an openSUSE mirror is not a priority.
But of course. That's why I said "automated". Little configuration, no maintenance.
That is the status quo - little configuration, minimal maintenance.
Sure. But prone to overloading.
I read your "automated" to mean the running (which is obvious), but I'm talking about the setting up, the configuration. I don't think that is possible. Even if we design a brand new mirroring system and package everything into a neat container, how many sysadmins are going to run stuff they don't know? Will they permit us to have access?
I was thinking of something related to torrents and rsync. An rsync that wouldn't have a defined source or destination, but would determine which to use based on p2p network, automatically and dynamically. The openSUSE infra would do the seeding.
Insignificant hurdles, in comparison - openSUSE Heroes.
The only way I know to set this up is manually. Define a few master rsync mirrors which are directly updated by "us",
We already have that.
and the rest must be updated indirectly from those 1st level mirrors.
That is the situation today. Maybe depending on what you mean by "indirectly".
opensuse-----mirror A --> mirror 01 infra \----mirror B --> mirror 02 \---mirror C --> mirror 03 \--mirror D --> mirror 04 --> mirror 05 --> mirror 06 --> mirror 07 --> mirror 08 ... --> mirror 99 Distributed load with a tree hierarchy. The admins of mirrors A..D would have to care for their own cron scripts sending to a portion of mirrors 00 to 99. Undoable, too much work. -- Cheers / Saludos, Carlos E. R. (from Elesar, using openSUSE Leap 15.4)
participants (2)
-
Carlos E. R. -
Per Jessen