Mailinglist Archive: mirror (8 mails)

< Previous Next >
Re: [suse-mirror] rsync module for the sources - now there
On Mon, Mar 30, 2009 at 02:45:22PM -0300, Carlos Carvalho wrote:
Peter Poeml (poeml@xxxxxxx) wrote on 30 March 2009 17:02:
>So, I have created opensuse-source as rsync module at
>rsync://stage.opensuse.org/opensuse-source now.

It's rather cumbersome to have to do separate syncs for parts of the
same repository.

Yes, I see that. I am not sure though if it would be better if there was
only one rsync module for the entire tree, because, you would still need
to set up different syncs, because there are parts of the tree that
change frequently (updates) and other parts that change nearly never
(released products). It wouldn't make sense to sync the released
products every four hours, and in addition to that, we would not be able
to deal with this, with our resources. Or do you have a different
opinion?

A trigger-based sync mechanism might be a way around this. I have some
things in mind, and know some ways how other projects deal with this,
but other than ideas there is not much resources to work on this.

Maybe you have ideas to contribute on this, examples how others manage
to do better, and such. Please continue to let me/us know.

Priority is not measured only by downloads, importance also counts,
and sources are essential for free software. That's why it should be
possible to mirror them together.

I might miss the context that you refer to, but what do you mean by
"Priority" here? And "importance"?

By the way, maybe you also followed the argument about the openSUSE
build service for source access. It is the "modern", convenient and
powerful way to access the sources, for openSUSE 11.1 onwards. We
basically need source RPMs right now only for those people who don't
know that yet, and because the build service doesn't allow anonymous
access yet. People tend to be irritated when they don't find the source
packages and think that something is wrong, even though it isn't.

I checked the download numbers of today, and the handful of people that
have downloaded a source package today is neglectable; so it doesn't
make sense to put this stuff out to mirrors.
Anyway, sources are sensitive issues; and the tree is available now for
everybody who's interested!

>> stage.opensuse.org should have it all, have it all in place, and give
>> access to the topmost point.

Yes.

>Yes, in a way this would make a lot things easier. I appreciate your
>thoughts. I lack the time to really do something about this right now,
>but I'll keep it noted.

One possibility that seems easy enough is to include sources in the
larger modules, like full-with-factory, in stage. Looks like just
removing some exclusions would do. Is it possible Peter?

That's not possible without telling all mirrors that use that module
that it suddenly contains the additional stuff, and giving them the
opportunity to exclude it. This is especially true since the module is
already overly large. And it would be not easy to achieve, because I
have no contact address for many of the mirrors.

Note, there is a LOT of stuff we can't just add to full-with-factory.
The module would be 1.4 TB in size. There would hardly any mirrors
interested in this. We mirror too much already; what helps openSUSE
*users* is *many* mirrors mirroring what's popular, not a few mirrors
that have "everything". I believe that this is one reason why Ubuntu has
more mirrors than openSUSE. It is simply easier to find mirrors for 30GB
than for 300GB.

Please also note that an rsync over the full tree (even with -n) takes a
LONG time, and we can't just make available the full tree in a single
rsync module for pragmatical reasons. It would be simple enough to do,
but the stage server would not be able to deal with it (and still be
useful for what we need it for); it's just totally impractical for us.

The direction that is most useful to develop, in my humble opinion, is a
push sync, in the way we already have in the /repositories tree. The
same I want to implement for /factory and /update, for which we would
benefit most from that because these trees are short-lived, and sync
lags could be avoided. Also, unneeded pull syncs can be avoided in that
way (which cost resources too). The push sync could start feeding the
mirrors with the changed content right after it was changed on
stage.opensuse.org. Of course, this makes matters even more complicated
than what you have now - syncing from separate rsync modules. So for
this to work, we need to make it easy for everybody and provide a
working, ready solution. One way could be to implement it in the same
way we implemented the push for /repositories - by getting write access
to the mirror. That makes it easiest for the mirrors and actually saves
them work. A different way could be to provide ssh triggers, and provide
scripts for mirroring that are prepared to deal with the triggers (and
already know and use the modules).

Thanks!
Peter
--
Contact: admin@xxxxxxxxxxxx (a.k.a. ftpadmin@xxxxxxxx)
#opensuse-mirrors on freenode.net
Info: http://en.opensuse.org/Mirror_Infrastructure

SUSE LINUX Products GmbH
Research & Development
< Previous Next >
List Navigation
Follow Ups