On Mon, Mar 30, 2009 at 02:45:22PM -0300, Carlos Carvalho wrote:
Peter Poeml (poeml@suse.de) wrote on 30 March 2009 17:02:
So, I have created opensuse-source as rsync module at rsync://stage.opensuse.org/opensuse-source now.
It's rather cumbersome to have to do separate syncs for parts of the same repository.
Yes, I see that. I am not sure though if it would be better if there was only one rsync module for the entire tree, because, you would still need to set up different syncs, because there are parts of the tree that change frequently (updates) and other parts that change nearly never (released products). It wouldn't make sense to sync the released products every four hours, and in addition to that, we would not be able to deal with this, with our resources. Or do you have a different opinion? A trigger-based sync mechanism might be a way around this. I have some things in mind, and know some ways how other projects deal with this, but other than ideas there is not much resources to work on this. Maybe you have ideas to contribute on this, examples how others manage to do better, and such. Please continue to let me/us know.
Priority is not measured only by downloads, importance also counts, and sources are essential for free software. That's why it should be possible to mirror them together.
I might miss the context that you refer to, but what do you mean by "Priority" here? And "importance"? By the way, maybe you also followed the argument about the openSUSE build service for source access. It is the "modern", convenient and powerful way to access the sources, for openSUSE 11.1 onwards. We basically need source RPMs right now only for those people who don't know that yet, and because the build service doesn't allow anonymous access yet. People tend to be irritated when they don't find the source packages and think that something is wrong, even though it isn't. I checked the download numbers of today, and the handful of people that have downloaded a source package today is neglectable; so it doesn't make sense to put this stuff out to mirrors. Anyway, sources are sensitive issues; and the tree is available now for everybody who's interested!
stage.opensuse.org should have it all, have it all in place, and give access to the topmost point.
Yes.
Yes, in a way this would make a lot things easier. I appreciate your thoughts. I lack the time to really do something about this right now, but I'll keep it noted.
One possibility that seems easy enough is to include sources in the larger modules, like full-with-factory, in stage. Looks like just removing some exclusions would do. Is it possible Peter?
That's not possible without telling all mirrors that use that module that it suddenly contains the additional stuff, and giving them the opportunity to exclude it. This is especially true since the module is already overly large. And it would be not easy to achieve, because I have no contact address for many of the mirrors. Note, there is a LOT of stuff we can't just add to full-with-factory. The module would be 1.4 TB in size. There would hardly any mirrors interested in this. We mirror too much already; what helps openSUSE *users* is *many* mirrors mirroring what's popular, not a few mirrors that have "everything". I believe that this is one reason why Ubuntu has more mirrors than openSUSE. It is simply easier to find mirrors for 30GB than for 300GB. Please also note that an rsync over the full tree (even with -n) takes a LONG time, and we can't just make available the full tree in a single rsync module for pragmatical reasons. It would be simple enough to do, but the stage server would not be able to deal with it (and still be useful for what we need it for); it's just totally impractical for us. The direction that is most useful to develop, in my humble opinion, is a push sync, in the way we already have in the /repositories tree. The same I want to implement for /factory and /update, for which we would benefit most from that because these trees are short-lived, and sync lags could be avoided. Also, unneeded pull syncs can be avoided in that way (which cost resources too). The push sync could start feeding the mirrors with the changed content right after it was changed on stage.opensuse.org. Of course, this makes matters even more complicated than what you have now - syncing from separate rsync modules. So for this to work, we need to make it easy for everybody and provide a working, ready solution. One way could be to implement it in the same way we implemented the push for /repositories - by getting write access to the mirror. That makes it easiest for the mirrors and actually saves them work. A different way could be to provide ssh triggers, and provide scripts for mirroring that are prepared to deal with the triggers (and already know and use the modules). Thanks! Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development