Mailinglist Archive: mirror (7 mails)

< Previous Next >
Re: [suse-mirror] rsync module for the sources - now there

On Fri, 3 Apr 2009, Peter Poeml wrote:
On Tue, Mar 31, 2009 at 04:06:22PM -0300, Carlos Carvalho wrote:
Peter Poeml (poeml@xxxxxxx) wrote on 30 March 2009 20:49:

How about creating another module: full-with-factory-and-sources? This
way you'll be sure that only those who *really* want them will bother

Yes, that makes a lot of sense.
You are right, choice is good and it is easy to arrange.

However, a module with "everything" in is something I always tried to
get rid of, to avoid load problems and save resources. You are a very
considerate person and would not abuse the module, but it isn't obvious
to everybody in the first place how to use it sensibly. In the past we
sometimes had too many syncs on the complete "everything" module. That
module contained both stuf that is huge but never chages, and other
stuff that's even more huge and changes constantly. Such a module can
only be sensibly used when the mirror accesses parts of it.

You can use a module named opensuse-full-really-everything which
isn't publicly documented, but it's there; I can't remember exactly if
it has always existed or if I made it at one point in time, but never
announced it. (I think the former.)

Very good (long appreciated), but:

receiving file list ...
rsync: The server is configured to refuse --hard-links (-H)
rsync error: requested action not supported (code 4) at
clientserver.c(839) [sender=3.0.5]

You should allow -H, especially at this "top level".

I'm sure that problems can be avoided by proper documentation and good
guidance; and I'm looking forward to implement a proper sync framework
which automates these issues (or helps at least), instead of letting
everybody solve this on his own.

About update frequency, I usually sync a release only once, when it
appears, and never again, because they should NOT change. What do you
mean by "nearly never"? Aren't ALL changes done in updates?????
Anyway, if changes do happen they should be announced here.

With "nearly never" I meant: yes, it isn't supposed to happen that
released products change; you should not need to resync later. However,
I have seen cases where we needed to correct some problem so there might
actually have been a change. Yes, that should be announced.

On the other hand, since we host the alphas, betas and final product in
the same tree, so far we have most mirrors syncing from /distribution
daily I assume. So a change would propagate to them.

This separation between releases and the rest needs manual
intervention only at release times and should be enough to avoid
overloading stage.

The distribution tree gives us less headache than the buildservice tree,
which is much much larger, but yes, you are right, the distribution tree
could be split into a short-lived part for daily syncing, and a frozen
part which needs to sync never. (Alternative to "never" could be
"weekly", which allows for correction of problems without notice, and

>A trigger-based sync mechanism might be a way around this. I have some
>things in mind, and know some ways how other projects deal with this,
>but other than ideas there is not much resources to work on this.

Perhaps the easiest and most effective way is a social one: mirror
tiering. You chose the bigger, better connected and better managed
mirrors spread around the world, and ask them a commitment in being
tier-1 mirrors for opensuse. They'd need to have at least
full-with-factory-and-sources [oh!! :-)], plus factory ppc, and allow
public access via rsync. Only these would have access to stage, the
others would use the tier-1s, so that you keep the crowd off your
machine. Tiering would give you a solid distribution network without
consuming your resources. This is what most distros do. There'd be no
changes in using mirror brain to monitor all mirrors and sending
clients to them. You could perhaps also count on the tier-1s to
implement some of the technical methods below.

Yes. It's amazing that we have survived, bandwidth-wise, without tiering
for so long.

No wonder. SUSE could from the beginning rely on the german university net
structure (DFN). It just was appropriate to the needs of the university
data centers...

An issue that I see with tiering is a little give-up of control, which
doesn't matter much (knowing who syncs; discovering new mirrors that
would go unnoticed otherwise), but some things could be a loss (the
possibility of updating the mirror database tied to rsync log parsing,
which is something that I wanted to explore a little further, but
which is probably better done by some form of triggering or pushing

You already have this tiering, by tradition.
The german DFN network always was (and still is) the "technological
leader" in bandwidth, so since ever it is best choice to use a tier-1
mirror within the DFN.

In the context of reducing rsync load on the master, triggering is
only useful if it avoids full rsyncs. The only way to avoid it is to
deal with the changes only. This can be done in several ways. One is
what does, emailing only the changes. We use it here, keeps
us very close indeed to the master with negligible load.

Dealing with the changes only is the way to go IMO.

Peter, please extend the push service soon.
factory and update would win most, but in essence, every rsync target can
get speeded up with a push service.

That's what we do for the buildservice tree, and nothing else would
scale there. And the problem exists in more places than just mirror
syncing. For example, we have buildservice -> ->
mirrors and buildservice -> -> ->

Similarly, we have the same problem with updates (/update):
buildsystem -> internal stage host -> stage.o.o -> mirrors
buildsystem -> internal stage host -> stage.o.o -> rsync.o.o -> mirrors

Same for factory, and so on.

(tier 1 could be inserted there ;)

At each of these stages, the sync needs to be timely (triggered pull or
push), and not overlapping a sync at the stage before or after. At the
same time, at each stage load can be wasted, or saved by avoiding
unneeded syncs. At each of the stages, the order of files synced could
matter (packages before repodata is better...).

A good point - Peter, you are the only one who could help to make this

Once content reaches the end points (the mirrors), the mirror database
should be updated simultaneously, or soon afterwards.

With a full push service, no problem.

I don't know details about how detects changes (it may depend
on their tree), but I have two things in mind for that:

1) a generic way to detect changes in a file tree based on the inotify
mechanism. I'm thinking of a recursive directory watch which notices
when changes occur, awaits the end of the changes (by waiting for a
long enough period of time without further changes happening), thereby
detecting the moment when an incoming sync is finished. Then it could
trigger the next-stage mirror, or start a push sync to it.

Can't you watch the incoming syncs by inspecting the logs?

2) a more specialized mechanism for too large trees where inotify
doesn't scale (that's the case with the buildservice with its 70.000
directories). This mechanism is specific to the build service tree and
its layout.

I'm sure you already _do_ have a good mechanism for the buildservice tree.

I am not sure if 1) and 2) can be combined; but I'm sure that I would
like to have something that is not a hacky local solution but rather
something that's reusable enough for other people to use, and last but
not least also for myself to easily install on other systems where a
"organized" sync is needed.

Our publishing of the factory tree and update tree would greatly benefit
from this. With 2) (and the buildservice tree), we have a very tight
integration with the mirror database and redirection now. If a new
build shows up on, it's pushed out directly (well,
free slots provided...); creation of md5 and sha1 hashes is done at the
same time, and once a project has been pushed to a mirror, the files are
entered into the mirror database right away, so
starts redirection to the mirror right away. And the latter is really
essential for us -- we may not even survive for an hour, when we can't
push out stuff timely. And for /update, where we only have "random" pull
syncs, we often see periods "without mirrors" which can keep
download.o.o very busy (and that's slow for customers).

Yes, the buildservice pushes are working as expected! Please try to
extend this mechanism to _everything_!

Another possibility, as you say, is to have write access, which is
equivalent to having an account on the machine. We also do it here;
sourceforge is a very big example. They're very good at keeping a 10
times bigger-than-a-distro tree in sync with minimal load.

Sourceforge is also very good in selecting what to actually sync to
mirrors. That's also a direction that I want to explore (and talk to
them if we can share something).

Good. But for the biggest tree you already have your solution.

A third method is to use rsync in a better way. We don't do disk
scanning here when we update; only the master is hammered :-) However
if you give me a list of files in your site, such as the one created
by find or

rsync localhost::a-[hidden]module-with-everything > filelist

then we'll do *no* disk scans at *either* end, and only pull the
necessary files. We do this for another distro...

That's nice idea that I haven't looked into so far. I need to
familiarize a bit with that. So far, I tended to believe that it
wouldn't work for us; not for the large trees. Collecting the filelist
locally is a job that takes hours, which makes it a hopeless undertake
for a tree that changes every 30 seconds. Nice for trees that don't
change I guess...

A full directory scan ("ls -lR") at (7.7 TB, 11.5 million
files) needs 38 minutes - can you add some more RAM to your machine?
ftp5 has 32 GB, and the only bottleneck is network.
The filespace is LVM'ed over a single Transtec/Infortrend raid array
with two controllers (each serving a raid6 set of 11 disks).
Filesystem type: xfs.

Sounds doable if the filelists are collected not from the filesystem,
but e.g. from the buildservice (which already knows what it updates).

Surely the best way.

Even better, if you
give a list of checksums we'll use it both for updating and for
verifying that our repo is correct. We also do it with another distro.
The disadvantage of this method is that it needs a complicated script,
so mirrors are unlikely to use it.

I have a local tree of checksums for most files, but unfortunately that
tree has as many directories and as many files as the actual file tree.
So collecting them is also much too slow for anything which looks at the
complete tree. Maybe there are smarter ways to use them for something
that I haven't thought about yet.

In my experience, xfs is about "double fast" against ext3 regarding
"ls -lR" with big directories.

Viele Grüße
Eberhard Mönkeberg (emoenke@xxxxxxx, em@xxxxxxx)

Eberhard Mönkeberg
Arbeitsgruppe IT-Infrastruktur
E-Mail: emoenke@xxxxxxx Tel.: +49 (0)551 201-1551
Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG)
Am Fassberg 11, 37077 Göttingen
URL: E-Mail: gwdg@xxxxxxx
Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150
Geschäftsführer: Prof. Dr. Bernhard Neumair
Aufsichtsratsvorsitzender: Dipl.-Kfm. Markus Hoppe
Sitz der Gesellschaft: Göttingen
Registergericht: Göttingen Handelsregister-Nr. B 598
< Previous Next >
List Navigation
Follow Ups