Hi, For the mirrors we push the repositories on there is a problem: we do not push deletes due the way the whole process works. For this I consider doing a weekly full sync of the full repositories tree. What would be a good time for doing that? Is the weekend for everyone ok or should we do different times per mirror? Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, On Fri, 18 Mar 2011, Stephan Kulow wrote:
For the mirrors we push the repositories on there is a problem: we do not push deletes due the way the whole process works.
For this I consider doing a weekly full sync of the full repositories tree. What would be a good time for doing that? Is the weekend for everyone ok or should we do different times per mirror?
Best would be to watch /var/log/rsyncd.log at stage.opensuse.org for begin and end of "client initiated" sessions with buildservice-repos-full and to delay all pushes during that period. Viele Gruesse Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoenke@gwdg.de Tel.: +49 (0)551 201-1551 ------------------------------------------------------------------------- Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL: http://www.gwdg.de E-Mail: gwdg@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Oswald Haan und Dr. Paul Suren Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 ------------------------------------------------------------------------- -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Am Freitag, 18. März 2011 schrieb Eberhard Moenkeberg:
Hi,
On Fri, 18 Mar 2011, Stephan Kulow wrote:
For the mirrors we push the repositories on there is a problem: we do not push deletes due the way the whole process works.
For this I consider doing a weekly full sync of the full repositories tree. What would be a good time for doing that? Is the weekend for everyone ok or should we do different times per mirror?
Best would be to watch /var/log/rsyncd.log at stage.opensuse.org for begin and end of "client initiated" sessions with buildservice-repos-full and to delay all pushes during that period.
If you pull in addition to the pushing you get into trouble as repos-full is using a different white and blacklist then what we have right now per mirror. Greetings, Stephan -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, On Fri, 18 Mar 2011, Stephan Kulow wrote:
Am Freitag, 18. März 2011 schrieb Eberhard Moenkeberg:
On Fri, 18 Mar 2011, Stephan Kulow wrote:
For the mirrors we push the repositories on there is a problem: we do not push deletes due the way the whole process works.
For this I consider doing a weekly full sync of the full repositories tree. What would be a good time for doing that? Is the weekend for everyone ok or should we do different times per mirror?
Best would be to watch /var/log/rsyncd.log at stage.opensuse.org for begin and end of "client initiated" sessions with buildservice-repos-full and to delay all pushes during that period.
If you pull in addition to the pushing you get into trouble as repos-full is using a different white and blacklist then what we have right now per mirror.
I don't understand - what ist that "white and black" list? My seen problem with "push and pull concurrence" is that rsync would delete on my side some freshly pushed stuff because it was initially not present at the server side. Viele Gruesse Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoenke@gwdg.de Tel.: +49 (0)551 201-1551 ------------------------------------------------------------------------- Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL: http://www.gwdg.de E-Mail: gwdg@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Oswald Haan und Dr. Paul Suren Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 -------------------------------------------------------------------------
On Fri, 18 Mar 2011 14:35:00 +0100 (CET)
Eberhard Moenkeberg
I don't understand - what ist that "white and black" list?
we exclude some older distros from pushing while still allow people to pull those via rsync.
My seen problem with "push and pull concurrence" is that rsync would delete on my side some freshly pushed stuff because it was initially not present at the server side.
yes. you can be sure that repo pusher is sending files to you almost constantly. you really should stop your pull rsync for that part of the tree. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, On Fri, 18 Mar 2011, Marcus Rückert wrote:
On Fri, 18 Mar 2011 14:35:00 +0100 (CET) Eberhard Moenkeberg
wrote:
I don't understand - what ist that "white and black" list?
we exclude some older distros from pushing while still allow people to pull those via rsync.
I see no problem with that.
My seen problem with "push and pull concurrence" is that rsync would delete on my side some freshly pushed stuff because it was initially not present at the server side.
yes. you can be sure that repo pusher is sending files to you almost constantly. you really should stop your pull rsync for that part of the tree.
No, we need some mechanism to stop pushing during running pulls. Viele Gruesse Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoenke@gwdg.de Tel.: +49 (0)551 201-1551 ------------------------------------------------------------------------- Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL: http://www.gwdg.de E-Mail: gwdg@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Oswald Haan und Dr. Paul Suren Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 -------------------------------------------------------------------------
On Fri, 18 Mar 2011 15:01:30 +0100 (CET)
Eberhard Moenkeberg
No, we need some mechanism to stop pushing during running pulls.
not really. pushing is in this case much more efficient than pulling. the tree sees so many changes that it is much faster to just sync the changed parts with a push mirror. on busy days it can happen that you just pulled stuff that got obsoleted already and is queued to be pushed again. on top of that ... if we would block pushing to you while you pull, the events will still be queued for your mirror and it would sync all the "changed" stuff again after your pull. last but not least we scan your mirror after each push, also limited to the subdir, so we get the database up2date as fast as possible. why do you need the pull sync additionally to the pushes? darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, On Fri, 18 Mar 2011, Marcus Rückert wrote:
On Fri, 18 Mar 2011 15:01:30 +0100 (CET) Eberhard Moenkeberg
wrote:
No, we need some mechanism to stop pushing during running pulls.
not really. pushing is in this case much more efficient than pulling. the tree sees so many changes that it is much faster to just sync the changed parts with a push mirror. on busy days it can happen that you just pulled stuff that got obsoleted already and is queued to be pushed again.
That would be OK and consume no additional bandwidth.
on top of that ... if we would block pushing to you while you pull, the events will still be queued for your mirror and it would sync all the "changed" stuff again after your pull.
Why not. "Alrady present" gets detected.
last but not least we scan your mirror after each push, also limited to the subdir, so we get the database up2date as fast as possible.
why do you need the pull sync additionally to the pushes?
To force the deleting of obsolete files, just like coolo has pointed. Viele Gruesse Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoenke@gwdg.de Tel.: +49 (0)551 201-1551 ------------------------------------------------------------------------- Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL: http://www.gwdg.de E-Mail: gwdg@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Oswald Haan und Dr. Paul Suren Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 -------------------------------------------------------------------------
On Fri, 18 Mar 2011 15:17:22 +0100 (CET)
Eberhard Moenkeberg
To force the deleting of obsolete files, just like coolo has pointed.
obsolete files get deleted, obsolete project directories dont. thats why we want to add the full push sync on regular base. darix -- openSUSE - SUSE Linux is my linux openSUSE is good for you www.opensuse.org -- To unsubscribe, e-mail: mirror+unsubscribe@opensuse.org For additional commands, e-mail: mirror+help@opensuse.org
Hi, On Fri, 18 Mar 2011, Marcus Rückert wrote:
On Fri, 18 Mar 2011 15:17:22 +0100 (CET) Eberhard Moenkeberg
wrote:
To force the deleting of obsolete files, just like coolo has pointed.
obsolete files get deleted, obsolete project directories dont. thats why we want to add the full push sync on regular base.
I prefer doing the deletes by a pull run. Doing it by a central push to all mirrors can lead to a worldwide loss in case of error. Or you should use rsync with the options "-b --backup-dir=.old-saved --exclude /.old-saved". Viele Gruesse Eberhard Moenkeberg (emoenke@gwdg.de, em@kki.org) -- Eberhard Moenkeberg Arbeitsgruppe IT-Infrastruktur E-Mail: emoenke@gwdg.de Tel.: +49 (0)551 201-1551 ------------------------------------------------------------------------- Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen (GWDG) Am Fassberg 11, 37077 Goettingen URL: http://www.gwdg.de E-Mail: gwdg@gwdg.de Tel.: +49 (0)551 201-1510 Fax: +49 (0)551 201-2150 Geschaeftsfuehrer: Prof. Dr. Oswald Haan und Dr. Paul Suren Aufsichtsratsvorsitzender: Prof. Dr. Christian Griesinger Sitz der Gesellschaft: Goettingen Registergericht: Goettingen Handelsregister-Nr. B 598 -------------------------------------------------------------------------
participants (3)
-
Eberhard Moenkeberg
-
Marcus Rückert
-
Stephan Kulow