I guess not everyone may be aware (?), but we offer a push-service for the build service repositories, such that they can be made available on mirrors as soon as possible. It is up to the mirror admin "on the other end" to ask for this service to be set up. Currently we have six active mirrors with repositories being pushed out to them. There are also five that have stopped for some or other reason. We are pushing out a lot of changes every day, and the original scheme dates back to a different era. The rate of change keeps going up and up and the setup was growing increasingly slow. Well, for the last couple of weeks, I have been looking at how to improve this - I think I have come up with an improved repopush setup - see attached. I put my scheme into motion on 28Aug. Initially I had some teething problems, but as of 2 September, it seems that things have been improving steadily. A fast mirror like ftp.gwdg.de can now complete a push in about three hours, although it depends on how busy build service has been. Our own rsync.o.o is also doing quite well. The new scheme is to generate a Makefile based on the contents of /srv/bs/arrived/ and /srv/bs/pushed/ - 'make' is very good at figuring out what needs to be pushed and 'make' is also good at running things in parallel. pseudo-code: do forever generate makefile.repos run pushes in parallel, one per mirror when one push finishes, cancel the rest [1] done [1] to avoid starving a fast mirror. There is some work left to do - the server push scripts need to return a proper return code to indicate if a push was successful, for instance. They also still do locking, which is no longer necessary. Key code elements - /srv/bs/bin/push_repositories /srv/bs/bin/repopush.genmake I also need to create a systemd service unit, but that's no big deal. -- Per Jessen, Zürich (18.8°C) Member, openSUSE Heroes
