Hi, On Mon, Jan 26, 2009 at 05:10:54PM +0100, Robert Kaiser wrote:
Stephan Kulow wrote:
The solution to your problem contains two parts (and both are worked on):
- Factory will push out smaller deltas. So you only need to download let's say 400MB instead of 5GB.
- zypper will download these files first before starting installation.
Both sound interesting and could surely help. OTOH, I tend to have the feeling that applying deltas takes long enough that a download of the full package is often faster.
Keeping files for 24 hours sounds like an interesting idea, but I have no idea how to implement this technically. If a new factory has built, it's pushed out by a script. That script first uploades the new files and then deletes the old ones. Hmm, it could keep an index to see when it uploaded specific files and delete from there. Need to think about it.
The script could keep an index of when files became obsolete (instead of deleting them right away) and only delete those that have that date more than 24 hours in the past.
Good idea; I have been ventilating this for some time. A simple time-based deletion scheme, which came to mind first, might not be enough, though -- tracking whether files have actually been referenced by metadata in some timeframe before deletion could be better. An additional aspect is the decision whether these duplicate (or more) files should be present on Factory mirrors, which effectively doubles the storage need for Factory (and hardlinks won't change this) -- or if it is sufficient if download.opensuse.org is offering them for some time, and knows places where they still exist which are possibly closer to the user (mirrors that are not yet up to date). All this will be pretty fragile without Factory having "libzypp download failover" (http://en.opensuse.org/Libzypp/Failover) enabled. I hope this will be enabled in the near future. If it is enabled, it will make sure that outdated URLs don't matter. So it is quite critical for us. A different, server-side way to make sure that clients don't end up on a mirror that has *just* deleted a file (because transfer is in progress) would be very hard to implement, without tight integration of rsync transfer and mirror database (and rsync deletes on the receiver side are not actually seen (logged) on the sender side; they could only be implied. It is a bit more feasible when we sync the factory/ tree exclusively per push sync (rsync originating from our side, connecting to the mirrors), because we are then in more control and know when the mirror tree changes in which way. Push sync makes this slightly more feasible than an ssh-triggered pull sync (which is also conceivable, in order to sync Factory out in a more organized fashion). A push sync is the way to go anyway, and we have good field experience with it for the repositories/ tree. Still experimenting. The outdatedness of many Factory mirrors during the last 10 days is fallout from my work on this; I could not continue working on it for a while though; sorry folks. Current proposal: 1) come up with a push syncing solution and set it up for some mirrors, and delayed deletion on download.o.o 2) implement libzypp download failover Both play together. 2) alone does not fix this particular problem; it requires 1) because download.opensuse.org cannot suggest mirrors for files that are unknown to it. And 1) requires 2) because 100% mirror control is not feasible, especially if mirrors pull sync.
Robert Kaiser
Peter -- Contact: admin@opensuse.org (a.k.a. ftpadmin@suse.com) #opensuse-mirrors on freenode.net Info: http://en.opensuse.org/Mirror_Infrastructure SUSE LINUX Products GmbH Research & Development