On Monday 05 November 2007 08:59:41 Michael Matz wrote:

Hey, great minds think alike as they say :) Yes, we are actively trying to do just that with the SOLV files which Stano already mentioned in the plan. The general plan is to rely on an xdelta/rsync like algorithm, where we have two possibilities:

1) the requester generates a list of blocks it has, sends them to the server, that one figures out which blocks are missing, and sends a diff description to the client, which reconstructs the file.

2) The server generates once (when the file changed) a list of block it has, client downloads that one, and actively asks for the blocks it still needs.

3) server pre-generates a set of xdelta diffs from every file revision to the current one, clients asks for the xdelta matching his own revision, and updates according to that.

ZSync does that already. But as I had said lot of times, it wont work with current metadata because it changes too much and too often. Duncan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

Am Montag 05 November 2007 schrieb Benji Weber:

On Monday 05 November 2007 08:59:41 Michael Matz wrote:

...

The general plan is to rely on an xdelta/rsync like algorithm,

This should give some savings. It should be possible to get better savings with a custom diff that could be generated by createrepo. It would only need to include the changed package details, and could diff things like the description which I suspect don't change frequently even in factory. Then generate a primary.xml.<oldtimestamp> or something as a diff from that timestamp.

This would be a lot more work than just adding zsync to the download layer though.

On 05/11/2007, Duncan Mac-Vicar Prett wrote:

...

ZSync does that already. But as I had said lot of times, it wont work with current metadata because it changes too much and too often.

Really only factory this applies to I think. I update only packages that have changed (checksum) for the package search, and this is faster for many repositories. Only for factory is it counterproductive. The biggest benefit is with packman where rebuilds are rare and one or two packages are usually appended to metadata. The frequently changing repositories that most people have will be the update repository & packman.

I think it can be beneficial even though it doesn't help for factory. It's also true for KDE:KDE3 for example where one checkin of kdelibs3 will recompile everything (which happens often).

Or many other build service repos. So your patch logic would need to find an intelligent way to figure which oldtimestamps do not make sense to be supported because they are better off downloading HEAD and remove these old diffs. Greetings, Stephan -- SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

On Monday 05 November 2007 16:36:58 Klaus Kaempf wrote:

Do we have hard numbers about how much of a package's metadata actually changes between rebuilds ?

Usually, a lot of metadata changes only very slowly, like summary, description, changelog, dependencies, etc. Only checksum and release number have a high change frequency in Factory.

Block-wise syncing is certainly one approach, grouping metadata pieces into blocks of high change (checksums, release numbers) and low change (summary, description) might be worth investigation.

with YUM Factory was only worth to zsync for one rebuild, after 1 rebuild was minimal gain, and 4 days later it was pretty useless. It is easy to regenerate them with any new type of metadata, zsync is in my OBS home: and you can generate the metadata for one file, then wait, apply the zsync metadata to a new one and seeing how many blocks apply, wait, and repeat). Of course you can control also the block size, that could affect XML results, smaller block checksum can perhaps capture xml descriptions but also makes the zsync metadata to grow, as you have more blocks. The best would be first to design the ideal way to store things, and then to repeat zsync experiments and find the optimal block size. Duncan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

Hi, On Mon, 5 Nov 2007, Duncan Mac-Vicar Prett wrote:

On Monday 05 November 2007 08:59:41 Michael Matz wrote:

...

Hey, great minds think alike as they say :) Yes, we are actively trying to do just that with the SOLV files which Stano already mentioned in the plan. The general plan is to rely on an xdelta/rsync like algorithm, where we have two possibilities:

1) the requester generates a list of blocks it has, sends them to the server, that one figures out which blocks are missing, and sends a diff description to the client, which reconstructs the file.

2) The server generates once (when the file changed) a list of block it has, client downloads that one, and actively asks for the blocks it still needs.

3) server pre-generates a set of xdelta diffs from every file revision to the current one, clients asks for the xdelta matching his own revision, and updates according to that.

ZSync does that already.

Good.

But as I had said lot of times, it wont work with current metadata because it changes too much and too often.

That's why I talk about SOLV files :) Ciao, Michael. -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

Dňa Friday 02 November 2007 15:23:03 Benji Weber ste napísal:

...

- callbacks cleanup The annoying YaST popups are a direct consequence of the complicated callback system of libzypp. A redesign and cleanup of the callbacks should give much better user experience, like

I had a look at this earlier this week, see http://bw.uwcs.co.uk/new_callbacks.ogg

Wow. Is this an actual code? Stano -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

Dňa Monday 05 November 2007 10:29:13 Benji Weber ste napísal:

On 05/11/2007, Stanislav Visnovsky wrote:

...

Wow. Is this an actual code?

Yes, but I modified PackageCallbacks which will break a lot of things. I was going to create a new EmbeddableCallbacks or something that can be used alongside.

Cool! I'd suggest to create an experimental branch so everyone can try it out, maybe also using buildservice to get the packages. This is definitely very interesting! Stano -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

* Duncan Mac-Vicar Prett [Nov 05. 2007 12:12]:

On Monday 05 November 2007 08:44:25 Cristian Rodriguez wrote:

...

e) zypper "patch" operations should only read an refresh repos that actually **contain** patches (i.e doing something like curl -Iv http://ftp4.gwdg.de/pub/opensuse/update/10.3/repodata/patches.xml as detection) currently all repos are refreshed regardless the contents.

That would make patch checking MUCH faster, but it depends on the assumption that all needed packages from the patch are in the same update repo, and that you can upgrade your system only with installed packages + update repos. This happens 99% of the time

Right. However, patches should be seen as supportive infrastructure to (package) updates and not as the primary means to deliver updates. Esp. 3rd party (and customer specific inhouse) repos will most probably stay with packages and not make use of patches. For these, we need a full refresh.

Not useful if you want to mention a foreign package in a patch.

Which happened quite some times in the past, esp. with changing requirements dragging in additional packages from the original installation repo. Klaus -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

Cristian Rodriguez wrote:

Cristian Rodriguez escribió:

...

I have more suggestions and ideas, but that's all for now ;)

Also, while downloading large packages, zypp does not seem to "resume" the download if you stop the process but starts once again, this is very unneficient, will be cool if that functionality can be implemented.

Note that you dont need much code to add, curl is abstracted in such a way that is trasparent to the protocol, you can even obtain that

It would be little code to add if the RPMs would be downloaded into a well known place, but they are not. But we should definitely bear this in mind when implementing the "download-only--install-later" feature. Jano

functionality,and others like that timebased ones for CD/DVD/NFS/SMB/CIFS or whatever you need using the "file://" pseudo-protocol that curl supports ( as long as you mount the CD/DVD/NFS/SMB/CIFS media before, of course ;) )

-- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

* Stefan Hundhammer [Nov 07. 2007 11:41]:

On Wednesday 07 November 2007 10:52, Klaus Kaempf wrote:

...

- Policy layer   Solving package dependencies consists of clearly defined dependency   semantics and assumptions not expressable by dependencies.   Those assumptions are commonly referred to as 'policies'.   Future versions of libzypp must expose these policies, so an   application or system administrator (system wide policies) can   change them.

Please keep in mind that this will have to be handled by the application layer. This is nothing the UI layer will ever want to be concerned with.

Actually, its a layer separate from application. UIs for advanced users might expose 'policy knobs' for tuning, though. Klaus -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org