[zypp-devel] Detecting non-synced repos
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I have implemented the basics for feature #301904, which calls for having a way to detect that a repository may be outdated. http://en.opensuse.org/Standards/Rpm_Metadata#Outdated_Metadata_Hint I am citing what I documented there: Joe user has an update repo pointing to a mirror directly (no redirector). The mirror goes out of sync (however it is still reachable). He does not receive any warning, and meanwhile the main repo has lot of updates. The server will have a hint on how often the metadata is regenerated due to a change. The client could detect this and hint the user that the metadata has not been updated since long time. It is responsibility of the server then to update the timestamp of the files if there are no changes in the expire time the repo administrator decided for his own repo. The value is only a hint. The user should be able to disable or alter it. (global in zypp.conf and per repo in the .repo file) Implementation An attribute "expire" is added to repomd section: <repomd expire="100000"> <data type="primary"> <location href="repodata/primary.xml.gz"/> <timestamp>342423423> Originally in the first proposal, the timestamp was another attribute, but now I calculate it as the newer timestamp of all the resources listed in the repomd.xml file. This enhances zypp::Repository class API as follows: zypp::Date Repository::generatedTimestamp() const; zypp::Date Repository::suggestedExpirationTimestamp() const; bool Repository::maybeOutdated() const; This can be used by applications to check whether a repository is "dead". However we could as well implement easier hooks for applications to receive a report, or signal. The actual implementation is to read the values from repomd.xml and merge them in the .solv file as an extra attributes from the extra non-solvable number -1. (which is tricky, as if there are deltarpms for example, there will be a deltarpm data in position -1, but we don't care as the rpm reader don't look at them ), may be Michael Matz can comment if my approach is ok. And of course, there is a testcase. Any feedback on how to mature this, welcome. Duncan -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org iEYEARECAAYFAkh4EMEACgkQzR62qWZ+QtGhpwCgmcugRResmxRlSmcOjGWegtl4 DswAoK+DIcsfuXdovOtziwgND+4Ie18D =+PSa -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Dne Saturday 12 of July 2008 04:02:41 Duncan Mac-Vicar P. napsal(a):
I have implemented the basics for feature #301904, which calls for having a way to detect that a repository may be outdated.
http://en.opensuse.org/Standards/Rpm_Metadata#Outdated_Metadata_Hint
I am citing what I documented there:
Joe user has an update repo pointing to a mirror directly (no redirector). The mirror goes out of sync (however it is still reachable). He does not receive any warning, and meanwhile the main repo has lot of updates.
The server will have a hint on how often the metadata is regenerated due to a change. The client could detect this and hint the user that the metadata has not been updated since long time. It is responsibility of the server then to update the timestamp of the files if there are no changes in the expire time the repo administrator decided for his own repo.
The value is only a hint. The user should be able to disable or alter it. (global in zypp.conf and per repo in the .repo file)
Implementation
An attribute "expire" is added to repomd section:
<repomd expire="100000"> <data type="primary"> <location href="repodata/primary.xml.gz"/> <timestamp>342423423>
Originally in the first proposal, the timestamp was another attribute, but now I calculate it as the newer timestamp of all the resources listed in the repomd.xml file.
This enhances zypp::Repository class API as follows:
zypp::Date Repository::generatedTimestamp() const; zypp::Date Repository::suggestedExpirationTimestamp() const; bool Repository::maybeOutdated() const;
This can be used by applications to check whether a repository is "dead". However we could as well implement easier hooks for applications to receive a report, or signal.
The actual implementation is to read the values from repomd.xml and merge them in the .solv file as an extra attributes from the extra non-solvable number -1. (which is tricky, as if there are deltarpms for example, there will be a deltarpm data in position -1, but we don't care as the rpm reader don't look at them ), may be Michael Matz can comment if my approach is ok.
And of course, there is a testcase.
Any feedback on how to mature this, welcome.
I like it, IMO it does exactly what user may expect. I just added the perhaps obvious requirement to regenerate the metadata before it expires even if there is no change. Jiri -- Regards, Jiri Srain YaST Team Leader --------------------------------------------------------------------- SUSE LINUX, s.r.o. e-mail: jsrain@suse.cz Lihovarska 1060/12 tel: +420 284 028 959 190 00 Praha 9 fax: +420 284 028 951 Czech Republic http://www.suse.cz
On Sat, Jul 12, Duncan Mac-Vicar P. wrote:
This enhances zypp::Repository class API as follows:
zypp::Date Repository::generatedTimestamp() const; zypp::Date Repository::suggestedExpirationTimestamp() const; bool Repository::maybeOutdated() const;
Why class Repository? This requires to load the .solv into the pool, just to be able to access these data. Wouldn't you like to have this informatin in advance. I would expect this to be offered by RepoInfo, retrieving the data directly from the .solv file / repomd.xml, if not yet loaded into the pool. for ( all repoInfos ) if ( ! repoInfo.enabled() ) continue refreshIfNeeded( repoInfo ); if ( repoInfo.maybeOutdated() && userWntsToSkipIt() ) continue loadFromCache( repoInfo ); -- cu, Michael Andres +------------------------------------------------------------------+ Key fingerprint = 2DFA 5D73 18B1 E7EF A862 27AC 3FB8 9E3A 27C6 B0E4 +------------------------------------------------------------------+ Michael Andres YaST Development ma@novell.com SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nuernberg) Maxfeldstrasse 5, D-90409 Nuernberg, Germany, ++49 (0)911 - 740 53-0 +------------------------------------------------------------------+ -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Michael Andres wrote:
Why class Repository? This requires to load the .solv into the pool, just to be able to access these data. Wouldn't you like to have this informatin in advance.
I would expect this to be offered by RepoInfo, retrieving the data directly from the .solv file / repomd.xml, if not yet loaded into the pool.
for ( all repoInfos )
if ( ! repoInfo.enabled() ) continue
refreshIfNeeded( repoInfo );
if ( repoInfo.maybeOutdated() && userWntsToSkipIt() ) continue
loadFromCache( repoInfo );
Valid point. The problem is we get that values from the metadata, so is the sat-solver who parses the value, and we don't want to break the clean design of those tools to write to a non-solv file. May be the solution is to output that data to a separate solv file, however, you still need to load the repo in the pool to look the attributes. Duncan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
On Sat, Jul 12, 2008 at 04:02:41AM +0200, Duncan Mac-Vicar P. wrote:
An attribute "expire" is added to repomd section:
<repomd expire="100000">
As in "repodata/primary.xml.gz expires in the year 100000"? Seconds? Since when? We should remember to keep http://en.opensuse.org/Standards/Rpm_Metadata/repomd updated. -- Martin Vidner, YaST developer http://en.opensuse.org/User:Mvidner Kuracke oddeleni v restauraci je jako fekalni oddeleni v bazenu -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
Martin Vidner wrote:
On Sat, Jul 12, 2008 at 04:02:41AM +0200, Duncan Mac-Vicar P. wrote:
An attribute "expire" is added to repomd section:
<repomd expire="100000">
As in "repodata/primary.xml.gz expires in the year 100000"? Seconds? Since when? We should remember to keep http://en.opensuse.org/Standards/Rpm_Metadata/repomd updated.
Well, I documented it in the parent page, because we have a extensions section there. have to specify the semantics (seconds) for example. Duncan -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org
participants (5)
-
Duncan Mac-Vicar P.
-
Duncan Mac-Vicar Prett
-
Jiri Srain
-
Martin Vidner
-
Michael Andres