https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c2 Michael Andres changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |dimstar@opensuse.org --- Comment #2 from Michael Andres 2014-09-09 12:12:40 CEST --- @Dominique: Actually i don't like to have too much knowledge about the application details coded in libzypp. So I'd suggest to use a plugin for this. (http://doc.opensuse.org/projects/libzypp/HEAD/zypp-plugins.html) - I'd provide a plugin hook within libzypp; called after a repos raw matadata were updated. Zypp could pass the path to the directory tree containing the new metadata and the repos type to the plugin. - Some gnome package had to provide the plugin script picking and consuming the appdata. Would that be ok? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c4 --- Comment #4 from Michael Andres 2014-09-10 14:40:13 CEST --- It might not work that way. Zypp can easily tell you that _a_repo was refreshed. But triggers will also be issued for temporary repos applications may create, or for repos which are just temporarily enabled for a specific task and otherwise disabled. _You_ need to know whether a repo is interesting for you. You can't determine or follow the set of active system repos by evaluating the repo triggers sketched above. System repos can also be created, modified and removed by directly editing the repo descriptions, which will cause no zypp trigger to be issued at all. Also creating/enabling a repo does not include downloading the repo-metadata. 'zypper lr' will list all enabled system repos, but their metadata are downloaded/updated only if some action actually requires or requests it. As long as no one else accesses the repo for you, you may not see the data. If you want to present relevant and up-to date data in your application, you should somehow retrieve the current state through libzypp. Maybe through a dedicated zypper command or via some python script. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c6 --- Comment #6 from Michael Andres 2014-09-11 16:32:40 CEST ---

Would that make things easier?

I try to explain...

As for 'temporary' repositories: I guess they should not be a problem: if it refreshes, we 'install' the appdata (which likely won't be found in the flat repo) and, as it's a temp repo, we delete it again on repo removal.

- Temp repos are never added, thus never removed, just refreshed - There are zypper commands which 'silently' scan through disabled repos (i.e. refresh them) to look for specific packages (e.g debuginfo packages) The coupling between system repository actions (add/remove/enable/disable/rename) and metadata download (refresh) is quite loose. The repo data download (refresh) in libzypp is using a repo description (URLs, a few settings, a bunch of cache directories consider). The origin of such a description is basically unknown to the downloader. It just tries to provide a consistent set of metadata based on the settings in the description. The purpose these data serve is also unknown. The 'system repos' are serialized repo descriptions. Repository actions (add/remove/enable/disable/rename) create/remove/modify these serialized descriptions. But you may also use cp, vi and rm. libzypp offers the 'system repos' descriptions to the application. But it's the application that picks/creates descriptions and passes them to the downloader according to it's needs. 'zypper --no-refresh' won't refresh any repo 'zypper --plus-repo' will refresh using a self created repo description 'zypper --plus-content' will also refresh disabled system repos If libzypp indicates a 'repo add' you should use the repo, but at that time only the serialized description exists. Your app won't get any data from the repo. However if libzypp indicates a 'refresh', it's not necessarily a repo that you want to use in your application. Can't you simply poll the list of currently active system repos and their current metadata path? ===[requires libzypp-bindings]=== #! /usr/bin/python import zypp for repo in zypp.RepoManager().knownRepositories(): if not repo.enabled(): continue print '%-40s %s' % ( repo.alias(), repo.metadataPath() ) ======== - not imported: import if metadata are already present (else e.g 'zypper ref') - imported: update (e.g. if metadata dir is newer than last import) - imported but not listed -> delete import (repo deleted/disabed). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c8 --- Comment #8 from Dominique Leuenberger 2014-09-12 14:05:52 UTC --- python-zypp is now in the mirrors.. so my very first attempt, that does something useful (yes, it is ugly): ====== #! /usr/bin/python import zypp import os for repo in zypp.RepoManager().knownRepositories(): if not repo.enabled(): continue print ("Installing appdata for repository %-40s..." % (repo.alias())) if str(repo.type()) == 'rpm-md': os.system("appstream-util install-origin %s %s/repodata/*appdata.xml.gz %s/repodata/*app-icons.tar.gz" % (repo.alias(), repo.metadataPath(), repo.metadataPath())) if str(repo.type()) == 'yast2': os.system("appstream-util install-origin %s %s/suse/setup/descr/*appdata.xml.gz %s/suse/setup/descr/*app-icons.tar.gz" % (repo.alias(), repo.metadataPath(), repo.metadataPath())) ====== If goes through all the repos and just tries to install all available data (appstream-util fails if not found; that's ok) What's still missing is a cleanup; I think simply cleaning up 'everything' before installing everything new would be the cleanest my shell magic for that is for file in /usr/share/app-info/xmls/*; do appstream-util uninstall $(echo $(basename $file) | sed "s:.xml.gz::"); done Seems if we manage to hook this into libzypp at the right point, this should be a workable solution in the end -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c10 --- Comment #10 from Dominique Leuenberger 2014-09-17 10:00:55 UTC --- (In reply to comment #9)

(In reply to comment #7)

...

- once zypp signals that 'all repos are refreshed' (I hope I can get this?

As mentioned above, the applications usually take the repo descriptions one by one and pass it to zypp in order to refresh. Zypp knows that _a_ repo was refreshend but not when _all_ were processed.

Is some zypp trigger required, or would gnome-software be able to execute the script at it's startup?

g-s runs as user and does not have the needed permission to install meta-data to the required location (/usr/share/app-info)

Another issue we may need to solve if we respond to a libzypp trigger, is the fact that the refreshing zypp instance might hold locks which prevent the script (if running as root) from getting the data at that point in time.

Maybe we can work around this by a file being touched upon 'any' refresh and triggering the external script when the locks disappear (I'm sure zypp knows when it's no longer locked)

...

Assuming the code extract above gives, as it indicates, the location where the metadata is stored, it should be rather simple... only question for me left at this moment:

- for rpm-md, I guess metadataPath() points to /var/cache/zypp/raw/<repo>/repodata (or to <repo>/?)

and for yast2? to /var/cache/zypp/raw/<repo>/ or somewhere deeper?

Metadata path is always the '/var/cache/zypp/raw/<repo>/' level. The files below metadata path are at the same location as they are on the media below repo URL:

<metadatapath>/

*rpmmd* /repodata/repomd.xml ...

*susetags* /content /suse/setup/descr/ ...

...

for type, i assume repo.type() would be the right thing to identify this as well.

Yes, but keep in mind that type is just a hint. Type is usually UNKNOWN until we probe the repo or we got the value from a serialized description. A known type does not imply metadata are available; UNKNOWN does not imply metadata are missing.

repos that have metadata cached locally are the only ones we care for and those always have a valid type.... good enough -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c12 --- Comment #12 from Dominique Leuenberger 2014-09-18 15:30:41 UTC --- (In reply to comment #11)

(In reply to comment #10)

...

g-s runs as user and does not have the needed permission to install meta-data to the required location (/usr/share/app-info)

Then g-s uses PackageKit? Why not using PK for refreshing the appdata?

g-s does... other similar software installers might or might not (appstream goes beyond PackageKit, see for example also listaller, using the same data set) Hence the actual goal to not only make PK work, but get a solution for the final thing. Additionally: if a user does zypper refresh, then PK uses the zypp internal refresh code and won't trigger refreshs anytime soon, thus putting a good risk that the meta data is different (and a forced refresh sounds like a performance penalty I'd like to avoid)

AFAIK zypp_refresh_cache is the location that loops over the 'interesting' repos and refreshes their metadata. After the loop all repos are up-to-date. Isn't this exactly where you want to update the appdata? This way g-s should see the same applications PK is able to deliver.

I will have another look here, to make sure we don't trigger this too often or risk of missing refreshes... but it will mean that any other tool might have to implement similar things to get such triggers to work too -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

http://bugzilla.novell.com/show_bug.cgi?id=866257 --- Comment #15 from Dominique Leuenberger --- Created attachment 608778 --> http://bugzilla.novell.com/attachment.cgi?id=608778&action=edit Install Appstrea metadata For reference: this is the script I work with so far on my system and it takes nicely care of installing repo data of all repos currently found on the system -- You are receiving this mail because: You are on the CC list for the bug.