[Bug 866257] New: Can't browse or install software via gnome-software
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c0 Summary: Can't browse or install software via gnome-software Classification: openSUSE Product: openSUSE Factory Version: 13.2 Milestone 0 Platform: x86-64 OS/Version: openSUSE 13.2 Status: NEW Severity: Normal Priority: P5 - None Component: GNOME AssignedTo: bnc-team-gnome@forge.provo.novell.com ReportedBy: Antoine.Saroufim@gmail.com QAContact: qa-bugs@suse.de Found By: --- Blocker: --- User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X) AppleWebKit/538.15 (KHTML, like Gecko) Safari/538.15 SUSE/13.2 (3.11.90) Epiphany/3.11.90 It's not possible to browse software using gnome-software. Installed apps are listed but new apps can't be browsed or searched for. The result for a search is always "No Application Found" even if the application is already installed and listed in the installed section. Reproducible: Always Steps to Reproduce: 1.open gnome-software 2.Try to search for software 3.Notice that you can't browse software, or see any categories Actual Results: Can't see software to install Expected Results: To be able to browse and install software pkcon refresh didn't help. Zypper works perfectly. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c1 Dominique Leuenberger <dimstar@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |dimstar@opensuse.org Component|GNOME |libzypp AssignedTo|bnc-team-gnome@forge.provo. |zypp-maintainers@forge.prov |novell.com |o.novell.com --- Comment #1 from Dominique Leuenberger <dimstar@opensuse.org> 2014-09-08 18:32:36 UTC --- The main issue here is that despite us having appdata in the repositories, this data is not made available on the right locations in the file systems. appstream-glib looks for the files in /var/cache/app-info/xmls and the corresponding icons set extracted to /var/cache/app-info/icons/<name_of_xml>/ zypp should ensure the data repo metadata is made available in the right spots (/var/cache/zypp/raw can't be used, as depending on repo type, the structure is different.. also, the icon tarball must be extracted) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c2 Michael Andres <ma@suse.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO InfoProvider| |dimstar@opensuse.org --- Comment #2 from Michael Andres <ma@suse.com> 2014-09-09 12:12:40 CEST --- @Dominique: Actually i don't like to have too much knowledge about the application details coded in libzypp. So I'd suggest to use a plugin for this. (http://doc.opensuse.org/projects/libzypp/HEAD/zypp-plugins.html) - I'd provide a plugin hook within libzypp; called after a repos raw matadata were updated. Zypp could pass the path to the directory tree containing the new metadata and the repos type to the plugin. - Some gnome package had to provide the plugin script picking and consuming the appdata. Would that be ok? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c3 Dominique Leuenberger <dimstar@opensuse.org> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW InfoProvider|dimstar@opensuse.org | --- Comment #3 from Dominique Leuenberger <dimstar@opensuse.org> 2014-09-09 20:12:32 UTC --- (In reply to comment #2)
@Dominique: Actually i don't like to have too much knowledge about the application details coded in libzypp. So I'd suggest to use a plugin for this. (http://doc.opensuse.org/projects/libzypp/HEAD/zypp-plugins.html)
- I'd provide a plugin hook within libzypp; called after a repos raw matadata were updated. Zypp could pass the path to the directory tree containing the new metadata and the repos type to the plugin.
- Some gnome package had to provide the plugin script picking and consuming the appdata.
Would that be ok?
I guess that would be fine. I would need hooks on: - Repo refresh (install new appdata into the right location) - Repo disable/enable (remove/install appdata) - Repo rename (uninstall old appdata/install new appdata) - Repo delete (uninstall appdata) I think that's about all the cases I would need to hook in. In any case I would need to know 'where' to find the raw cache data (/var/cache/zypp/raw/<reponame>, the 'repo type' (rpm-md has a repodata folder with all the content, while yast has the /suse/setup/descr structure) and the 'type' of action. The zypp hooks would be installed by gnome-software (or any package it depends on). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c4 --- Comment #4 from Michael Andres <ma@suse.com> 2014-09-10 14:40:13 CEST --- It might not work that way. Zypp can easily tell you that _a_repo was refreshed. But triggers will also be issued for temporary repos applications may create, or for repos which are just temporarily enabled for a specific task and otherwise disabled. _You_ need to know whether a repo is interesting for you. You can't determine or follow the set of active system repos by evaluating the repo triggers sketched above. System repos can also be created, modified and removed by directly editing the repo descriptions, which will cause no zypp trigger to be issued at all. Also creating/enabling a repo does not include downloading the repo-metadata. 'zypper lr' will list all enabled system repos, but their metadata are downloaded/updated only if some action actually requires or requests it. As long as no one else accesses the repo for you, you may not see the data. If you want to present relevant and up-to date data in your application, you should somehow retrieve the current state through libzypp. Maybe through a dedicated zypper command or via some python script. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c5 --- Comment #5 from Dominique Leuenberger <dimstar@opensuse.org> 2014-09-10 19:35:28 UTC --- Hmm. let's see if we can get this somehow together.. in essence, appstream-util does 'it all' for us.. we 'just' need to know when to start it with what parameters/ eg: appstream-util uninstall openSUSE:Factory Removes the 'old' meta data that has been installed appstream-util install-origin openSUSE:Factory \ /var/cache/zypp/raw/openSUSE\:Factory/suse/setup/descr/appdata.xml.gz \ /var/cache/zypp/raw/openSUSE\:Factory/suse/setup/descr/app-icons.tar.gz this will do 'the right thing(tm)' and copy the files where they belong (so this would need to be launched on repo refresh) so in essence: whenever the repo is refreshed, those two commands (with appropriate names) would need to be launched. I think the repo disable/enable case can be ignored for now (PK will also ignore those repos for package search... so should be good). so to summarize: - On repository add: no action - On repository refresh: execute the two above commands (uninstall 'fails' if not present.. we'd need to ignore that) - On repo disable/enable: no action (PK ignores the repos as well) - on repo rename: that's a bit trickier I guess... I'm tempted to ignore that case to start with :) - on repo delete: call the uninstall command (clean up the cache) Would that make things easier? As for 'temporary' repositories: I guess they should not be a problem: if it refreshes, we 'install' the appdata (which likely won't be found in the flat repo) and, as it's a temp repo, we delete it again on repo removal. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c6 --- Comment #6 from Michael Andres <ma@suse.com> 2014-09-11 16:32:40 CEST ---
Would that make things easier?
I try to explain...
As for 'temporary' repositories: I guess they should not be a problem: if it refreshes, we 'install' the appdata (which likely won't be found in the flat repo) and, as it's a temp repo, we delete it again on repo removal.
- Temp repos are never added, thus never removed, just refreshed - There are zypper commands which 'silently' scan through disabled repos (i.e. refresh them) to look for specific packages (e.g debuginfo packages) The coupling between system repository actions (add/remove/enable/disable/rename) and metadata download (refresh) is quite loose. The repo data download (refresh) in libzypp is using a repo description (URLs, a few settings, a bunch of cache directories consider). The origin of such a description is basically unknown to the downloader. It just tries to provide a consistent set of metadata based on the settings in the description. The purpose these data serve is also unknown. The 'system repos' are serialized repo descriptions. Repository actions (add/remove/enable/disable/rename) create/remove/modify these serialized descriptions. But you may also use cp, vi and rm. libzypp offers the 'system repos' descriptions to the application. But it's the application that picks/creates descriptions and passes them to the downloader according to it's needs. 'zypper --no-refresh' won't refresh any repo 'zypper --plus-repo' will refresh using a self created repo description 'zypper --plus-content' will also refresh disabled system repos If libzypp indicates a 'repo add' you should use the repo, but at that time only the serialized description exists. Your app won't get any data from the repo. However if libzypp indicates a 'refresh', it's not necessarily a repo that you want to use in your application. Can't you simply poll the list of currently active system repos and their current metadata path? ===[requires libzypp-bindings]=== #! /usr/bin/python import zypp for repo in zypp.RepoManager().knownRepositories(): if not repo.enabled(): continue print '%-40s %s' % ( repo.alias(), repo.metadataPath() ) ======== - not imported: import if metadata are already present (else e.g 'zypper ref') - imported: update (e.g. if metadata dir is newer than last import) - imported but not listed -> delete import (repo deleted/disabed). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c7 --- Comment #7 from Dominique Leuenberger <dimstar@opensuse.org> 2014-09-12 10:41:15 UTC --- Thanks for the tip! That might actually work. - once zypp signals that 'all repos are refreshed' (I hope I can get this?) an external script could launch, ensuring to install the latest metadata from the meta datas into the right locations. Can't test your snippet above yet, as libzypp-bindings is 'too new in Factory' (It's actually not yet on the mirror). Assuming the code extract above gives, as it indicates, the location where the metadata is stored, it should be rather simple... only question for me left at this moment: - for rpm-md, I guess metadataPath() points to /var/cache/zypp/raw/<repo>/repodata (or to <repo>/?) and for yast2? to /var/cache/zypp/raw/<repo>/ or somewhere deeper? (doc for zypp is generally a bit 'sparse'..for metadataPath, it states: Pathname zypp::RepoInfo::metadataPath ( ) const Path where this repo metadata was read from. Note could be an empty pathname for repo infos created in memory. ) for type, i assume repo.type() would be the right thing to identify this as well. Anyway: once zypp-bindings hits the mirrors in the next factory sync, I will try to write a script that 'does the magic' of installing the stuff.. then we only need to hook it up to libzypp in the end. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c8 --- Comment #8 from Dominique Leuenberger <dimstar@opensuse.org> 2014-09-12 14:05:52 UTC --- python-zypp is now in the mirrors.. so my very first attempt, that does something useful (yes, it is ugly): ====== #! /usr/bin/python import zypp import os for repo in zypp.RepoManager().knownRepositories(): if not repo.enabled(): continue print ("Installing appdata for repository %-40s..." % (repo.alias())) if str(repo.type()) == 'rpm-md': os.system("appstream-util install-origin %s %s/repodata/*appdata.xml.gz %s/repodata/*app-icons.tar.gz" % (repo.alias(), repo.metadataPath(), repo.metadataPath())) if str(repo.type()) == 'yast2': os.system("appstream-util install-origin %s %s/suse/setup/descr/*appdata.xml.gz %s/suse/setup/descr/*app-icons.tar.gz" % (repo.alias(), repo.metadataPath(), repo.metadataPath())) ====== If goes through all the repos and just tries to install all available data (appstream-util fails if not found; that's ok) What's still missing is a cleanup; I think simply cleaning up 'everything' before installing everything new would be the cleanest my shell magic for that is for file in /usr/share/app-info/xmls/*; do appstream-util uninstall $(echo $(basename $file) | sed "s:.xml.gz::"); done Seems if we manage to hook this into libzypp at the right point, this should be a workable solution in the end -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c9 --- Comment #9 from Michael Andres <ma@suse.com> 2014-09-15 10:59:09 CEST --- (In reply to comment #7)
- once zypp signals that 'all repos are refreshed' (I hope I can get this?
As mentioned above, the applications usually take the repo descriptions one by one and pass it to zypp in order to refresh. Zypp knows that _a_ repo was refreshend but not when _all_ were processed. Is some zypp trigger required, or would gnome-software be able to execute the script at it's startup? Another issue we may need to solve if we respond to a libzypp trigger, is the fact that the refreshing zypp instance might hold locks which prevent the script (if running as root) from getting the data at that point in time.
Assuming the code extract above gives, as it indicates, the location where the metadata is stored, it should be rather simple... only question for me left at this moment:
- for rpm-md, I guess metadataPath() points to /var/cache/zypp/raw/<repo>/repodata (or to <repo>/?)
and for yast2? to /var/cache/zypp/raw/<repo>/ or somewhere deeper?
Metadata path is always the '/var/cache/zypp/raw/<repo>/' level. The files below metadata path are at the same location as they are on the media below repo URL: <metadatapath>/ *rpmmd* /repodata/repomd.xml ... *susetags* /content /suse/setup/descr/ ...
for type, i assume repo.type() would be the right thing to identify this as well.
Yes, but keep in mind that type is just a hint. Type is usually UNKNOWN until we probe the repo or we got the value from a serialized description. A known type does not imply metadata are available; UNKNOWN does not imply metadata are missing. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c10 --- Comment #10 from Dominique Leuenberger <dimstar@opensuse.org> 2014-09-17 10:00:55 UTC --- (In reply to comment #9)
(In reply to comment #7)
- once zypp signals that 'all repos are refreshed' (I hope I can get this?
As mentioned above, the applications usually take the repo descriptions one by one and pass it to zypp in order to refresh. Zypp knows that _a_ repo was refreshend but not when _all_ were processed.
Is some zypp trigger required, or would gnome-software be able to execute the script at it's startup?
g-s runs as user and does not have the needed permission to install meta-data to the required location (/usr/share/app-info)
Another issue we may need to solve if we respond to a libzypp trigger, is the fact that the refreshing zypp instance might hold locks which prevent the script (if running as root) from getting the data at that point in time.
Maybe we can work around this by a file being touched upon 'any' refresh and triggering the external script when the locks disappear (I'm sure zypp knows when it's no longer locked)
Assuming the code extract above gives, as it indicates, the location where the metadata is stored, it should be rather simple... only question for me left at this moment:
- for rpm-md, I guess metadataPath() points to /var/cache/zypp/raw/<repo>/repodata (or to <repo>/?)
and for yast2? to /var/cache/zypp/raw/<repo>/ or somewhere deeper?
Metadata path is always the '/var/cache/zypp/raw/<repo>/' level. The files below metadata path are at the same location as they are on the media below repo URL:
<metadatapath>/
*rpmmd* /repodata/repomd.xml ...
*susetags* /content /suse/setup/descr/ ...
for type, i assume repo.type() would be the right thing to identify this as well.
Yes, but keep in mind that type is just a hint. Type is usually UNKNOWN until we probe the repo or we got the value from a serialized description. A known type does not imply metadata are available; UNKNOWN does not imply metadata are missing.
repos that have metadata cached locally are the only ones we care for and those always have a valid type.... good enough -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c11 --- Comment #11 from Michael Andres <ma@suse.com> 2014-09-18 17:20:22 CEST --- (In reply to comment #10)
g-s runs as user and does not have the needed permission to install meta-data to the required location (/usr/share/app-info)
Then g-s uses PackageKit? Why not using PK for refreshing the appdata? AFAIK zypp_refresh_cache is the location that loops over the 'interesting' repos and refreshes their metadata. After the loop all repos are up-to-date. Isn't this exactly where you want to update the appdata? This way g-s should see the same applications PK is able to deliver. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c12 --- Comment #12 from Dominique Leuenberger <dimstar@opensuse.org> 2014-09-18 15:30:41 UTC --- (In reply to comment #11)
(In reply to comment #10)
g-s runs as user and does not have the needed permission to install meta-data to the required location (/usr/share/app-info)
Then g-s uses PackageKit? Why not using PK for refreshing the appdata?
g-s does... other similar software installers might or might not (appstream goes beyond PackageKit, see for example also listaller, using the same data set) Hence the actual goal to not only make PK work, but get a solution for the final thing. Additionally: if a user does zypper refresh, then PK uses the zypp internal refresh code and won't trigger refreshs anytime soon, thus putting a good risk that the meta data is different (and a forced refresh sounds like a performance penalty I'd like to avoid)
AFAIK zypp_refresh_cache is the location that loops over the 'interesting' repos and refreshes their metadata. After the loop all repos are up-to-date. Isn't this exactly where you want to update the appdata? This way g-s should see the same applications PK is able to deliver.
I will have another look here, to make sure we don't trigger this too often or risk of missing refreshes... but it will mean that any other tool might have to implement similar things to get such triggers to work too -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=866257 https://bugzilla.novell.com/show_bug.cgi?id=866257#c13 --- Comment #13 from Michael Andres <ma@suse.com> 2014-09-19 14:53:54 CEST ---
Hence the actual goal to not only make PK work, but get a solution for the final thing.
So back to the python snippet? The python snippet 'knows' about the system repos and it can feed the 'final thing'. Without performing a refresh by itself, it could check and sync the latest apdata state into the 'final thing' quite fast if there's nothing todo (and without need to care about any zypp lock). A 'repo refresh' - if performed - just tells you that something in the repo changed. I assume the appdata in a repo change far less frequently than the repos package data. So updating the 'final thing' with each repo refresh is probably superfluous. The python snippet could remember size/md5 sum of the appdata files it already fed, and feed only changed stuff/add new/remove outdated. If nothing has changed, there's not much to do. You could think about putting it into /etc/cron.hourly and it will pick up the latest state once an hour... ...while I'm looking for a place in libzypp to trigger the script. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
http://bugzilla.novell.com/show_bug.cgi?id=866257 --- Comment #15 from Dominique Leuenberger <dimstar@opensuse.org> --- Created attachment 608778 --> http://bugzilla.novell.com/attachment.cgi?id=608778&action=edit Install Appstrea metadata For reference: this is the script I work with so far on my system and it takes nicely care of installing repo data of all repos currently found on the system -- You are receiving this mail because: You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com