[opensuse-buildservice] API query to return all _real_ source packages
Is there a variant of /source/<project> that will return either enough information in XML to filter client-side or pre-filter server-side to achieve the following. A list of _real_ source packages which excludes: - maintenance updates (revisions of the release source package) - multi-spec self links (Mesa-drivers to Mesa) - _multibuild entries (?view=info expands them) I had same filtering working on top of ?view=info which works well enough for Leap, but does not work well for maintenance projects and SLE (mix of both) as anticipated. Hacks like excluding sourceinfo entries with ':' or '.' end up not working (at least the '.'). Perhaps ':' is sufficient if no packages are allowed (or exist) with ':' in name. There are packages with '.' the name that are not maintenance updates (ex. python-jaraco.base which links python- jaraco.packaging). The proper way to differentiate a maintenance update package seems to be <releasename> in the package meta. <package name="GraphicsMagick.10066" project="openSUSE:Leap:15.0:Update"> ... <releasename>GraphicsMagick</releasename> </package> The problem is that information is not available in view=info nor does it seem searchable via /search/package[/id]. The search route also does not expand project links which is necessary for SLE. Even playing with checking the filename element does not work since image (ie. non-spec) packages and product packages end up with special cases. In conclusion, even with a rather complex client-side xpath query it is not possible to get the list of "real packages" without making lots of followup queries. Is there a proper way to query the list or can such a method be added? -- Jimmy -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Thursday, May 9, 2019 5:20:55 PM CDT Jimmy Berry wrote:
In conclusion, even with a rather complex client-side xpath query it is not possible to get the list of "real packages" without making lots of followup queries. Is there a proper way to query the list or can such a method be added?
Essentially what would be great is the the following. /source/<project>?maintenance=0&multibuild=0&multispec=0 or /source/<project>?real=1 The definition of real being packages to which one might submit an update for either my way of maintenance or direct submit request. One should not be updating a _multibuild psuedo package, nor a multispec sub-package, nor a maintenance update (submit another for the releasepackage). -- Jimmy -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Freitag, 10. Mai 2019, 00:38:13 CEST Jimmy Berry wrote:
On Thursday, May 9, 2019 5:20:55 PM CDT Jimmy Berry wrote:
In conclusion, even with a rather complex client-side xpath query it is not possible to get the list of "real packages" without making lots of followup queries. Is there a proper way to query the list or can such a method be added?
Essentially what would be great is the the following.
/source/<project>?maintenance=0&multibuild=0&multispec=0
or
/source/<project>?real=1
The definition of real being packages to which one might submit an update for either my way of maintenance or direct submit request.
I admit that I still struggle a bit of the definition of "real packag es" here... In first place I would say these are all packages which actually got build, no? So the ones below /build/$project. Because beside the listing of source package containers also the project configuration (_meta and _config) has an influence what get built and what not. Yes, it becomes a bit more difficult for maintenance_release projects since these projects are not supposed to get built, yes getting releases. Could you explain a bit more what you plan to do with the information of "real packages" ? We could do that also in a chat later today ....
One should not be updating a _multibuild psuedo package
Our tooling does not allow you to do that and it should guide you to the right package automatically .... (you may want to point out to some example where we could improve :)
, nor a multispec sub-package
mostly not, but there are exceptions so can not forbid that to 100%. However, I saw that we stopped to set the devel package definition on these packages. So a "osc branch" is not taking the right packge always anymore ...
, nor a maintenance update (submit another for the releasepackage).
yes, but our tooling is supposed to do the right thing here. It recognise it and creates maintenance requests instead. Again, there might be cases where we need to improve but some real life examples would be nice ... -- Adrian Schroeter SUSE Linux Products GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Friday, May 10, 2019 2:17:15 AM CDT Adrian Schröter wrote:
In first place I would say these are all packages which actually got build, no? So the ones below /build/$project. Because beside the listing of source package containers also the project configuration (_meta and _config) has an influence what get built and what not.
Yes, it becomes a bit more difficult for maintenance_release projects since these projects are not supposed to get built, yes getting releases.
Could you explain a bit more what you plan to do with the information of "real packages" ? We could do that also in a chat later today ....
My first e-mail explains exactly what information is desired. Trying to filter the list to remove "packages" that are implementation details and not the thing actually being maintained. Such a distriction is useful for filtering lists presented in interface we use as well as not wasted queries on "packages" that are no relevant to tools since they cannot be maintained. There is no need to define this in some abstract sense on which we agree. Want a mechanism to filter maintenance update packages, multibuild sub packages and multi-spec sub packages. Either all the information needs to be in a listing API or the filtering done server side. As a side note, you can in-fact submit an update to Mesa-drivers. It will not have the result you want, but it is not prevented. We have had packages be submitted backwards, especally a few confusingly named python ones. In a summary interface one does not want nor desire to see that dozen maintenance update packages for every package bloating the list. One wants to see Mesa, not Mesa, Mesa.123, Mesa.124, Mesa.125, ... Regardless of implementation OBS side there should be a way to differentiate in a listing API call. -- Jimmy -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Friday, May 10, 2019 9:51:38 AM CDT Jimmy Berry wrote:
My first e-mail explains exactly what information is desired. Trying to filter the list to remove "packages" that are implementation details and not the thing actually being maintained. Such a distriction is useful for filtering lists presented in interface we use as well as not wasted queries on "packages" that are no relevant to tools since they cannot be maintained.
To further illustrate the point roughly the same number of packages are available from beginning to end of SLE-12, but in OBS one sees ~3000 for SUSE:SLE-12:GA and over 15,000 for SUSE:SLE-12-SP4:GA mostly due to the implementation detail of maintenance updates (what is essentially a commit becomes a package). If one wants to see the status of zypper one is not particularly interested in looking at ~30 zyppers. Having this information presented as such in OBS is what it is, but a way to distinguish for external use is rather reasonable. For origin-manager specifically queries need to be performed for each package both on-demand and for expensive weekly summary. Short of making a query for every package there is no way to know if I can avoid bothering with zypper.895 vs zypper. I can make this extra query, but would seem rather reasonable to be able to filter the initial list of packages to ... real packages as definined in initial mail (regardless of how it is exposed in API). -- Jimmy -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
To make this even further convoluted /search/package[/id] appears to have enough information to filter client-side. The output contains <releasename> when it is a maintenance update and <bcntsynctag> when it is a multi-spec package. Unfortunately, the search route does not include inherited packages while the /source route does with ?view=info. Although the source route expands things are not sources (_multibuild psuedo packages) while search contains neither. A side topic is the multispec implementation of a package that links to another within same project makes for fun stuff like this: osc api '/search/package/id?match=@project="openSUSE:Factory"+and+devel/ @project=@project' <collection matches="334"> <package project='openSUSE:Factory' name='FreeCAD-test'/> <package project='openSUSE:Factory' name='Mesa-drivers'/> <package project='openSUSE:Factory' name='NetworkManager-branding-SLE'/> ... </collection> If one writes a tool, like repo-checker, which wants to notify maintainers about a problem with a specific binary it asks OBS what package the binary came from and if one of the above 334 packages gets the multi-spec implementation detail package and looks up the devel maintainer which is Factory not the real one which would requiring following the multi-spec link to _real_ source package and then following that back to devel project. The fact that these implementation details are exposed at all and worse inconsistently like they are makes writing tools almost impossible. Merging multi-spec into _multibuild so that they can optionally be handled the same way (generate psuedo packages with : in name) instead of linked container (while still supporting old style) would make this at least one step better. Then one can determine multibuild packages using : in name since reserved character and cases like repo-checker work as expected. It's also logicaly what you expect, I have two builds of Mesa: one for X and one for Y. I do not have Mesa and Mesa-drivers. Handling this via multibuild should be fairly simple: _multibuild: mesa, mesa-drivers mesa.spec mesa-drivers.spec if exists(multibuild_name + '.spec'): run that spec the same way multi-spec did else: current _multibuild behavior (variable to single spec) Still not a solution for the immediate problem of knowing the "kind" of package since the tooling will need to work for existing products even if this is cleaned up. -- Jimmy -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Freitag, 10. Mai 2019, 22:22:51 CEST Jimmy Berry wrote:
To make this even further convoluted /search/package[/id] appears to have enough information to filter client-side. The output contains <releasename> when it is a maintenance update
yes
and <bcntsynctag> when it is a multi-spec package.
but also in other cases.
Unfortunately, the search route does not include inherited packages while the /source route does with ?view=info. Although the source route expands things are not sources (_multibuild psuedo packages) while search contains neither.
A side topic is the multispec implementation of a package that links to another within same project makes for fun stuff like this:
osc api '/search/package/id?match=@project="openSUSE:Factory"+and+devel/ @project=@project'
<collection matches="334"> <package project='openSUSE:Factory' name='FreeCAD-test'/> <package project='openSUSE:Factory' name='Mesa-drivers'/> <package project='openSUSE:Factory' name='NetworkManager-branding-SLE'/> ... </collection>
If one writes a tool, like repo-checker, which wants to notify maintainers about a problem with a specific binary it asks OBS what package the binary came from and if one of the above 334 packages gets the multi-spec implementation detail package and looks up the devel maintainer which is Factory not the real one which would requiring following the multi-spec link to _real_ source package and then following that back to devel project.
The fact that these implementation details are exposed at all and worse inconsistently like they are makes writing tools almost impossible. Merging multi-spec into _multibuild so that they can optionally be handled the same way (generate psuedo packages with : in name) instead of linked container (while still supporting old style) would make this at least one step better.
first of all, _multibuild and multiple spec files via source links are two independend concepts. They can be even combined, only a factory policy forbids this atm. _multibuild is a definition inside of the source, while the package links are a decision on the project level.
Then one can determine multibuild packages using : in name since reserved character and cases like repo-checker work as expected. It's also logicaly what you expect, I have two builds of Mesa: one for X and one for Y. I do not have Mesa and Mesa-drivers.
Handling this via multibuild should be fairly simple:
_multibuild: mesa, mesa-drivers mesa.spec mesa-drivers.spec
if exists(multibuild_name + '.spec'): run that spec the same way multi-spec did else: current _multibuild behavior (variable to single spec)
Still not a solution for the immediate problem of knowing the "kind" of package since the tooling will need to work for existing products even if this is cleaned up.
We could indeed support specific build description files in _multibuild, just we need to make this most likely an explicit setting inside of the _multibuild xml. However, back to the original problem. Multibuild ========== My question regarding the purpose was because I think that you don't need to have this logic outside of OBS. It is even harmfull when we enhance our mechanics more (eg. further multibuild features), because it will break this logic again. Eg. instead of delivering specific search calls, we should do support instead osc branch $project $package:$multibuild_flavor or URLs like https://build.opensuse.org/package..../$package:$multibuild_flavor so it does not matter that you get the mutibuild source containers. This is why I keep asking what you want to do with that information... Local Links =========== Same regarding the project local links, you should be fine if all packages get the devel definition back. Meaning that in the package meta of Mesa- drivers you add again <devel package="Mesa"/> so operations like branch calls get automatically redirected to the right place. Yes, this is manual work here, but this is basically the main difference between _multibuild and local links. Maintenance Release Projects ============================ Last, we can indeed support some simplified listings for maintenance_release projects. We will need an extra option, but shouldn't be too complicate. Does this sound like a plan? -- Adrian Schroeter SUSE Linux Products GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Mon, 2019-05-13 at 08:31 +0200, Adrian Schröter wrote:
Local Links ===========
Same regarding the project local links, you should be fine if all packages get the devel definition back. Meaning that in the package meta of Mesa- drivers you add again
<devel package="Mesa"/>
so operations like branch calls get automatically redirected to the right place. Yes, this is manual work here, but this is basically the main difference between _multibuild and local links.
Add the info back?! Our staging accept tooling does create the linked packages as well as the meta info to hae the 'main' set as devel - and has been doing this for a long time. In case of Mesa-drivers. the meta is: <package name="Mesa-drivers" project="openSUSE:Factory"> <title>System for rendering interactive 3-D graphics</title> <description>Mesa is a 3-D graphics library with an API which[ .. stripped description, irrelevant here...]</description> <devel project="openSUSE:Factory" package="Mesa"/> <bcntsynctag>Mesa</bcntsynctag> </package> Any linked package not having this meta was most likely not accepted using the Staging tooling. Cheers, Dominique
On Montag, 13. Mai 2019, 11:14:58 CEST Dominique Leuenberger / DimStar wrote:
On Mon, 2019-05-13 at 08:31 +0200, Adrian Schröter wrote:
Local Links ===========
Same regarding the project local links, you should be fine if all packages get the devel definition back. Meaning that in the package meta of Mesa- drivers you add again
<devel package="Mesa"/>
so operations like branch calls get automatically redirected to the right place. Yes, this is manual work here, but this is basically the main difference between _multibuild and local links.
Add the info back?! Our staging accept tooling does create the linked packages as well as the meta info to hae the 'main' set as devel - and has been doing this for a long time. In case of Mesa-drivers. the meta is:
<package name="Mesa-drivers" project="openSUSE:Factory"> <title>System for rendering interactive 3-D graphics</title> <description>Mesa is a 3-D graphics library with an API which[ .. stripped description, irrelevant here...]</description> <devel project="openSUSE:Factory" package="Mesa"/> <bcntsynctag>Mesa</bcntsynctag> </package>
Any linked package not having this meta was most likely not accepted using the Staging tooling.
okay, so there should be no problems with the standard workflows, if you have that. It may be a problem inside of maintenance_release projects only, where we seem not to have that information .... -- Adrian Schroeter SUSE Linux Products GmbH, Maxfeldstr. 5, 90409 Nuernberg, Germany email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Monday, May 13, 2019 1:31:07 AM CDT Adrian Schröter wrote:
first of all, _multibuild and multiple spec files via source links are two independend concepts. They can be even combined, only a factory policy forbids this atm.
It was a side discussion, but definitely very similar concepts that could be simplified a lot on both sides by being combined regardless of some abstract way to utilize both simultaneously.
Multibuild ==========
My question regarding the purpose was because I think that you don't need to have this logic outside of OBS. It is even harmfull when we enhance our mechanics more (eg. further multibuild features), because it will break this logic again.
The purpose of this email was to request such functionality be in OBS. Otherwise, there would be no point in discussing as I would just modify the code to perform an additional 15,000+ queries.
so it does not matter that you get the mutibuild source containers. This is why I keep asking what you want to do with that information...
I was very clear about what the purpose of the information was in the original email. Both tools and interfaces (for humans) to review status or perform operations on all packages without being cluttered by all the extra psuedo- packages. Performing a bunch of expensive queries on all multi-build packages (or local links) which have the same sources, and thus the same result, is not helpful. What would be helpful is that the /source/<project> API do what the prefix implies and provide a listing of source containers which can then be further filtered to exclude local links and maintenance packages. Another example is a simple tool to email all maintainers of the packages they have within Leap before release. Ludwig runs this tool manually with custom e- mail text to give maintainers a final heads up before release. Running such a tool on a SLE project would result in 15,000 entries instead of 3,000. The origin-manager tooling would be useful to apply to maintenance projects, but runs into the same problem. For SLE, instead of taking an hour to run it takes 17 hours due to getting 15,000 entries instead of 3,000. Making the rest of the APIs handle the psuedopackages is nice, but filtering them to avoid the useless query entirely is much much better.
Maintenance Release Projects ============================
Last, we can indeed support some simplified listings for maintenance_release projects. We will need an extra option, but shouldn't be too complicate.
Does this sound like a plan?
It's not just for maintenance_release projects. SLE :GA projects contain all the maintenance packages due to the layering. Regardless of what is done to the web interface an API-level filter is needed. I specified exactly what is needed in the original email.
/source/<project>?maintenance=0&multibuild=0&multispec=0
However one words it or implements it is up to the implementor. -- Jimmy -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Monday, May 13, 2019 3:20:46 PM CDT Jimmy Berry wrote:
Regardless of what is done to the web interface an API-level filter is needed. I specified exactly what is needed in the original email.
/source/<project>?maintenance=0&multibuild=0&multispec=0
However one words it or implements it is up to the implementor.
Since there has been no indication this will be implemented I went ahead and wrote a workaround [1] that makes 0-2 API calls per package to make a determination, but there are still special-cases (that really shouldn't be special, but are represented differently) that are not handled. Either way it reduces the SP5 package clutter from 19,095 "packages" to 3,965. There is still other code that is utilizing /search/request[/id] that is expecting inherited packages, but clearly that does not work that will need to be refactored at some point. If OBS ends up with a server-side filter (or exposes in /source/<project> response XML for client-side filtering), let me know and I'll replace this to avoid 20k+ queries. [1] https://github.com/openSUSE/openSUSE-release-tools/pull/2014 -- Jimmy -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
Adrian Schröter schrieb:
On Freitag, 10. Mai 2019, 00:38:13 CEST Jimmy Berry wrote:
On Thursday, May 9, 2019 5:20:55 PM CDT Jimmy Berry wrote:
In conclusion, even with a rather complex client-side xpath query it is not possible to get the list of "real packages" without making lots of followup queries. Is there a proper way to query the list or can such a method be added?
Essentially what would be great is the the following.
/source/<project>?maintenance=0&multibuild=0&multispec=0
or
/source/<project>?real=1
The definition of real being packages to which one might submit an update for either my way of maintenance or direct submit request.
I admit that I still struggle a bit of the definition of "real packag es" here...
Let's ask a different question and use a naive approach to answer it for illustration: Which packages were released as online update for 15.1 already? $ osc ls openSUSE:Leap:15.1:Update 00Meta bzip2 bzip2.10237 evolution evolution.10239 graphviz graphviz-addons graphviz-addons.10241 graphviz.10241 libxslt libxslt-python libxslt-python.10243 libxslt.10243 openssl-1_0_0 openssl-1_0_0.10244 patchinfo.10237 patchinfo.10239 patchinfo.10241 patchinfo.10243 patchinfo.10244 patchinfo.10245 patchinfo.10246 patchinfo.10247 patchinfo.32bit.8555 patchinfo.8555 patchinfo.affects-package-manager.8555 patchinfo.broken.8555 patchinfo.feature.8555 patchinfo.interactive.8555 patchinfo.optional.8555 patchinfo.reboot.8555 patchinfo.relogin-suggested.8555 patchinfo.security.8555 postfix postfix.10245 sensors sensors.10246 speech-dispatcher speech-dispatcher.10247 update-test-trivial update-test-trivial.8555 What we expected to see: $ osc ls openSUSE:Leap:15.1:Update 00Meta bzip2 evolution graphviz libxslt openssl-1_0_0 postfix sensors speech-dispatcher update-test-trivial cu Ludwig -- (o_ Ludwig Nussel //\ V_/_ http://www.suse.com/ SUSE Linux GmbH, GF: Felix Imendörffer, Mary Higgins, Sri Rasiah, HRB 21284 (AG Nürnberg) -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On Thu, May 23, 2019 at 03:03:03PM +0200, Ludwig Nussel wrote:
What we expected to see:
$ osc ls openSUSE:Leap:15.1:Update 00Meta bzip2 evolution graphviz libxslt openssl-1_0_0 postfix sensors speech-dispatcher update-test-trivial
But that's not what maintenance wants to see when they do that call. Cheers, Michael. -- Michael Schroeder mls@suse.de SUSE LINUX GmbH, GF Jeff Hawn, HRB 16746 AG Nuernberg main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);} -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
On 5/23/19 3:10 PM, Michael Schroeder wrote:
On Thu, May 23, 2019 at 03:03:03PM +0200, Ludwig Nussel wrote:
What we expected to see:
$ osc ls openSUSE:Leap:15.1:Update 00Meta bzip2 evolution graphviz libxslt openssl-1_0_0 postfix sensors speech-dispatcher update-test-trivial
But that's not what maintenance wants to see when they do that call.
I think that's understood. But an API side filter to do what Jimmy does in the client is much easier to implement. Greetings, Stephan -- To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-buildservice+owner@opensuse.org
participants (6)
-
Adrian Schröter -
Dominique Leuenberger / DimStar -
Jimmy Berry -
Ludwig Nussel -
Michael Schroeder -
Stephan Kulow