A few "Who/Where" questions ahead of a community REST API
Dear Heroes, As mentioned in the last Heroes meeting we (Jens, Martín and I) are looking into building a user-friendly REST API to allow users across the entire platforms (web, Matrix, Telegram, Discord, future forums) to perform a variety of queries. The ultimate goal is to make important community tools and services more accessible by providing a single, user-friendly interface where basically any question that doesn't need a human respondent can be addressed by a simple query. (That's half the of it; it other half is to make channels, people and activities more discoverable, but that's for another day.) The domain of resources we want to provide access to is evolving as we learn new things about oS infra but it goes roughly like: - packages listed under "zypper search" - packages displayed on build.opensuse.org - documentation - the opensue bugzilla - progress/pagure - community news & activities (from news-o-o) and more. Now before we start building new tools we'd like to see if we can use tools that exist already. Thus we would like to know a few things: 1) Since this API is likely to depend on bots to provide in-app searches, we need to know if the bots bridging between the various instant messaging services could be used to forward queries to the API. Who's maintaining that / these bots and where can read the code? 2) Bugzilla comes with a REST API from version 5+; does bugzilla.opensuse.org have this feature currently enabled? Who should we get in touch with to negotiate enabling it? 3) Does Progress expose an API for doing queries and if yes, where's the docs about using this feature? Thanks a lot, any piece of information would be greatly appreciated! Best, Adrien
On Mon, Jul 12, 2021 at 4:23 AM Adrien Glauser <adrien.glauser@gmail.com> wrote:
Dear Heroes,
As mentioned in the last Heroes meeting we (Jens, Martín and I) are looking into building a user-friendly REST API to allow users across the entire platforms (web, Matrix, Telegram, Discord, future forums) to perform a variety of queries. The ultimate goal is to make important community tools and services more accessible by providing a single, user-friendly interface where basically any question that doesn't need a human respondent can be addressed by a simple query. (That's half the of it; it other half is to make channels, people and activities more discoverable, but that's for another day.)
The domain of resources we want to provide access to is evolving as we learn new things about oS infra but it goes roughly like: - packages listed under "zypper search" - packages displayed on build.opensuse.org - documentation - the opensue bugzilla - progress/pagure - community news & activities (from news-o-o) and more.
Now before we start building new tools we'd like to see if we can use tools that exist already. Thus we would like to know a few things:
1) Since this API is likely to depend on bots to provide in-app searches, we need to know if the bots bridging between the various instant messaging services could be used to forward queries to the API. Who's maintaining that / these bots and where can read the code?
Today, openSUSE Heroes has no bots in service beyond the bridge bots for the Matrix server.
2) Bugzilla comes with a REST API from version 5+; does bugzilla.opensuse.org have this feature currently enabled? Who should we get in touch with to negotiate enabling it?
We do not have Bugzilla 5. Sasi and I had been looking at trying to get SUSE IT to deploy a variant of the Red Hat Bugzilla 5 software[1] as it provides a variant of Bugzilla 5 with features that would be useful for distributions (including tracking third-party bug trackers and multi-SSO). Today, we have Bugzilla 4 with its XML-RPC API. I strongly recommend using python-bugzilla[2] to interface with it.
3) Does Progress expose an API for doing queries and if yes, where's the docs about using this feature?
Progress uses Redmine, however the project does not generally use this much, so it's not of much use to query for the project as a whole. That said, we *can* enable the REST API[3] if desired (if it's not already enabled, I'm not sure). Pagure has an API, and it's documentation is on the instance itself[4].
Thanks a lot, any piece of information would be greatly appreciated!
Hope this helps! [1]: https://pagure.io/Red-Hat-Bugzilla/rh-bugzilla [2]: https://github.com/python-bugzilla/python-bugzilla [3]: https://www.redmine.org/projects/redmine/wiki/rest_api [4]: https://code.opensuse.org/api -- 真実はいつも一つ！/ Always, there's only one truth!
Thank you very much Neal, this is immensely helpful!
Today, openSUSE Heroes has no bots in service beyond the bridge bots for the Matrix server.
Indeed, it is about the bridge bots that I am asking: Can we take a look at the code currently in production? It might be possible to use these bots as a point of entry for allowing users to consume the future API from any bridged channel while avoiding duplicate responses. And should we use another bot instead, looking at the bots currently in use would still be instructive.
We do not have Bugzilla 5. Sasi and I had been looking at trying to get SUSE IT to deploy a variant of the Red Hat Bugzilla 5 software[1] as it provides a variant of Bugzilla 5 with features that would be useful for distributions (including tracking third-party bug trackers and multi-SSO).
Today, we have Bugzilla 4 with its XML-RPC API. I strongly recommend using python-bugzilla[2] to interface with it.
3) Does Progress expose an API for doing queries and if yes, where's the docs about using this feature?
Progress uses Redmine, however the project does not generally use this much, so it's not of much use to query for the project as a whole. That said, we *can* enable the REST API[3] if desired (if it's not already enabled, I'm not sure). Pagure has an API, and it's documentation is on the instance itself[4].
Duly noted about Progress. May we request API keys for both Pagure and the openSUSE bugzilla right now? Or shall I open a ticket or something to make it more official? Gratefully yours, Adrien
On Mon, Jul 12, 2021 at 8:54 AM Adrien Glauser <adrien.glauser@gmail.com> wrote:
Thank you very much Neal, this is immensely helpful!
Today, openSUSE Heroes has no bots in service beyond the bridge bots for the Matrix server.
Indeed, it is about the bridge bots that I am asking: Can we take a look at the code currently in production? It might be possible to use these bots as a point of entry for allowing users to consume the future API from any bridged channel while avoiding duplicate responses. And should we use another bot instead, looking at the bots currently in use would still be instructive.
Sasi knows more here, but our deployment is documented in Salt: https://code.opensuse.org/heroes/salt/blob/production/f/pillar/role/matrix.s...
We do not have Bugzilla 5. Sasi and I had been looking at trying to get SUSE IT to deploy a variant of the Red Hat Bugzilla 5 software[1] as it provides a variant of Bugzilla 5 with features that would be useful for distributions (including tracking third-party bug trackers and multi-SSO).
Today, we have Bugzilla 4 with its XML-RPC API. I strongly recommend using python-bugzilla[2] to interface with it.
3) Does Progress expose an API for doing queries and if yes, where's the docs about using this feature?
Progress uses Redmine, however the project does not generally use this much, so it's not of much use to query for the project as a whole. That said, we *can* enable the REST API[3] if desired (if it's not already enabled, I'm not sure). Pagure has an API, and it's documentation is on the instance itself[4].
Duly noted about Progress. May we request API keys for both Pagure and the openSUSE bugzilla right now? Or shall I open a ticket or something to make it more official?
I don't know about Bugzilla, someone else on the team with more familiarity will have to chime in here. As for Pagure, API keys are self-service: https://code.opensuse.org/settings#nav-api-tab API keys are not required for read operations on public projects. -- 真実はいつも一つ！/ Always, there's only one truth!
On 7/12/21 10:28 PM, Neal Gompa wrote:
On Mon, Jul 12, 2021 at 8:54 AM Adrien Glauser <adrien.glauser@gmail.com> wrote:
Thank you very much Neal, this is immensely helpful!
Today, openSUSE Heroes has no bots in service beyond the bridge bots for the Matrix server.
Indeed, it is about the bridge bots that I am asking: Can we take a look at the code currently in production? It might be possible to use these bots as a point of entry for allowing users to consume the future API from any bridged channel while avoiding duplicate responses. And should we use another bot instead, looking at the bots currently in use would still be instructive.
Sasi knows more here, but our deployment is documented in Salt: https://code.opensuse.org/heroes/salt/blob/production/f/pillar/role/matrix.s...
We do not have Bugzilla 5. Sasi and I had been looking at trying to get SUSE IT to deploy a variant of the Red Hat Bugzilla 5 software[1] as it provides a variant of Bugzilla 5 with features that would be useful for distributions (including tracking third-party bug trackers and multi-SSO).
Today, we have Bugzilla 4 with its XML-RPC API. I strongly recommend using python-bugzilla[2] to interface with it.
3) Does Progress expose an API for doing queries and if yes, where's the docs about using this feature?
Progress uses Redmine, however the project does not generally use this much, so it's not of much use to query for the project as a whole. That said, we *can* enable the REST API[3] if desired (if it's not already enabled, I'm not sure). Pagure has an API, and it's documentation is on the instance itself[4].
Duly noted about Progress. May we request API keys for both Pagure and the openSUSE bugzilla right now? Or shall I open a ticket or something to make it more official?
I don't know about Bugzilla, someone else on the team with more familiarity will have to chime in here.
From memory python-bugzilla uses user auth rather then an API key, I think we handle bots by just creating a separate IDP account for them and logging in with that, its been a couple of years since I have used it though. Also be warned it isn't great for handling large number of requests at one time. -- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
Hey Simons, thanks for this addendum. Who should I get in touch with to create this IDP account? Also, we anticipate that implementing a good caching strategy will be our main job in the short run. Are there /Redis/ instances that SUSE or openSUSE could provide us with? Setting up our own server or using a cloud solution is not likely to cut it... Thanks, Adrien Le 14/07/2021 à 08:06, Simon Lees a écrit :
On Mon, Jul 12, 2021 at 8:54 AM Adrien Glauser <adrien.glauser@gmail.com> wrote:
Thank you very much Neal, this is immensely helpful!
Today, openSUSE Heroes has no bots in service beyond the bridge bots for the Matrix server. Indeed, it is about the bridge bots that I am asking: Can we take a look at the code currently in production? It might be possible to use these bots as a point of entry for allowing users to consume the future API from any bridged channel while avoiding duplicate responses. And should we use another bot instead, looking at the bots currently in use would still be instructive.
Sasi knows more here, but our deployment is documented in Salt: https://code.opensuse.org/heroes/salt/blob/production/f/pillar/role/matrix.s...
We do not have Bugzilla 5. Sasi and I had been looking at trying to get SUSE IT to deploy a variant of the Red Hat Bugzilla 5 software[1] as it provides a variant of Bugzilla 5 with features that would be useful for distributions (including tracking third-party bug trackers and multi-SSO).
Today, we have Bugzilla 4 with its XML-RPC API. I strongly recommend using python-bugzilla[2] to interface with it.
3) Does Progress expose an API for doing queries and if yes, where's the docs about using this feature?
Progress uses Redmine, however the project does not generally use this much, so it's not of much use to query for the project as a whole. That said, we *can* enable the REST API[3] if desired (if it's not already enabled, I'm not sure). Pagure has an API, and it's documentation is on the instance itself[4]. Duly noted about Progress. May we request API keys for both Pagure and the openSUSE bugzilla right now? Or shall I open a ticket or something to make it more official?
I don't know about Bugzilla, someone else on the team with more familiarity will have to chime in here. From memory python-bugzilla uses user auth rather then an API key, I
On 7/12/21 10:28 PM, Neal Gompa wrote: think we handle bots by just creating a separate IDP account for them and logging in with that, its been a couple of years since I have used it though. Also be warned it isn't great for handling large number of requests at one time.
On 7/15/21 6:33 PM, Adrien Glauser wrote:
Hey Simons, thanks for this addendum. Who should I get in touch with to create this IDP account?
This is just a standard bugzilla / obs account, all you should need is an email address, obviously if its for a community service it would be best if its an email address that multiple people in the community can eventually have access to.
Also, we anticipate that implementing a good caching strategy will be our main job in the short run. Are there /Redis/ instances that SUSE or openSUSE could provide us with? Setting up our own server or using a cloud solution is not likely to cut it...
Ill leave this question for someone else on the list, its something I know nothing about, I know about bugzilla because my team uses it at times for some internal scripts.
Le 14/07/2021 à 08:06, Simon Lees a écrit :
On Mon, Jul 12, 2021 at 8:54 AM Adrien Glauser <adrien.glauser@gmail.com> wrote:
Thank you very much Neal, this is immensely helpful!
Today, openSUSE Heroes has no bots in service beyond the bridge bots for the Matrix server. Indeed, it is about the bridge bots that I am asking: Can we take a look at the code currently in production? It might be possible to use these bots as a point of entry for allowing users to consume the future API from any bridged channel while avoiding duplicate responses. And should we use another bot instead, looking at the bots currently in use would still be instructive.
Sasi knows more here, but our deployment is documented in Salt: https://code.opensuse.org/heroes/salt/blob/production/f/pillar/role/matrix.s...
We do not have Bugzilla 5. Sasi and I had been looking at trying to get SUSE IT to deploy a variant of the Red Hat Bugzilla 5 software[1] as it provides a variant of Bugzilla 5 with features that would be useful for distributions (including tracking third-party bug trackers and multi-SSO).
Today, we have Bugzilla 4 with its XML-RPC API. I strongly recommend using python-bugzilla[2] to interface with it.
3) Does Progress expose an API for doing queries and if yes, where's the docs about using this feature?
Progress uses Redmine, however the project does not generally use this much, so it's not of much use to query for the project as a whole. That said, we *can* enable the REST API[3] if desired (if it's not already enabled, I'm not sure). Pagure has an API, and it's documentation is on the instance itself[4]. Duly noted about Progress. May we request API keys for both Pagure and the openSUSE bugzilla right now? Or shall I open a ticket or something to make it more official?
I don't know about Bugzilla, someone else on the team with more familiarity will have to chime in here. From memory python-bugzilla uses user auth rather then an API key, I
On 7/12/21 10:28 PM, Neal Gompa wrote: think we handle bots by just creating a separate IDP account for them and logging in with that, its been a couple of years since I have used it though. Also be warned it isn't great for handling large number of requests at one time.
-- Simon Lees (Simotek) http://simotek.net Emergency Update Team keybase.io/simotek SUSE Linux Adelaide Australia, UTC+10:30 GPG Fingerprint: 5B87 DB9D 88DC F606 E489 CEC5 0922 C246 02F0 014B
Thanks Simon, you were correct in that an ordinary user account suffices to login against the XML-RPC Web Worker. I checked whether all methods exposed by https://github.com/python-bugzilla/python-bugzilla work and I stumbled into one pretty annoying blocker: the 'quicksearch' query parameter does not seem to be handled by the instance. In other words, ``` query = bzapi.build_query( quicksearch="snapper", include_fields=["id", "summary"]) bugs = bzapi.query(query) <- this throws 'xmlrpc.client.Fault: <Fault 53: 'quicksearch is not a valid parameter for the Bugzilla::Bug::match function.'>' ``` Any clue? Or any clue about someone who would have a clue? Best, and thanks, Adrien Le 19/07/2021 à 02:32, Simon Lees a écrit :
On 7/15/21 6:33 PM, Adrien Glauser wrote:
Hey Simons, thanks for this addendum. Who should I get in touch with to create this IDP account? This is just a standard bugzilla / obs account, all you should need is an email address, obviously if its for a community service it would be best if its an email address that multiple people in the community can eventually have access to.
Also, we anticipate that implementing a good caching strategy will be our main job in the short run. Are there /Redis/ instances that SUSE or openSUSE could provide us with? Setting up our own server or using a cloud solution is not likely to cut it... Ill leave this question for someone else on the list, its something I know nothing about, I know about bugzilla because my team uses it at times for some internal scripts.
Le 14/07/2021 à 08:06, Simon Lees a écrit :
On Mon, Jul 12, 2021 at 8:54 AM Adrien Glauser <adrien.glauser@gmail.com> wrote:
Thank you very much Neal, this is immensely helpful!
Today, openSUSE Heroes has no bots in service beyond the bridge bots for the Matrix server. Indeed, it is about the bridge bots that I am asking: Can we take a look at the code currently in production? It might be possible to use these bots as a point of entry for allowing users to consume the future API from any bridged channel while avoiding duplicate responses. And should we use another bot instead, looking at the bots currently in use would still be instructive.
Sasi knows more here, but our deployment is documented in Salt: https://code.opensuse.org/heroes/salt/blob/production/f/pillar/role/matrix.s...
> We do not have Bugzilla 5. Sasi and I had been looking at trying to > get SUSE IT to deploy a variant of the Red Hat Bugzilla 5 software[1] > as it provides a variant of Bugzilla 5 with features that would be > useful for distributions (including tracking third-party bug trackers > and multi-SSO). > > Today, we have Bugzilla 4 with its XML-RPC API. I strongly recommend > using python-bugzilla[2] to interface with it. > > 3) Does Progress expose an API for doing queries and if yes, where's the > docs about using this feature? > > Progress uses Redmine, however the project does not generally use this > much, so it's not of much use to query for the project as a whole. > That said, we *can* enable the REST API[3] if desired (if it's not > already enabled, I'm not sure). Pagure has an API, and it's > documentation is on the instance itself[4]. Duly noted about Progress. May we request API keys for both Pagure and the openSUSE bugzilla right now? Or shall I open a ticket or something to make it more official?
I don't know about Bugzilla, someone else on the team with more familiarity will have to chime in here. From memory python-bugzilla uses user auth rather then an API key, I
On 7/12/21 10:28 PM, Neal Gompa wrote: think we handle bots by just creating a separate IDP account for them and logging in with that, its been a couple of years since I have used it though. Also be warned it isn't great for handling large number of requests at one time.
On 19/07/2021 10.51, Adrien Glauser wrote:
I checked whether all methods exposed by https://github.com/python-bugzilla/python-bugzilla work and I stumbled into one pretty annoying blocker: the 'quicksearch' query parameter does not seem to be handled by the instance. In other words,
``` query = bzapi.build_query( quicksearch="snapper", include_fields=["id", "summary"]) bugs = bzapi.query(query) <- this throws 'xmlrpc.client.Fault: <Fault 53: 'quicksearch is not a valid parameter for the Bugzilla::Bug::match function.'>' ```
Any clue? Or any clue about someone who would have a clue?
https://www.bugzilla.org/docs/4.4/en/html/api/Bugzilla/WebService/Bug.html documents the available API and it does not mention quicksearch. Maybe the client side expects a newer bugzilla server version (that is in planning, but could still take a while)
You are correct Bernhard! I supposed the current version exposed this method because there is something very much like it usable from the web interface, but I was wrong. That's quite a significant setback though. Since the plain `search` method is not fuzzy and heavily parameterized, `quicksearch` was *the* method that made querying from a "poor context" (CLI or a Matrix bot) possible. I guess we'll have to abandon querying the oS Bugzilla instance, then. Le 19/07/2021 à 11:19, Bernhard M. Wiedemann a écrit :
On 19/07/2021 10.51, Adrien Glauser wrote:
I checked whether all methods exposed by https://github.com/python-bugzilla/python-bugzilla work and I stumbled into one pretty annoying blocker: the 'quicksearch' query parameter does not seem to be handled by the instance. In other words,
``` query = bzapi.build_query( quicksearch="snapper", include_fields=["id", "summary"]) bugs = bzapi.query(query) <- this throws 'xmlrpc.client.Fault: <Fault 53: 'quicksearch is not a valid parameter for the Bugzilla::Bug::match function.'>' ```
Any clue? Or any clue about someone who would have a clue? https://www.bugzilla.org/docs/4.4/en/html/api/Bugzilla/WebService/Bug.html
documents the available API and it does not mention quicksearch. Maybe the client side expects a newer bugzilla server version (that is in planning, but could still take a while)
Sorry if I have asked this already: Is there a chance we get access to a *Reddis instance* from SUSE / openSUSE? It would be a central piece of the caching system. Thanks in advance and have a nice day, Adrien Le 19/07/2021 à 15:49, Adrien Glauser a écrit :
You are correct Bernhard! I supposed the current version exposed this method because there is something very much like it usable from the web interface, but I was wrong.
That's quite a significant setback though. Since the plain `search` method is not fuzzy and heavily parameterized, `quicksearch` was *the* method that made querying from a "poor context" (CLI or a Matrix bot) possible. I guess we'll have to abandon querying the oS Bugzilla instance, then.
Le 19/07/2021 à 11:19, Bernhard M. Wiedemann a écrit :
On 19/07/2021 10.51, Adrien Glauser wrote:
I checked whether all methods exposed by https://github.com/python-bugzilla/python-bugzilla work and I stumbled into one pretty annoying blocker: the 'quicksearch' query parameter does not seem to be handled by the instance. In other words,
``` query = bzapi.build_query( quicksearch="snapper", include_fields=["id", "summary"]) bugs = bzapi.query(query) <- this throws 'xmlrpc.client.Fault: <Fault 53: 'quicksearch is not a valid parameter for the Bugzilla::Bug::match function.'>' ```
Any clue? Or any clue about someone who would have a clue? https://www.bugzilla.org/docs/4.4/en/html/api/Bugzilla/WebService/Bug.html
documents the available API and it does not mention quicksearch. Maybe the client side expects a newer bugzilla server version (that is in planning, but could still take a while)
On 23/07/2021 10.57, Adrien Glauser wrote:
Sorry if I have asked this already: Is there a chance we get access to a *Reddis instance* from SUSE / openSUSE?
salt says, we run redis servers on mickey pagure01 nuka matomo mx1 mx2 but only the 2 on mx are listening on non-localhost IPs. So it is likely that we do not have centralized redis servers. And that might be OK as for caches you want the lowest possible latency and do not need backups that would be easier with centralization. If you really need another VM for redis, filing a ticket on progress.o.o and marking it for the suse-it group would be a good start (though we are rather busy these days and it won't get better soon with summer vacations)
Thanks again for your helpful answer Bernhard, ideally I would like to test with any of the two mx servers. What do I have to do in order to secure an access? Our tests don't require persistence or anything too exotic, I am using for now a cloud solution from RedisLabs with neither persistence nor backups. But I have to admit that having persistence would be great. Best, Adrien Le 23/07/2021 à 15:42, Bernhard M. Wiedemann a écrit :
On 23/07/2021 10.57, Adrien Glauser wrote:
Sorry if I have asked this already: Is there a chance we get access to a *Reddis instance* from SUSE / openSUSE?
salt says, we run redis servers on mickey pagure01 nuka matomo mx1 mx2
but only the 2 on mx are listening on non-localhost IPs.
So it is likely that we do not have centralized redis servers. And that might be OK as for caches you want the lowest possible latency and do not need backups that would be easier with centralization.
If you really need another VM for redis, filing a ticket on progress.o.o and marking it for the suse-it group would be a good start (though we are rather busy these days and it won't get better soon with summer vacations)
participants (4)
-
Adrien Glauser -
Bernhard M. Wiedemann -
Neal Gompa -
Simon Lees