Mailinglist Archive: opensuse-buildservice (176 mails)

< Previous Next >
Re: [opensuse-buildservice] Track statistics on the openSUSE staging process to gain feedback on changes
  • From: Jimmy Berry <jberry@xxxxxxxx>
  • Date: Wed, 01 Mar 2017 08:43:39 -0600
  • Message-id: <1675448.8Sf1SHiZoL@boomba.local>
On Wednesday, March 1, 2017 9:54:14 AM CST Adrian Schröter wrote:
On Mittwoch, 1. März 2017, 01:48:19 CET wrote Jimmy Berry:
I am looking to provide a variety of statistics and metrics relating to
the
staging workflow on OBS in onder to see the impact of automation and tune
the tools. I have spent some time setting up a local OBS, obs_factory
engine, looking through the database structure, and reviewing tools
designed to present metrics. My understanding from @hennevogel is that
Influx Data is planned to be used to provide this type of information and
it makes sense to get a feel for the existing plans and how this may fit
into that.

The biggest hurdle seems to be trying to create timeseries information
from
event data by walking the events backwards from the known state (ie
current) to find states of interest. At some point whatever solution is
built will need access to the relevant data to do some aggregation and
store intermediate results that can then be drawn on for presentation.
Alternatively, I had some success using sub-selects, but that is likely
not the most performant way forward.

Given the design of the submit request/staging workflow, that no events
are
recorded after a request is declined, it is not possible to determine when
a obsolete request was unstaged. The overall state of stagings cannot be
accurately determined when such a workflow has occurred. It will likely
also be difficult/impossible to determine when a build state complete,
re-entered building due to manual rebuilds or re-freeze, or failed/passed
testing. The dashboard [1] already provides this information, but
re-creating a history in an aggregate form is likely near impossible if
not very difficult. As such it likely makes sense to create a new polling
job that stores facets of the information collected by the dashboard in a
timerseries database.

For the rest of the information tied specific to individual requests it
seems possible to figure everything out from a handful of tables in the
OBS database.

We did this for the maintenance statistics:

osc api /statistics/maintenance_statistics/openSUSE:Maintenance:6433

it also handles assignments from a group to a user to calculate how
long the group took to review.

This looks somewhat similar although the example provided does not have any
reviews, but based on your comment I can assume what that might look like.
What tool consumes this API? Presumably the tool then has to scrape all the
information from this API in a similar manor to what I was trying to avoid.


Alternatively, I can write something that polls and scrapes data
using the APIs into local storage, but that seems like unnecessary extra
work. Is it feasible for me to be granted read access to the few tables
of interest or the Influx/similar tool setup with access so that I can
begin setting up some metrics of interest?

Your code is very much isolated, so I can't really judge about it.
Just one hint, your code runs in an environemnt and host which is critical
for our security. And also for all people using repositories from it.

So another service means another potential weakness, it would be good if
that does not need to run on our main server at least.

Any code I mention at this point is running on my local machine. If read
access to the source tables is not possible the code can be hosted entirely
separate from OBS.


A bit more on the topic of generating timeseries data. As an example,
consider presenting a graph of the request backlog against Factory over
time, the time until first staging, or the number of empty stagings over
time. The event information is collected in the form of reviews. Anytime
the request is staged or re-staged reviews for the particular staging
project are added or accepted. Accepting or declining the request
unfortunately stops future changes from being recorded which means the
staging tools cannot indicate when the request is removed from a staging,
but for simplicity it can be assumed complete after one of those states.

I think most of this is generic and not specific to staging projects.
So it would be good to extend the generic request system with statistics
IMHO.

That was my understanding as well, which is why I am not proceeding any
further until I get an understanding of any existing plans surrounding OBS
metrics.

Assuming one has a known state from which to start it should be possible
to
walk the event history and annotate states of interest. Given that the
current state can be queried which indicates what requests are currently
in a staging that provides a starting point. Assuming the script can be
stopped anytime it encounters an annotation again (or other mechanism)
the job can be run on some timely basis to annotate the desired
information.

The polling technique could be used to avoid all this walking the event
history which is simpler, but cannot backfill the data. As such it is
preferable to walk the event tree where possible.

I look forward to your thoughts.


[1] https://build.opensuse.org/project/staging_projects/openSUSE:Factory


--
Jimmy
--
To unsubscribe, e-mail: opensuse-buildservice+unsubscribe@xxxxxxxxxxxx
To contact the owner, e-mail: opensuse-buildservice+owner@xxxxxxxxxxxx

< Previous Next >
List Navigation
Follow Ups