Dear OBS Devs,
I was wondering if there is a way to collect metrics from OBS. Of
particular interest to me are build queue length, number of
in-progress package builds and regular CPU/memory usage metrics.
In my use case, I want to use the metrics to let Kubernetes determine
automatically how many workers to deploy. This works almost alright
with the limited metrics support that I already have from Kubernetes
itself, but there are some pitfalls and edge cases where Kubernetes
removes workers before they are fully done building a package.
My current situation is this:
I have a StatefulSet for the workers and a HorizontalPodAutoscaler
which will scale up the StatefulSet once CPU usage (as determined by
Kubernetes itself) exceeds the threshold (workers are no longer idle).
Due to long periods of IO bound operations where CPU usage may fall
below the threshold, the autoscaler needs to delay scale-down
operations by at least 30 min.
This is quite hacky and only really works as long as the build time of
the packages does not significantly exceed 30 minutes.
The downsides are that the scaling only happens once CPU usage
increases, which is usually quite late in the package building process
and that workers may be killed too early, if CPU usage at the end of
the build process is too low for too long.
If I had the build queue length and the number of in-progress package
builds as scrapeable metric,
I could utilize that to let Kubernetes determine the right amount of
workers more reliably.
Quobyte GmbH, Berlin, AG Charlottenburg HRB 149012 B, Felix Hupfeld,
Quobyte GmbH, Berlin, AG Charlottenburg HRB 149012 B, Dr. Felix Hupfeld,
Dr. Bjoern Kolbeck