Am 27.11.24 um 10:25 schrieb alessio.biancalana@suse.com:
But at least right now I don't think they have dedicated people for performance, and I'm not aware of anyone regularly doing this kind of work in the community.
If they exist, please step up. Howdy! The very first reply to this thread was mine, and it was pretty idiotic from my side not to introduce myself. I'm Alessio, I hack
Il giorno mer, 27/11/2024 alle 04.54 +0100, Aaron Puchert ha scritto: things on openSUSE when I have some free time, I maintain some packages and I try to champion all-things-observability both in the community _and_ in my day by day at SUSE. Thanks for stepping up! As Neal pointed out, right now eBPF profiling isn't really a thing because eBPF unwinding itself isn't really a thing.
Coming to GNOME, as Neal was reporting, Christian Hergert wrote a couple posts that were really eye-opening about this kind of system- wide profiling even being possible.
https://fedoramagazine.org/performance-profiling-in-fedora-linux/
Got it, so the idea is to profile something like a GUI application with lots of dependencies. Most of the work doesn't happen in the application itself, but in all kinds of libraries (fonts, rendering, IO, etc.) that all need FPs. That's different from my profiling, where the application itself sits on the CPU all the time. Do you think that SUSE will systematically use this in the long term? I know data center operators do this (Meta, Google, Netflix), because they run their software at a large scale, and want to reduce their server footprint. They have dedicated performance engineers. SUSE isn't quite in the same situation, because most workloads on the software that we build happens out of our sight. (You could profile OBS or openQA, but I'm not sure how useful that would be.) So would we systematically look at usage scenarios? Would you expect that package maintainers do profiling for their packages?
https://blogs.gnome.org/chergert/2022/12/31/frame-pointers-and-other-practic...
On the contrary, he also wrote something about system-wide profiling actually _being_ possible even without frame pointers _but_ the overhead on the observed system IMHO makes it almost pointless (sorry for the blunt judgement here):
https://blogs.gnome.org/chergert/2024/11/03/profiling-w-o-frame-pointers/ Here I have to agree with Richard, 10% overhead by itself is not much. You're not going to profile all the time, so the cost should be OK. And
Let's put the near term aside for now. What happens in the long term? When all distros have FP enabled, are we still going to see effort poured into out-of-band unwinding? Or is it just going to be written off? As I've written in reply to Andrii, in the compiler world a 2% regression is quite significant. So I'd like to see a way out. the overhead itself should also not distort that much. One issue with profiling overhead is that it can hide latency or affect caches and other CPU structures. But this depends on sampling frequency. If you have too much overhead, reduce the frequency and repeat your workload until you have enough samples for a statistically meaningful result.
I hope my contribution was valuable to you in some way. It was, thanks!