https://bugzilla.novell.com/show_bug.cgi?id=850794 https://bugzilla.novell.com/show_bug.cgi?id=850794#c1 Borislav Petkov changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bpetkov@suse.com --- Comment #1 from Borislav Petkov 2013-11-18 10:24:31 UTC --- Hmm, you could try to trace your benchmark with perf on 12.3 and 13.1 and upload perf.data from both runs somewhere. Something like perf record -a <benchmark> and then move the created perf.data somewhere else by naming it properly. HTH. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=850794 https://bugzilla.novell.com/show_bug.cgi?id=850794#c3 --- Comment #3 from Yasha Gindikin 2013-11-18 17:13:50 UTC --- The result for openSUSE 12.3 is here: https://drive.google.com/file/d/0B79cj6_sE491YXVrLU4zQkFkQ3M/edit?usp=sharin... I simply booted from KDE live USB to perform the benchmark, with the results being only negligibly smaller than those for the system installed on the ssd. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=850794 https://bugzilla.novell.com/show_bug.cgi?id=850794#c5 --- Comment #5 from Borislav Petkov 2013-11-19 14:15:26 UTC --- Ok, let's see: 12.3: 57.40% MathKernel libiomp5.so [.] 0x00000000000183db 20.15% MathKernel libmkl_def.so [.] 0x0000000000099f5a 5.49% MathKernel MathKernel [.] 0x00000000001b16be 5.14% MathKernel [kernel.kallsyms] [k] 0xffffffff8118b190 4.83% MathKernel libmkl_vml_def.so [.] 0x0000000000065210 2.18% MathKernel libgmp.so.10 [.] 0x0000000000029a80 1.29% MathKernel libmkl_core.so [.] 0x00000000000d1de2 0.95% MathKernel libc-2.17.so [.] 0x0000000000076b80 0.75% swapper [processor] [k] 0x0000000000001961 0.70% MathKernel libm-2.17.so [.] 0x0000000000008530 The kernel symbol 0xffffffff8118b190 points within __fd_install which is part of the vfs connecting file descriptors to files, i.e. your application is doing a bunch of syscalls. 13.1: 57.05% MathKernel libiomp5.so [.] 0x000000000006c598 17.37% MathKernel libmkl_def.so [.] 0x0000000000099f6b 5.96% MathKernel [kernel.kallsyms] [k] 0xffffffff811d1f6a 4.94% MathKernel MathKernel [.] 0x00000000011ce004 4.21% MathKernel libmkl_vml_def.so [.] 0x000000000006522b 2.37% MathKernel libc-2.18.so [.] 0x000000000011df1e 1.90% MathKernel libgmp.so.10 [.] 0x0000000000029a47 1.13% MathKernel libmkl_core.so [.] 0x00000000000d1db6 1.08% swapper [processor] [k] 0x0000000000002293 The kernel symbol 0xffffffff811d1f6a points within locks_remove_flock which is also part of the file descriptors handling code in vfs. Everything else is userspace, I'm guessing part of mathematica. And the most amount of time you're spending is in the openmp library and I wouldn't wonder if it is querying the environment and using optimized code. And that querying is detecting different things on 12.3 and 13.1 And we also have two different glibc functions and I can see an increase glibc utilization on 13.1. So, I'd like to check whether the kernel is causing any differences here so can you run our good ole kernel build test as described below, on 12.3 and 13.1 and collect the perf output of both and post it here. That'll hopefully tell us something. For that, download the upstream 3.12 kernel sources. Unpack them and do: $ make allmodconfig This will resemble a distro config. Then once on 12.3 and once on 13.1 do: $ perf stat --repeat 10 -a --sync --pre pre-build-kernel.sh make -s -j64 bzImage with pre-build-kernel.sh contents being: --- #!/bin/bash make -s clean echo 3 > /proc/sys/vm/drop_caches --- Don't hesitate to ask if there are questions. HTH. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=850794 https://bugzilla.novell.com/show_bug.cgi?id=850794#c7 --- Comment #7 from Borislav Petkov 2013-11-19 17:43:18 UTC ---

1. perf does not understand the option --pre. Is it a misprint?

Oh, right, --pre is not in oS's version of perf probably. You can use the perf in 3.12, just cd into the kernel source dir of the downloaded 3.12, then do $ cd tools/perf/ $ make -j8 $ cd ../../ and then call the perf binary in there. I.e., /tools/perf/perf ...

2. where do I put the file pre-build-kernel.sh?

you can put it in the same directory where the kernel is or anywhere else you want: /tools/perf/perf stat --pre /home/yasha/bin/pre-build-kernel.sh ... would work too. It needs an absolute path so that perf can do system(3) on it. HTH. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=850794 https://bugzilla.novell.com/show_bug.cgi?id=850794#c9 --- Comment #9 from Borislav Petkov 2013-11-19 20:09:24 UTC --- (In reply to comment #8)

Created an attachment (id=568087) --> (http://bugzilla.novell.com/attachment.cgi?id=568087) [details] The output of ./tools/perf/perf stat --repeat 10 -a --sync --pre /root/pre-build-kernel.sh make -s -j64 bzImage

Thank you for your kind explanations. The console output for openSUSE 12.3 is attached to the comment.

Glad you liked them :-) Now I'd need you to do the exact same perf exercise on 13.1 too :-) Thanks. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=850794 https://bugzilla.novell.com/show_bug.cgi?id=850794#c13 --- Comment #13 from Yasha Gindikin 2013-11-20 14:14:00 UTC --- Considering Mathematica benchmark: I have made a clean fresh install of openSUSE 13.1, and you know what? The performance drop is gone. Well, almost. Mathematica default benchmark (Benchmark[]) gives 0.94, which is comparable to that of openSUSE 12.3. My own (highly parallelized) code runs 10% slower, but this drop-down is not as dramatic as the one observed previously. So, maybe we should close the bug, should we? If there remain questions pertaining to the general performance of openSUSE 13.1 kernel, I am nonetheless eager to help. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=850794 https://bugzilla.novell.com/show_bug.cgi?id=850794#c15 --- Comment #15 from Borislav Petkov 2013-11-21 21:59:59 UTC --- (in reply to comment #13)

Considering Mathematica benchmark: I have made a clean fresh install of openSUSE 13.1, and you know what? The performance drop is gone. Well, almost. Mathematica default benchmark (Benchmark[]) gives 0.94, which is comparable to that of openSUSE 12.3. My own (highly parallelized) code runs 10% slower, but this drop-down is not as dramatic as the one observed previously. So, maybe we should close the bug, should we?

Yes, we will :) Great, so some libs didn't get updated properly or whatever else happened. I must remember this for the next time to ask bug reporters to install afresh. But yeah, normally one can't know what gets and doesn't get overwritten during a distro upgrade (I'm assuming this is what you did the first time).

If there remain questions pertaining to the general performance of openSUSE 13.1 kernel, I am nonetheless eager to help.

Thanks but I can do it here too - basically I have to run the perf record workload on one of my test boxes here on 3.7 and 3.11 or maybe 3.12 and compare. Also have to check where we're spending the most time. Oh well, when there's time. For the record, hunting performance regressions in the kernel can be very error prone, time consuming and tedious business. (in reply to comment #14)

BTW, the perf stat results above both pertained to the fresh clean installations of openSUSE, where Mathematica performance issues were already absent.

Right. Thanks for your time. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=850794 https://bugzilla.novell.com/show_bug.cgi?id=850794#c17 Borislav Petkov changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |FIXED --- Comment #17 from Borislav Petkov 2013-11-21 22:02:50 UTC --- % -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.