[opensuse-kernel] threads and/or memory MUCH slower on 2.6.37 than 2.6.34 ?
Hi list, I was trying to check if updating to 2.6.37 on our openSUSE 11.3 nodes would bring any performance changes for our application (HPC CFD programs) and was surprised to see a big performance decrease on my test runs. Trying to follow this problem down I came to the following perl test line to show the differences. If I run the following perl line on the same hardware on openSUSE 11.3, with 2.6.34.7-0.5-default and with 2.6.37-6-default (from http://download.opensuse.org/repositories/Kernel:/stable/openSUSE_11.3/x86_6... ) I get VERY different results: On 2.6.34 it runs in about 10 secs. On 2.6.37 it takes at least 17 secs to run. Could someone else test this on another hardware? Please run a couple of times with 2.6.34 and then with 2.6.37. Do others also see this big differences? The system I am testing on is a dual Xeon X5650 @ 2.67 GHz (12 cores) with 24 GB memory. Please set $T to your number of cores. Perhaps you would also have to reduce the (1..9e6) if you have less than 2GB per core. time perl -e 'use threads; $T=12 ; foreach (1..$T) { $thr[$i++] = threads->create(sub { printf "I am thread %s\n", threads->tid(); foreach (1..9e6) { push(@a, sqrt(1234)/sin(1234)) } ; printf "thread %s finished.\n", threads->tid(); }); } foreach (0..$T-1) { $thr[$_]->join(); }' Many thanks for testing! Does anyone have an idea why this could be happening? Richard -- Richard Ems mail: Richard.Ems@Cape-Horn-Eng.com Cape Horn Engineering S.L. C/ Dr. J.J. Dómine 1, 5º piso 46011 Valencia Tel : +34 96 3242923 / Fax 924 http://www.cape-horn-eng.com -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
Could this BIG difference be just because 2.6.37 was compiled with CONFIG_HZ=1000 and 2.6.34 with CONFIG_HZ=250 ? That much impact? Anyone? Thanks, Richard -- Richard Ems mail: Richard.Ems@Cape-Horn-Eng.com Cape Horn Engineering S.L. C/ Dr. J.J. Dómine 1, 5º piso 46011 Valencia Tel : +34 96 3242923 / Fax 924 http://www.cape-horn-eng.com -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 8.2.2011 16:07, Richard Ems wrote:
Could this BIG difference be just because 2.6.37 was compiled with CONFIG_HZ=1000 and 2.6.34 with CONFIG_HZ=250 ? That much impact?
The -default flavor always had CONFIG_HZ=250, and the -desktop flavor had CONFIG_HZ=1000. So check if you are not comparing different flavors by accident. Michal -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 02/08/2011 04:12 PM, Michal Marek wrote:
On 8.2.2011 16:07, Richard Ems wrote:
Could this BIG difference be just because 2.6.37 was compiled with CONFIG_HZ=1000 and 2.6.34 with CONFIG_HZ=250 ? That much impact?
The -default flavor always had CONFIG_HZ=250, and the -desktop flavor had CONFIG_HZ=1000. So check if you are not comparing different flavors by accident.
Michal
Thank you for pointing this out. I was really comparing different flavor configs, but I am *almost* 100% sure I compared both default flavor on my .34 vs. .37 tests. I am rerunning things now again .... -- Richard Ems mail: Richard.Ems@Cape-Horn-Eng.com Cape Horn Engineering S.L. C/ Dr. J.J. Dómine 1, 5º piso 46011 Valencia Tel : +34 96 3242923 / Fax 924 http://www.cape-horn-eng.com -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 02/08/2011 04:41 PM, Richard Ems wrote:
On 02/08/2011 04:12 PM, Michal Marek wrote:
On 8.2.2011 16:07, Richard Ems wrote:
Could this BIG difference be just because 2.6.37 was compiled with CONFIG_HZ=1000 and 2.6.34 with CONFIG_HZ=250 ? That much impact?
The -default flavor always had CONFIG_HZ=250, and the -desktop flavor had CONFIG_HZ=1000. So check if you are not comparing different flavors by accident.
Michal
Thank you for pointing this out. I was really comparing different flavor configs, but I am *almost* 100% sure I compared both default flavor on my .34 vs. .37 tests. I am rerunning things now again ....
I can confirm that I still see the 80% slowdown, compared again on the same flavor kernel, both desktop ones running with CONFIG_HZ=250 . My first mail from today stating that CONFIG_HZ were different was just my error, comparing different flavor configs. The tests were ran (and reran today) with the same desktop flavor. Any ideas? Anyone? Michal? Thanks, Richard -- Richard Ems mail: Richard.Ems@Cape-Horn-Eng.com Cape Horn Engineering S.L. C/ Dr. J.J. Dómine 1, 5º piso 46011 Valencia Tel : +34 96 3242923 / Fax 924 http://www.cape-horn-eng.com -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
Hi, On Fri 04-02-11 14:12:12, Richard Ems wrote:
I was trying to check if updating to 2.6.37 on our openSUSE 11.3 nodes would bring any performance changes for our application (HPC CFD programs) and was surprised to see a big performance decrease on my test runs. Trying to follow this problem down I came to the following perl test line to show the differences.
If I run the following perl line on the same hardware on openSUSE 11.3, with 2.6.34.7-0.5-default and with 2.6.37-6-default (from http://download.opensuse.org/repositories/Kernel:/stable/openSUSE_11.3/x86_6... ) I get VERY different results:
On 2.6.34 it runs in about 10 secs. On 2.6.37 it takes at least 17 secs to run.
Could someone else test this on another hardware? I've run your script on my testing machine and I can see a difference as well (I've compared 2.6.34 with 2.6.36 kernel). I have dual quad-core AMD Opteron at 1 GHz with 4 GB of ram. The difference isn't that big as for you but still noticeable - about 20% on average I'd say.
Please run a couple of times with 2.6.34 and then with 2.6.37. Do others also see this big differences?
The system I am testing on is a dual Xeon X5650 @ 2.67 GHz (12 cores) with 24 GB memory. Please set $T to your number of cores. Perhaps you would also have to reduce the (1..9e6) if you have less than 2GB per core.
time perl -e 'use threads; $T=12 ; foreach (1..$T) { $thr[$i++] = threads->create(sub { printf "I am thread %s\n", threads->tid(); foreach (1..9e6) { push(@a, sqrt(1234)/sin(1234)) } ; printf "thread %s finished.\n", threads->tid(); }); } foreach (0..$T-1) { $thr[$_]->join(); }'
Does anyone have an idea why this could be happening? For such a simple test case I'd say that it's a difference in scheduling or some system call used implicitely by perl. What seems a bit strange though is that after all threads report they are finished, it still takes a noticeable time for parent thread to exit. From a quick strace I see a storm of madvise() calls when the thread exists so maybe this is the culprit? If yes, I'd try to oprofile the test on both kernels and compare results.
Umm, btw, I've just tried your test with just a single thread and it's
still slower on 2.6.36 than on 2.6.34 so indeed this seems like a
regression in some of the system calls.
Honza
--
Jan Kara
Jan Kara wrote:
time perl -e 'use threads; $T=12 ; foreach (1..$T) { $thr[$i++] = threads->create(sub { printf "I am thread %s\n", threads->tid(); foreach (1..9e6) { push(@a, sqrt(1234)/sin(1234)) } ; printf "thread %s finished.\n", threads->tid(); }); } foreach (0..$T-1) { $thr[$_]->join(); }'
what ugly perl :)
Does anyone have an idea why this could be happening? For such a simple test case I'd say that it's a difference in scheduling or some system call used implicitely by perl. What seems a bit strange though is that after all threads report they are finished, it still takes a noticeable time for parent thread to exit. From a quick strace I see a storm of madvise() calls when the thread exists so maybe this is the culprit? If yes, I'd try to oprofile the test on both kernels and compare results.
In my experience, a slow exit from a perl program, even if you watch it enter an explicit exit statement, is always due to perl's memory management system cleaning up. AIUI, the garbage collector has to garbage collect everything nicely because there's always the possibility that there's a DESTROY sub attached somewhere that needs to be executed. If you know that this isn't the case and you simply want to discard all the memory and exit then the best way I know is: use POSIX (); POSIX::_exit(0); In this case, the program made huge numbers of calls to the memory allocator by extending @a 12 x 9e6 times. So unwinding the memory could well take some time. If you want to benchmark threads instead of perl memory allocation then it probably would be better to preallocate the array. Incidentally, I noticed the other day that the vendor perl on an 11.3 box I have (with current kernel 2.6.34) states that it was compiled under 2.6.32. I haven't yet investigated why or whether it matters. Cheers, Dave -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
On 02/15/2011 11:29 AM, Dave Howorth wrote:
Jan Kara wrote:
time perl -e 'use threads; $T=12 ; foreach (1..$T) { $thr[$i++] = threads->create(sub { printf "I am thread %s\n", threads->tid(); foreach (1..9e6) { push(@a, sqrt(1234)/sin(1234)) } ; printf "thread %s finished.\n", threads->tid(); }); } foreach (0..$T-1) { $thr[$_]->join(); }'
what ugly perl :)
I started noticing this problem not with perl but with our CFD application compiled from C and FORTRAN. There I saw big performance differences between 2.6.34 and 2.6.37 on openSUSE 11.3. I saw the differences running on parallel and also in serial. Then I tried to find an example of this slowdown for others to test themselves on their machines, that's why I came out with this *simple* perl test, but I suppose any other language would do it. I could try a FORTRAN test and share my results. My last test some days ago was to update one of the openSUSE 11.3 nodes we have running to openSUSE 11.4 RC 1, and I saw no difference there on the perl test, so this looks promising again. I will have to run our own CFD application again on openSUSE 11.4 RC 1 and compare again. Richard -- Richard Ems mail: Richard.Ems@Cape-Horn-Eng.com Cape Horn Engineering S.L. C/ Dr. J.J. Dómine 1, 5º piso 46011 Valencia Tel : +34 96 3242923 / Fax 924 http://www.cape-horn-eng.com -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org For additional commands, e-mail: opensuse-kernel+help@opensuse.org
participants (4)
-
Dave Howorth
-
Jan Kara
-
Michal Marek
-
Richard Ems