Sheo Shanker Prasad wrote:

I will greatly appreciate help in understanding the puzzling wild variation in the execution time of small test/benchmark program, as explained below.

(1) The nature of the benchmark/test program:

A floating point arithmetic intensive atmospheric chemistry modeling task. It used to take, in a consistent manner, around 2m 37s on my machine when it was running under SuSe 9.1 Pro. When I run the benchmark program, I shut-off all other programs. Thus, during its run, it is the only active program. THe rest are belonging to root are sleeping. There is no other user.

(2) The machine:

Dual Opteron 250 with 4 GB of PC3200 (DDR) memory in 4 dimms that fill two dimms on cpu0 and two dimms on cpu1 installed in Tyan Thunder K8 system board that has American MegaTrend AMIBIOS V2-04. It now runs under SuSE 9.3 Pro and the Linux version 2.6.11.4-21.7-smp (geeko@buildhost) (gcc version 3.3.5 20050117 (prerelease) (SUSE Linux)) #1 SMP Thu Jun 2 14:23:14 UTC 2005

(3) Background of the problem:

After only 4 months of the delivery of the machine by the vendor, the mother board failed. The machine was sent to the vendor. He then replaced the mother board with a new one and installed SuSE 9.3 Pro to bring the machine up-to-date with the Linux version 2.6.11.4-21.7-smp (geeko@buildhost) (gcc version 3.3.5 20050117 (prerelease) (SUSE Linux)) #1 SMP Thu Jun 2 14:23:14 UTC 2005

(4) The problem:

After the repair and upgrade to 9.3 Pro, the execution times of the same test/benchmark program varies widely between run to run. It is never consistent. It varies from 2m 37s to more than 6m. A pattern is seen in the variation. Usually, the longest execution time is when the machine is started after remaining shut-off for a few hours. The shortest execution time is obtained after the machine has been constantly on for abot 7 to 10 hours.

(5) My puzzle:

While a computer is well known to be reproducible, the wild variation by a factor of almost 3 in the execution is very disturbing.

What could be the cause? Does it mean that some vital component involved in execution of the numerical modeling (RAM memory or the memory on the CPUs) are loose or erratic and may be about to go bad? I am just puzzled. Any clue from you all will be really very very much appreciated.

well, I'm not an expert on this kind of problem, but I'm still interested in problem solving... all you said brings me to think to some kind of memory leakage (stabilization of exec time after a long period of uptime)...supposing your algorithm is consumming a lot of memory... if this is true (memory leakage), the next step is to determine if the guilty is the kernel (9.1 was 2.6.5, 9.3 is 2.6.11 -please, kernel experts, are there a lot of changes in memory management between these two versions, and especially involving smp systems ?), or maybe the compiler (did you compile yourself you algorithm ? did you recompile it after the installation of suse93 ? written in C, C++, Fortran, other ?) maybe an other possibility is hardware/bios related problem (or incorrect bios settings), but I don't have any idea on how to diagnose a faulty cpu...please, tyan hardware experts, it's up to you... Ah, I've just reread your post. You said the longest time is after the machine is started after a long shut-off... What occurs if, after a long period of uptime you reboot the machine and immediately run your algorithm ? - If you get the same good results (2'37") then you certainly have a hardware problem (thermal stabilization problem on some component ?) - If you get bad results (longest time) then it's certainly a software related problem (bios/kernel/compiler) sorry not to be able to help you more...that was only to start the discussion...now it's time for true experts. cheers, fabrice

On Thu, 30 Jun 2005 11:02:34 -0700 Sheo Shanker Prasad wrote:

(5) My puzzle:

While a computer is well known to be reproducible, the wild variation by a factor of almost 3 in the execution is very disturbing.

It could be cache effects. Linux does not do enforced L2 cache coloring while allocating user memory so in memory intensive programs some variation can happen depending on how well the cache is used. However an Opteron has a 16 way associative L2 cache so such things normally don't happen there. One difference between 9.1 and 9.3 is that 9.3 allocates the memory from the start to the end after boot up, while 9.1 tended to allocate it from up to down (highest memory first). It is possible that your BIOS has a broken MTRR or somesuch and that some low page is not cached properly and slowing things down. If it was a high page it could be avoided by booting with mem=...M with a number smaller than your memory. However for low pages that's difficult. -Andi