Andi Kleen wrote:

On Tue, 6 Apr 2004 12:41:57 -0500 "Miller, Daniel J." wrote:

...

Support from the manufacturer for the Arima line of motherboards can be found here:

http://www.rioworks.com/Download/HDAMA.htm

http://www.rioworks.com/Service-Download.htm

One word of caution with Arima: While their motherboards seem to be very stable (not much problems reported) some of them like the HDAMB are not true NUMA: they have the memory connected only to a single CPU. If you want a well scaling Opteron system I would avoid such designs. Other boards from Arima may be ok in this regard.

This can be normally easily checked on board pictures - the traces from the DIMM slots to the CPUs are visible. They should lead to both CPU sockets.

-Andi

Hmmmm .... I thought the CPUs talked to eachother (at least the 200 & 800 series) through high speed busses & could shuttle data between eachother as fast as direct memory access (except for some small latency to start the proceedings), no ? I had been leaning toward some of the balanced MP boards (TYAN S2882, Arima HDAMA) on that count.

Darrell Shively wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Hi William:

On Wednesday 07 April 2004 05:54, William A. Mahaffey III wrote:

...

[...]

Hmmmm .... I thought the CPUs talked to eachother (at least the 200 & 800 series) through high speed busses & could shuttle data between eachother as fast as direct memory access (except for some small latency to start the proceedings), no ?

Turns out no. The Hyptertransport connection between the processors *is* very fast but not as fast as each processors' 128+ bit wide memory bus. This is why the processor affinity feature of NUMA kernels is important; it tries to keep a process on the processor whose RAM contains its data.

...

I had been leaning toward some of the balanced MP boards (TYAN S2882, Arima HDAMA) on that count.

It depends on your needs. A second processor can be useful even if it's memory access is via a hypertransport link. It depends on what sort of jobs you are running - if stuff fits mostly in the 2nd processors' cache then there is happiness.

Regards, - Darrell - -- sused@mucus.com "Perfect! ....what am I doing?" -- Washu -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQFAdB5Veo6c0kw6mZ0RAkurAKCiJKUfv8aSxeUjfS5hF9D4WfNf3wCgt6B1 KiTOydpVERJIfX8PLiCywCU= =nCUl -----END PGP SIGNATURE-----

Actually most of the stuff I run would be large jobs requiring a significant fraction of available RAM, too big to fit into cache. I thought the actual data speed of the hyper-transport bus (6.4 GB/s) was similar to the memory bus (6.4 GB/s using PC3200 RAM, 5.3 GB/s using PC2700 RAM), although by different means (64 bit dual-channel DDR bus at either 166 MHz or 200 MHz for the RAM, 16 bit DDR at 1600 MHz for the hyper-transport bus). I would also be interested in knowing how SMP is working .... just to help keep the already busy thread going :-).

...

Actually most of the stuff I run would be large jobs requiring a significant fraction of available RAM, too big to fit into cache. I thought the actual data speed of the hyper-transport bus (6.4 GB/s) was similar to the memory bus (6.4 GB/s using PC3200 RAM, 5.3 GB/s using PC2700 RAM), although by different means (64 bit dual-channel DDR bus at either 166 MHz or 200 MHz for the RAM, 16 bit DDR at 1600 MHz for the hyper-transport bus). I would also be interested in knowing how SMP is working .... just to help keep the already busy thread going :-).

I can relate my experience with a quad opteron. Using the STREAM benchmark, which calculates bandwidth by measuring the time required to perform simple operations on large arrays (where the bottleneck is the streaming speed), I get about 2 GB/s on a single CPU, (PC2700 ->333MHz*16 5.3 GB/s, 1.4 GHz CPU speed). Under a NUMA kernel (e.g. 2.4.21-207, SLES SP3), the collective bandwidth gets as high as 7.5 GB/s, running on 4 CPUS, whereas running a non NUMA kernel (2.4.24) it rarely exceeds 3 GB/s. I have observed similar trends with different codes, all constrained by the memory bandwidth. The bottom line is

My STREAMS figures are somewhat higher: one cpu: 3497.4544 MByte/s for STREAMS Triad four cpus: 12970.4028 MByte/s for STREAMS Triad These are 2.2GHz CPus in a 4-way AMD/Celestica 'Quartet' with 333MHz 512MB DIMMS. kernel is 2.4.21-171.4.5qsnet_numa, otherwise it is a standard SLES8 disto (on a 2-way node with 1.8GHz CPUs + 333MHz DIMMS I get 3528.9499 MBytes/s on 1 cpu and 6968.9103 MBytes/s for Triad when using both CPUs) As you may know, the Opteron Memory bus speed is a function of both the CPU speed and the DIMMS speed. What CPUs did you have ? Also make sure the BIOs is interleaving pairs of DIMMS to get 128 bit access.

3) Using standard MPICH benchmarks, CPU X to CPU Y bandwidth is of the order of 750 MB/s, for large packets. Why so low, I do not know. It could be related to the compiler used to compile the MPICH library (pgcc with -fastsse -Mvect=prefetch, v 5.1, 64 bit).

not really a compiler issue here . MPI needs to copy data from the userspace on one CPU to the userspace on the other CPU. Standard security rules stop one process from directly writing into the memory space of the other. Hence MPICH creates a SystemV shared memory segment that one writes into and the other reads from. So if the memory bus (on each CPU) can be kept saturated at 2GB/s you can never get more than half this with intranode MPI. Small messages can take some advantage of the cache though - I get a best intra-node MPI bandwidth of 1423.49 MB/s - this is with a message size of 256kBytes. This falls to 893 MB/s for say MPI_Sends of 4MBytes at a time. (compare this with around 875 MBytes/s when using MPI between nodes and a Quadrics interconnect) -- Yours, Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com --------------------

ascotti@email.unc.edu wrote:

--On Wednesday, April 07, 2004 7:00 PM -0500 "William A. Mahaffey III" wrote:

...

Darrell Shively wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

Hi William:

On Wednesday 07 April 2004 05:54, William A. Mahaffey III wrote:

...

[...]

Hmmmm .... I thought the CPUs talked to eachother (at least the 200 & 800 series) through high speed busses & could shuttle data between eachother as fast as direct memory access (except for some small latency to start the proceedings), no ?

Turns out no. The Hyptertransport connection between the processors *is* very fast but not as fast as each processors' 128+ bit wide memory bus. This is why the processor affinity feature of NUMA kernels is important; it tries to keep a process on the processor whose RAM contains its data.

...

I had been leaning toward some of the balanced MP boards (TYAN S2882, Arima HDAMA) on that count.

It depends on your needs. A second processor can be useful even if it's memory access is via a hypertransport link. It depends on what sort of jobs you are running - if stuff fits mostly in the 2nd processors' cache then there is happiness.

Regards, - Darrell - -- sused@mucus.com "Perfect! ....what am I doing?" -- Washu -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQFAdB5Veo6c0kw6mZ0RAkurAKCiJKUfv8aSxeUjfS5hF9D4WfNf3wCgt6B1 KiTOydpVERJIfX8PLiCywCU= =nCUl -----END PGP SIGNATURE-----

Actually most of the stuff I run would be large jobs requiring a significant fraction of available RAM, too big to fit into cache. I thought the actual data speed of the hyper-transport bus (6.4 GB/s) was similar to the memory bus (6.4 GB/s using PC3200 RAM, 5.3 GB/s using PC2700 RAM), although by different means (64 bit dual-channel DDR bus at either 166 MHz or 200 MHz for the RAM, 16 bit DDR at 1600 MHz for the hyper-transport bus). I would also be interested in knowing how SMP is working .... just to help keep the already busy thread going :-).

I can relate my experience with a quad opteron. Using the STREAM benchmark, which calculates bandwidth by measuring the time required to perform simple operations on large arrays (where the bottleneck is the streaming speed), I get about 2 GB/s on a single CPU, (PC2700 ->333MHz*16 5.3 GB/s, 1.4 GHz CPU speed). Under a NUMA kernel (e.g. 2.4.21-207, SLES SP3), the collective bandwidth gets as high as 7.5 GB/s, running on 4 CPUS, whereas running a non NUMA kernel (2.4.24) it rarely exceeds 3 GB/s. I have observed similar trends with different codes, all constrained by the memory bandwidth. The bottom line is 1) The theoretical Bandwidth is an ephemeral goal. Compilers just are not that tuned. I can get closer if I start writing my own assembler code, but that takes time.. 2) NUMA makes a big difference. 3) Using standard MPICH benchmarks, CPU X to CPU Y bandwidth is of the order of 750 MB/s, for large packets. Why so low, I do not know. It could be related to the compiler used to compile the MPICH library (pgcc with -fastsse -Mvect=prefetch, v 5.1, 64 bit). I have read elsewhere that pgcc does not produce good code (I think was for the ATLAS libraries), but that might have to do with the way ATLAS routines are written. Actually, it would be interesting to hear from other people on the best combination of compiler/switches. On a related note, I have not noticed much difference between compiling in 32 or 64 bit.

Having said that, my BIOS (quartet motherboard, manifactured by Celestica) allows to distribute memory addresses in a roundrobin fashion among CPUs. When this option is enabled, a large array is spread uniformly among CPUs. While this is bad running NUMA, it could improve the performance with traditional SMP kernels, but I have not tried that.

Overall, I am quite happy with the machine. Based on runs of a CFD code (parallel), each Opteron CPU is equivalent to 2.5 Athlon cpus at the same frequency. It took almost a month to put all the pieces of software and firmware to work together, but it was worth the hassle.

Alberto Scotti

I was revisiting this thread in more detail & came up w/ another question based on this response. What is the difference between NUMA & traditional SMP as referred to in this response ? Thanks in advance.

On Thu, 8 Apr 2004 12:05:10 +0100 Dan Kidger wrote:

This is since in most cases there is an application running on all CPUs (certainly for us scientific users). Each applciation can typically saturate the local memory bus. If applications from other CPus are also memory bandwidth hungry then all applications will slow each other down. This is of course the bane of all those dual-Xeon HPC servers where one copy of a finite element pogram takes say X minutes but with two running simultaneosuly (one per cpu) each now takes 1.4 X minutes to run. Under NUMA kernel on Opteron - they do not slow each other down at all.

The local policy used by the NUMA kernel is not always optimal though - that is why it is sometimes faster to configure node cache line interleaving in the BIOS. This happens for example when the working set of all CPUs exceeds a single node and the workload prefers bandwidth over latency. Or when you only have a program running on a single CPU, but it needs all the bandwidth it can get; then interleaving is the best policy, because you will combine the bandwidth of all available memory controllers. For most workloads local affinity seems to be pretty good though, so it's a good default. The 9.1/SLES9 kernel will have a new NUMA API that will allow to configure NUMA policies [local affinity, binding to a specific CPU, page interleaving] finegrained per process and per memory mapping without rebooting. -Andi