Hmmmm .... I thought the CPUs talked to eachother (at least the 200 & 800 series) through high speed busses & could shuttle data between eachother as fast as direct memory access (except for some small latency to start the proceedings), no ?
Turns out no. The Hyptertransport connection between the processors *is* very fast but not as fast as each processors' 128+ bit wide memory bus. This is why the processor affinity feature of NUMA kernels is important; it tries to keep a process on the processor whose RAM contains its data.
Even if the Hypertransort link is infinitely fast - you would still want memory local to each CPU. This is since in most cases there is an application running on all CPUs (certainly for us scientific users). Each applciation can typically saturate the local memory bus. If applications from other CPus are also memory bandwidth hungry then all applications will slow each other down. This is of course the bane of all those dual-Xeon HPC servers where one copy of a finite element pogram takes say X minutes but with two running simultaneosuly (one per cpu) each now takes 1.4 X minutes to run. Under NUMA kernel on Opteron - they do not slow each other down at all. -- Yours, Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com --------------------