cpu affinity with NUMA kernels - 'runon' like command? - openSUSE AMD64

cpu affinity with NUMA kernels - 'runon' like command?

daniel.kidger＠quadrics.com

16 Mar 2004 16 Mar '04

13:07

I have two Opteron clusters - one SLES8 based - the other Suse9. Both are running a 2.4.21 kernel with numa enabled. Are there any commands available to launch a process onto a given cpu (or set of cpus) within an SMP ? and likewise to check: which cpus a process in on? (cf /proc/12345/cpu) where it's memory is coming from? (cf say /proc/buddyinfo) Yours, Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com --------------------

Show replies by date

Dan Behman

16 Mar 16 Mar

14:50

New subject: [suse-amd64] cpu affinity with NUMA kernels - 'runon' like command?

Hi Daniel, Have a look at http://www.kernel.org/pub/linux/kernel/people/rml/cpu-affinity/ ... There's tons of information here that should answer most of your questions... To see what cpu a process is running, you can use top... By default, the "Last Used CPU" field may not be showing - simply hit "f" then "y" to enable that field and voila! (note that the commands may be slightly different depending on your version of top)... As for the where a process' memory is coming from, I don't know how to do that, but am certainly interested in hearing the answer... :) ---------- Dan Behman DB2 for Linux Platform Development IBM Software Development Lab, Office C4-432 Phone: (905) 413-4416 Email: dbehman@ca.ibm.com Please send DB2 for Linux questions to our DB2LinuxDev-CAN distribution list. ---------- daniel.kidger@qua drics.com To 03/16/2004 08:06 AM cc Subject [suse-amd64] cpu affinity with NUMA kernels - 'runon' like command? I have two Opteron clusters - one SLES8 based - the other Suse9. Both are running a 2.4.21 kernel with numa enabled. Are there any commands available to launch a process onto a given cpu (or set of cpus) within an SMP ? and likewise to check: which cpus a process in on? (cf /proc/12345/cpu) where it's memory is coming from? (cf say /proc/buddyinfo) Yours, Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- -- Check the List-Unsubscribe header to unsubscribe For additional commands, email: suse-amd64-help@suse.com

Andreas Jaeger

19:34

New subject: [suse-amd64] cpu affinity with NUMA kernels - 'runon' like command?

daniel.kidger@quadrics.com writes:

...

I have two Opteron clusters - one SLES8 based - the other Suse9. Both are running a 2.4.21 kernel with numa enabled.

Are there any commands available to launch a process onto a given cpu (or set of cpus) within an SMP ?

There's a numactl package that offers some options - but also some options that are not implemented in the kernel... Andreas -- Andreas Jaeger, aj@suse.de, http://www.suse.de/~aj SuSE Linux AG, Maxfeldstr. 5, 90409 Nürnberg, Germany GPG fingerprint = 93A3 365E CE47 B889 DF7F FED1 389A 563C C272 A126

Andi Kleen

21:08

New subject: [suse-amd64] cpu affinity with NUMA kernels - 'runon' like command?

On Tue, 16 Mar 2004 20:34:45 +0100 Andreas Jaeger wrote:

...

daniel.kidger@quadrics.com writes:

...
I have two Opteron clusters - one SLES8 based - the other Suse9. Both are running a 2.4.21 kernel with numa enabled.

Are there any commands available to launch a process onto a given cpu (or set of cpus) within an SMP ?

There's a numactl package that offers some options - but also some options that are not implemented in the kernel...

numactl won't work on the shipped kernels (it was for an experimental kernel patch was never production ready) In SL 9.0 you can use the commands in the scheduler-utilities rpm (not installed by default, but on the DVD) SLES8 currently doesn't have a tool, but there may be an update with a scheduler-utilities rpm. If you cannot wait for that you could compile the SL9 rpm yourself there. -Andi

Dan Kidger

17 Mar 17 Mar

10:27

New subject: [suse-amd64] cpu affinity with NUMA kernels - 'runon' like command?

Andi / Andreas, thanks for the information. We patch and compile our our own kernel - enabling the NUMA options amiongst other things. Under sles8 I installed numactl (from SP3) - it worked perfectly. :-) I could not only see the lastCPUused under top - but I coudl detect the performance change when I use a CPU close/far from the Golum PCI-X chips. Under suse9 - I installed the suse9 numactl - but it seem to be silently ignored - even though the the kernel binary is eactly the same as the sles8 cluster (2.4.21-171numa). I am investigating. Finally I tried 'scheduler-utilities' under suse9 - this only contains one file: $ rpm -qlp \ /home/packages/Linux_x86_64/suse9.0/scheduler-utilities-0.1-150.x86_64.rpm /usr/sbin/resched and this doesn't seem to have any NUMA options - just the usual nice-value et al. Am I missing something ? Daniel. -------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com -------------------- On Tuesday 16 March 2004 8:46 pm, Andi Kleen wrote:

...

On Tue, 16 Mar 2004 20:34:45 +0100

Andreas Jaeger wrote:

...
daniel.kidger@quadrics.com writes:

...
I have two Opteron clusters - one SLES8 based - the other Suse9. Both are running a 2.4.21 kernel with numa enabled.

Are there any commands available to launch a process onto a given cpu (or set of cpus) within an SMP ?

There's a numactl package that offers some options - but also some options that are not implemented in the kernel...

numactl won't work on the shipped kernels (it was for an experimental kernel patch was never production ready)

In SL 9.0 you can use the commands in the scheduler-utilities rpm (not installed by default, but on the DVD)

SLES8 currently doesn't have a tool, but there may be an update with a scheduler-utilities rpm. If you cannot wait for that you could compile the SL9 rpm yourself there.

-Andi

Dan Kidger

12:36

New subject: [suse-amd64] cpu affinity with NUMA kernels - 'runon' like command?

A quick update on this. It seem that the numactl shipped with suse9.0 doesn't work for --cpubind= - it simply ignores this option. However if I remove this and install the one shipped with SLES8-SP3 it does work. This is for the shipped '2.4.21-193-smp' kernel In all cases though /usr/bin/numademo reports: "your system does not support the numa API." I am looking through the source now. Also all the other options ( --membind, --localalloc, --interleave, --homenode) fail with: "numactl: this system does not support NUMA policy" presumably after check_numa() calls numa_available() which in turn calls numa_homenode() Daniel. On Wednesday 17 March 2004 10:27 am, Dan Kidger wrote:

...

Andi / Andreas, thanks for the information.

We patch and compile our our own kernel - enabling the NUMA options amiongst other things.

Under sles8 I installed numactl (from SP3) - it worked perfectly. :-) I could not only see the lastCPUused under top - but I coudl detect the performance change when I use a CPU close/far from the Golum PCI-X chips.

Under suse9 - I installed the suse9 numactl - but it seem to be silently ignored - even though the the kernel binary is eactly the same as the sles8 cluster (2.4.21-171numa). I am investigating.

Finally I tried 'scheduler-utilities' under suse9 - this only contains one file: $ rpm -qlp \ /home/packages/Linux_x86_64/suse9.0/scheduler-utilities-0.1-150.x86_64.rpm /usr/sbin/resched

and this doesn't seem to have any NUMA options - just the usual nice-value et al. Am I missing something ?

Daniel.

-------------------------------------------------------------- Dr. Dan Kidger, Quadrics Ltd. daniel.kidger@quadrics.com One Bridewell St., Bristol, BS1 2AA, UK 0117 915 5505 ----------------------- www.quadrics.com --------------------

On Tuesday 16 March 2004 8:46 pm, Andi Kleen wrote:

...
On Tue, 16 Mar 2004 20:34:45 +0100

Andreas Jaeger wrote:

...
daniel.kidger@quadrics.com writes:

...
I have two Opteron clusters - one SLES8 based - the other Suse9. Both are running a 2.4.21 kernel with numa enabled.

Are there any commands available to launch a process onto a given cpu (or set of cpus) within an SMP ?

There's a numactl package that offers some options - but also some options that are not implemented in the kernel...

numactl won't work on the shipped kernels (it was for an experimental kernel patch was never production ready)

In SL 9.0 you can use the commands in the scheduler-utilities rpm (not installed by default, but on the DVD)

SLES8 currently doesn't have a tool, but there may be an update with a scheduler-utilities rpm. If you cannot wait for that you could compile the SL9 rpm yourself there.

-Andi

Andi Kleen

12:58

New subject: [suse-amd64] cpu affinity with NUMA kernels - 'runon' like command?

On Wed, 17 Mar 2004 12:36:52 +0000 Dan Kidger wrote:

...

A quick update on this.

It seem that the numactl shipped with suse9.0 doesn't work for --cpubind= - it simply ignores this option. However if I remove this and install the one shipped with SLES8-SP3 it does work. This is for the shipped '2.4.21-193-smp' kernel

In all cases though /usr/bin/numademo reports: "your system does not support the numa API." I am looking through the source now.

Also all the other options ( --membind, --localalloc, --interleave, --homenode) fail with: "numactl: this system does not support NUMA policy" presumably after check_numa() calls numa_available() which in turn calls numa_homenode()

Yes, that is what I wrote in my first mail: the 2.4 NUMA API was an experimental kernel patch that was never included in any production kernel. -Andi

7349

Age (days ago)

7350

Last active (days ago)

List overview

Download

6 comments

5 participants

participants (5)

Andi Kleen
Andreas Jaeger
Dan Behman
Dan Kidger
daniel.kidger＠quadrics.com