What you're describing sounds like a kernel crash rather than just a program hang. Mind you it *could* just be that the kernel was "thrashing" as you would probably have been using a lot of the swap file by the time you got one of the processes up to 2350MB, bearing in mind that both the kernel and all other processes are using memory too. If I'm doing my assessment right then you have 2GB physical + 1GB swap = 3GB theoretical, you were using 2.35GB in one userspace thread (depending how you define a GB!) plus the kernel and other process might well be using 250MB or so, meaning only 400MB left in both swapfile and RAM together. Couldn't that mean quite a lot of thrashing? One test I use is if it still responds to IP pings. If not then the kernel (scheduling, interrupts, softirq, bh) must really be down - possibly due to a disastrous access violation, etc. in kernel space. If this is the case and if it always happens at the same point then there could be a kernel bug of some sort I suppose. If it was just thrashing (but scheduling, softirq, bh handlers, etc. are still running fine) then IMHO really the only solution is a smaller program, a bigger swapfile or more physical RAM. The "64GB"?... If you compile the kernel, you'll find in the kernel options that at one point you specify the size of physical RAM (or at least some limits on it). In particular, the IA32 processors cannot address over 4GB of address space "normally", they need to have a special paging mode switched on - this is supposedly supported in the Pentium Pro and all later processors so a P4 should be fine. To switch this on you set the physical memory setting to "64GB" instead of the default of "4GB". This simply means that the kernel can handle up to 64GB of physical RAM (but can also handle smaller amounts too). Why isn't this always switched on? Kernels with this paging active will not run on old processors (pre Pentium Pro). So that users don't have to keep compiling kernels (most don't like to), SuSE have built some for you. Often (like on my box) the rpm package with the "4GB" version of the kernel (vmlinuz and modules) will be installed. As an option you can choose the 64GB package - you need to do this if you have over 4GB of physical RAM otherwise the extra RAM is unuseable. Hardware wise - if you run another OS and get the same problems that's a key indicator. Temperature sensing can be done by the lm_sensors package. Be warned, it takes a little time to get it working well (especially if you want the graphs visible on a web page or a GUI or suchlike) but 32 deg C must really be OK I think - the north bridge could play up but if it always happens at the same point then this is pretty unlikely. As per usual the rule I've heard is that if a crash always occurs in the same place then it's software or OS, if it occurs seemingly randomly (except "always when someone's cooking in the room" or "never when the window is open") then it's hardware. Sorry my answers are so basic. If you can't get any satisfaction here then you could try a kernel mailing list or my recommendation is a suitable IRC channel. I like IRC because you get to chat to people, instant responses, etc. ----- Original Message ----- From: "William A. Mahaffey III" Cc: "SuSE 32 bit Apps. Mailing List" Sent: Tuesday, February 24, 2004 4:42 PM Subject: Re: [suse-programming-e] 4 GB address space ?

Carl Peto wrote:

...

AFAIK there is no problem with having a swap partition smaller than your physical RAM. I have been running my system in that configuration for over a year.

When you say "hung" what do you mean? What status flag did "ps" show for it? S? R? D? These would represent a sleeping and a running process and a process in uninterruptible sleep, waiting for IO from the kernel.

Bash doesn't respond to normal kill signals, by default and you need to do a kill -9 to kill it off, as I've found, could it be that the parent shell and login shell were just "asleep" waiting for the hung process to finish? Running the process in background would probably test that theory too.

Finally, (just to state the obvious) is your RAM OK? Could it have a fault? Any chance that the system overheated when doing lots of floating point operations, etc.?

----- Original Message ----- From: "William A. Mahaffey III" To: "SuSE 32 bit Apps. Mailing List" Sent: Tuesday, February 24, 2004 2:55 PM Subject: [suse-programming-e] 4 GB address space ?

...

I am new to the list. I have various codes currently native on SGI hardware that I am interested in porting to Linux, specifically SuSE. I was attracted to SuSE by the reputed ability to address a larger part of the 4 GB 32-bit address space (larger than the usual 2-GB). To that end, I have tried to port a small benchmarking program to SuSE 8.2, on an Intel P4 box, 2.4 GHz, Biostar P4TSV mbd, w/ 2 GB of RAM (for now) & 1 GB of swap space. This box is a compute node on my network & is accessed by ssh; it has no monitor, kbd, or mouse, & runs in runlevel 3 by default. The benchmarking program executes various floating point array operations on successively longer arrays (allocated w/ malloc) until the malloc calls fail or until an upper size limit is reached. I was running the program last night & observed that I could successfully allocate sloghtly over 2 GB of memory (~2350 MB), but when I began operations, the process (as well as the parent shell & login shell, apparently) hung. I killed the login processes from the machine I was logged in from & was able to log back in w/o rebooting. Several questions:

1. I tried to set up a swap partition of > 1 GB & the installer wouldn't let me, how do I get around that ? 2. Is there a problem w/ having more RAM than swap ? (I have 2 GB RAM, 1 GB swap at the moment) 3. I had problems w/ the installer w/ 2 GB of RAM installed, when I removed 1 GB to test why I was having problems, everything went well, total install in about 20 minutes on newly formatted system disc. Is there a known problem w/ the installer w/ more than 1 GB of RAM installed ?

-- To unsubscribe, email: suse-programming-e-unsubscribe@suse.com For additional commands, email: suse-programming-e-help@suse.com Archives can be found at: http://lists.suse.com/archive/suse-programming-e

-- To unsubscribe, email: suse-programming-e-unsubscribe@suse.com For additional commands, email: suse-programming-e-help@suse.comLinux Archives can be found at: http://lists.suse.com/archive/suse-programming-e

Thanks for your response. By hung, I mean no response to kbd (after several (~20) minutes). I was running tcsh as my login shell, not bash, & was running the process in BG under tcsh also. I also went back & checked my messages file & I did indeed have to reboot after the hang. With no kbd response, no ps, etc. While running successfully (pre-hang), top showed amount of RAM & swap used, they looked OK. Once hung, I couldn't log in over my network to do any checkouts & had to reboot it. I think your assessment may be correct (shells asleep awaiting child to die), I guess my question is why did the child die, I didn't think there was any problem until you got (much) closer to 4 GB of allocated RAM. Also, here is output from uname -a:

Linux cmi2400 2.4.20-64GB-SMP #1 SMP Mon Mar 17 17:56:03 UTC 2003 i686 unknown unknown GNU/Linux

This is what the installer configured, what's up w/ the 64GB in the kernel name ?

I think my RAM is OK, that's why I removed part of it during install, to eliminate that as a problem. When the install was done, I went through a cycle of power-down, swap RAM, power-up, run benchmark (which would access all RAM eventually), & checked out all RAM in all slots, seems AOK there. Good question about heat, is there any way to check that under Linux ? (I don't think that was a problem, I rebooted during RAM swaps & after hang & BIOS said 32-34C for CPU)

-- To unsubscribe, email: suse-programming-e-unsubscribe@suse.com For additional commands, email: suse-programming-e-help@suse.com Archives can be found at: http://lists.suse.com/archive/suse-programming-e

Carl Peto wrote:

What you're describing sounds like a kernel crash rather than just a program hang.

Mind you it *could* just be that the kernel was "thrashing" as you would probably have been using a lot of the swap file by the time you got one of the processes up to 2350MB, bearing in mind that both the kernel and all other processes are using memory too.

If I'm doing my assessment right then you have 2GB physical + 1GB swap = 3GB theoretical, you were using 2.35GB in one userspace thread (depending how you define a GB!) plus the kernel and other process might well be using 250MB or so, meaning only 400MB left in both swapfile and RAM together. Couldn't that mean quite a lot of thrashing?

One test I use is if it still responds to IP pings. If not then the kernel (scheduling, interrupts, softirq, bh) must really be down - possibly due to a disastrous access violation, etc. in kernel space. If this is the case and if it always happens at the same point then there could be a kernel bug of some sort I suppose.

If it was just thrashing (but scheduling, softirq, bh handlers, etc. are still running fine) then IMHO really the only solution is a smaller program, a bigger swapfile or more physical RAM.

The "64GB"?...

If you compile the kernel, you'll find in the kernel options that at one point you specify the size of physical RAM (or at least some limits on it).

In particular, the IA32 processors cannot address over 4GB of address space "normally", they need to have a special paging mode switched on - this is supposedly supported in the Pentium Pro and all later processors so a P4 should be fine. To switch this on you set the physical memory setting to "64GB" instead of the default of "4GB". This simply means that the kernel can handle up to 64GB of physical RAM (but can also handle smaller amounts too). Why isn't this always switched on? Kernels with this paging active will not run on old processors (pre Pentium Pro).

So that users don't have to keep compiling kernels (most don't like to), SuSE have built some for you. Often (like on my box) the rpm package with the "4GB" version of the kernel (vmlinuz and modules) will be installed. As an option you can choose the 64GB package - you need to do this if you have over 4GB of physical RAM otherwise the extra RAM is unuseable.

Hardware wise - if you run another OS and get the same problems that's a key indicator. Temperature sensing can be done by the lm_sensors package. Be warned, it takes a little time to get it working well (especially if you want the graphs visible on a web page or a GUI or suchlike) but 32 deg C must really be OK I think - the north bridge could play up but if it always happens at the same point then this is pretty unlikely. As per usual the rule I've heard is that if a crash always occurs in the same place then it's software or OS, if it occurs seemingly randomly (except "always when someone's cooking in the room" or "never when the window is open") then it's hardware.

Sorry my answers are so basic. If you can't get any satisfaction here then you could try a kernel mailing list or my recommendation is a suitable IRC channel. I like IRC because you get to chat to people, instant responses, etc.

----- Original Message ----- From: "William A. Mahaffey III" Cc: "SuSE 32 bit Apps. Mailing List" Sent: Tuesday, February 24, 2004 4:42 PM Subject: Re: [suse-programming-e] 4 GB address space ?

...

Carl Peto wrote:

...

AFAIK there is no problem with having a swap partition smaller than your physical RAM. I have been running my system in that configuration for over a year.

When you say "hung" what do you mean? What status flag did "ps" show for it? S? R? D? These would represent a sleeping and a running process and a process in uninterruptible sleep, waiting for IO from the kernel.

Bash doesn't respond to normal kill signals, by default and you need to do a kill -9 to kill it off, as I've found, could it be that the parent shell and login shell were just "asleep" waiting for the hung process to finish? Running the process in background would probably test that theory too.

Finally, (just to state the obvious) is your RAM OK? Could it have a fault? Any chance that the system overheated when doing lots of floating point operations, etc.?

----- Original Message ----- From: "William A. Mahaffey III" To: "SuSE 32 bit Apps. Mailing List" Sent: Tuesday, February 24, 2004 2:55 PM Subject: [suse-programming-e] 4 GB address space ?

...

I am new to the list. I have various codes currently native on SGI hardware that I am interested in porting to Linux, specifically SuSE. I was attracted to SuSE by the reputed ability to address a larger part of the 4 GB 32-bit address space (larger than the usual 2-GB). To that end, I have tried to port a small benchmarking program to SuSE 8.2, on an Intel P4 box, 2.4 GHz, Biostar P4TSV mbd, w/ 2 GB of RAM (for now) & 1 GB of swap space. This box is a compute node on my network & is accessed by ssh; it has no monitor, kbd, or mouse, & runs in runlevel 3 by default. The benchmarking program executes various floating point array operations on successively longer arrays (allocated w/ malloc) until the malloc calls fail or until an upper size limit is reached. I was running the program last night & observed that I could successfully allocate sloghtly over 2 GB of memory (~2350 MB), but when I began operations, the process (as well as the parent shell & login shell, apparently) hung. I killed the login processes from the machine I was logged in from & was able to log back in w/o rebooting. Several questions:

1. I tried to set up a swap partition of > 1 GB & the installer wouldn't let me, how do I get around that ? 2. Is there a problem w/ having more RAM than swap ? (I have 2 GB RAM, 1 GB swap at the moment) 3. I had problems w/ the installer w/ 2 GB of RAM installed, when I removed 1 GB to test why I was having problems, everything went well, total install in about 20 minutes on newly formatted system disc. Is there a known problem w/ the installer w/ more than 1 GB of RAM installed ?

-- To unsubscribe, email: suse-programming-e-unsubscribe@suse.com For additional commands, email: suse-programming-e-help@suse.com Archives can be found at: http://lists.suse.com/archive/suse-programming-e

-- To unsubscribe, email: suse-programming-e-unsubscribe@suse.com For additional commands, email: suse-programming-e-help@suse.comLinux Archives can be found at: http://lists.suse.com/archive/suse-programming-e

Thanks for your response. By hung, I mean no response to kbd (after several (~20) minutes). I was running tcsh as my login shell, not bash, & was running the process in BG under tcsh also. I also went back & checked my messages file & I did indeed have to reboot after the hang. With no kbd response, no ps, etc. While running successfully (pre-hang), top showed amount of RAM & swap used, they looked OK. Once hung, I couldn't log in over my network to do any checkouts & had to reboot it. I think your assessment may be correct (shells asleep awaiting child to die), I guess my question is why did the child die, I didn't think there was any problem until you got (much) closer to 4 GB of allocated RAM. Also, here is output from uname -a:

Linux cmi2400 2.4.20-64GB-SMP #1 SMP Mon Mar 17 17:56:03 UTC 2003 i686 unknown unknown GNU/Linux

This is what the installer configured, what's up w/ the 64GB in the kernel name ?

I think my RAM is OK, that's why I removed part of it during install, to eliminate that as a problem. When the install was done, I went through a cycle of power-down, swap RAM, power-up, run benchmark (which would access all RAM eventually), & checked out all RAM in all slots, seems AOK there. Good question about heat, is there any way to check that under Linux ? (I don't think that was a problem, I rebooted during RAM swaps & after hang & BIOS said 32-34C for CPU)

-- To unsubscribe, email: suse-programming-e-unsubscribe@suse.com For additional commands, email: suse-programming-e-help@suse.com Archives can be found at: http://lists.suse.com/archive/suse-programming-e

-- To unsubscribe, email: suse-programming-e-unsubscribe@suse.com For additional commands, email: suse-programming-e-help@suse.com Archives can be found at: http://lists.suse.com/archive/suse-programming-e

Thanks again. I think your assessment is right on. Yes, I could still ping it, but I couldn't get logged on because everything was paged out except the benckmark program, so it wasn't really "kernel hung", just so sluggish I couldn't tell the difference. OK, so far, so good. How do I get more than 1 GB of swap space (or am I stuck there) ? Thanks again.

Carl Peto wrote:

I've no idea I'm afraid!

Have you tried just creating the swap partition with fdisk and activating it in the normal way (like we had to in the days before fancy installers)?

Alternatively use the Expert Partitioning tool. If you don't know exactly what you're doing with these powerful tools then do test them on machines that are currently test/play machines and *not* a production machine.

If it's going to be a powerful server then it might be worth talking to the people who control your budget about stretching to some more RAM (we recently bought 1GB RAM for £100). In my experience if your server does a lot it needs a lot of RAM is one of the unavoidable facts of life. The other road leads to bad performance and people grumbling like, "...well I think things were better before 'they' switched to this 'Linux' thing...", etc.

--SNIP--

...

Thanks again. I think your assessment is right on. Yes, I could still ping

it,

...

but I couldn't get logged on because everything was paged out except the benckmark program, so it wasn't really "kernel hung", just so sluggish I couldn't tell the difference. OK, so far, so good. How do I get more than 1 GB of swap space (or am I stuck there) ? Thanks again.

-- To unsubscribe, email: suse-programming-e-unsubscribe@suse.com For additional commands, email: suse-programming-e-help@suse.com Archives can be found at: http://lists.suse.com/archive/suse-programming-e

I'm looking into more RAM as we speak. Thanks.

Steven Augart wrote:

William A. Mahaffey III wrote:

...

How do I get more than 1 GB of swap space (or am I stuck there) ? Thanks again.

First, an easy hack is to add more swap partitions or swap files (if it's inconvenient to add a partition, you can create a physical file in the filesystem to swap to. You probably know this already.) If you are facing a limit of 1 GB, then add them 1 GB at a time. You can have multiple swap partitions going at the same time.

Second, are you facing the 1 GB limit in the "mkswap" command or in some higher-level tool? If higher level, then you might get it to work by using "mkswap".

--Steve Augart

-- Steven Augart

Jikes RVM, open source Research Virtual Machine: http://oss.software.ibm.com/jikesrvm

Office: +1 914/784-6743 T.J. Watson Research Center, IBM

I was working through the installer & it wouldn't let me make a swap partition larger than 1 GB. I prefer partitions to files for swapping . I was really kinda asking if this was (still) a Linux limitation, it was in the past (actually the limit used to be 2 GB per swap device ....).