Odon Farkas wrote:
Hi Richard,
Try using the -mcmodel=medium option for pgf90. I assume your program works for smaller arrays.
Best wishes,
Ödön
On Wed, 2004-02-25 at 12:23, JOSE LUIS RICARDO-CHAVEZ wrote:
richard.brunner@amd.com wrote:
Physical memory available != virtual memory available for allocation. In an 8GB DRAM system, the BIOS gives the OS approximately 7.5GB (because of various I/O issues).
You should be able to get the kernel to allocate a large amount but once you allocate larger than phys memory, you will page awfully.
You don't say whether the app requesting the memory is 32-bit or 64-bit. I assume 64-bit.
Which PGI compiler (F77? F90?) and which version (5.0? 5.1? 5.1-3?) are you using? How are you allocating the array?
] -Rich ... ] AMD Fellow ] richard.brunner at amd com
-----Original Message----- From: JOSE LUIS RICARDO-CHAVEZ [mailto:ricardo@irsamc.ups-tlse.fr] Sent: Tuesday, February 24, 2004 4:33 AM To: Andreas Jaeger Cc: suse-amd64@suse.com Subject: Re: [suse-amd64] Kernel support for large arrays...
Andreas Jaeger wrote:
JOSE LUIS RICARDO-CHAVEZ <ricardo@irsamc.ups-tlse.fr> writes:
recently we have installed in our lab an Opteron system (Tyan 2882 motherboard, two Opteron 244 processor, 8 GB memory) running SuSE AMD64 9.0 Using the PGI compiler suite we did some memory
allocation
tests. We found a modest 4 GB memory limit per allocated array. According to PGI engineers this is not a compiler fault, the limit is just set in kernel's mmap.c routine.
Would it be safe to
change this value and recompile the kernel or it would require changing some other routines?
There's no such limit in the kernel. You should be allocate such an array without problems.
Andreas
Hi Andreas,
You are right, I looked into the file /usr/include/asm-x86-64/processor.h and found the following:
/* * User space process size: 512GB - 1GB (default). */ #define TASK_SIZE (0x0000007fc0000000)
/* This decides where the kernel will search for a free chunk of vm * space during mmap's. */ #define IA32_PAGE_OFFSET ((current->personality & ADDR_LIMIT_3GB) ? 0xc0000000 : 0xFFFFe000) #define TASK_UNMAPPED_32 (PAGE_ALIGN(IA32_PAGE_OFFSET / 3)) #define TASK_UNMAPPED_64 PAGE_ALIGN(TASK_SIZE/3) #define TASK_UNMAPPED_BASE \ (test_thread_flag(TIF_IA32) ? TASK_UNMAPPED_32 : TASK_UNMAPPED_64)
So the problem is not related to the way the OS maps physical memory into virtual memory. Could it be possible that the motherboard is tuned to support mainly a 32-bit system and by that reason it uses the last megabytes right before the 4 GB limit to allocate space for drivers? I've heard that Tyan is considering to provide a BIOS version designed to support only 64-bit operating systems, I hate to wait but maybe that's the solution.
Greeting from Toulose,
- Jose Luis
-- José Luis RICARDO-CHAVEZ Tel. +33-5-61-55-68-33 Laboratoire de Physique Quantique Fax +33-5-61-55-60-65 Université Paul Sabatier 118 route de Narbonne F-31062 Toulouse, France
e-mail: ricardo@irsamc.ups-tlse.fr
Hi Rich,
The memory allocation test is done with a simple F90 program:
PROGRAM bigalloc IMPLICIT NONE INTEGER :: i INTEGER, PARAMETER :: SIZE = 800 * 1000 * 1000 REAL (KIND=8), ALLOCATABLE :: a(:) PRINT *,'Allocating big array...' ALLOCATE ( a ( SIZE ) ) ! Array initialization. a = 0.0d0 ! Do something. DO i = 1, 10 a = DBLE(i) END DO PRINT *,'Done.' PRINT *,'Freeing memory...' IF ( ALLOCATED ( a ) ) DEALLOCATE ( a ) PRINT *,'Done.' END PROGRAM bigalloc
The system has a Tyan S2882 motherboard with two 244 processors and 8 GB memory, it runs kernel version 2.4.21-193-smp (SuSE 9.0 + SiI SATA drivers). The compiler version is pgf90 5.1-2 (linux86-64) and the F90FLAGS are " -i8 -O -tp k8-64". The kernel message is: bigalloc[6564]: segfault at 0000002b1318e000 rip 0000000000401b40 rsp 0000007fbffff560 error 6
Regards,
- Jose Luis
Hi Odon,
the program is able to allocate a maximum of 4 GB per array, i.e., on my system I can use two ~4GB arrays without problems. But for a single >4GB array it doesn't work. Btw, using the -mcmodel=medium option gives the same result: bigalloc[6919]: segfault at 0000002b1393e000 rip 0000000000400e30 rsp 0000007fbffff560 error 6 Greetings from Toulouse, - Jose Luis -- José Luis RICARDO-CHAVEZ Tel. +33-5-61-55-68-33 Laboratoire de Physique Quantique Fax +33-5-61-55-60-65 Université Paul Sabatier 118 route de Narbonne F-31062 Toulouse, France e-mail: ricardo@irsamc.ups-tlse.fr