On Thu, Nov 10, 2011 at 10:15 AM, Jeff Mahoney <jeffm@suse.de> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 11/10/2011 11:06 AM, milan zimmermann wrote:
On Wed, Nov 9, 2011 at 3:41 PM, Jeff Mahoney <jeffm@suse.de> wrote:
virtual memory (kbytes, -v) 29536000
... but here is your problem.
I disagree. With the switches I used, and with
Then test it. On my system, I see:
jeffm@jetfire:~> ulimit -v unlimited jeffm@jetfire:~> ./a.out Hello world Allocating 29Gb... Success: 0x7fc7732d6000
jeffm@jetfire:~> ulimit -v 28000000 jeffm@jetfire:~> ./a.out Hello world Allocating 29Gb... FAILED
You are right, that is how it works, I agree (my "disagree" was too harsh there :) ). I should have said I am not sure with these particular flags ulimit should matter, as there should be no memory allocated (reserved). But I am only basic that on half-understanding this: http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting (I think you are describing the difference below, also a question there)
sysctl -w vm.overcommit_memory=0 # or 1
according to the link I posted
http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting
the cost of allocation with these parameters should be 0, and the call
mmap((void *) NULL, someSize * (((size_t) 1) << 30), PROT_NONE, MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE, -1, 0); # someSize is integer with number of Gigs
should succeed no matter what virtual memory, and no matter what size I am allocating.
As far as the kernel generally is concerned, yes. You're hitting user resource limits, not a kernel out-of-memory condition. If you had overcommit disabled, then you might run into that issue.
So it seems that the user resource limit is checked first. Let me ask a speculative question (but I would appreciate a comment as it may help getting Nacl working on Linux, it seems their code is non-reserving 84Gb this way) :) So the question: Do you think that on any Linux, with ulimit virtual memory set to X Gb, this call will always fail: mmap((void *) NULL, (X+1) * (((size_t) 1) << 30), PROT_NONE, MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE, -1, 0); Thanks (one more comment below)
So far I have no other explanation then this is a bug (anyone agrees?). On the chrome list, people indicated same issue was fixed in Ubuntu lately (mmap 84Gb succeds with way less virtual memory). I only have 4 boxes with Opensuse ... will also try Opensuse 12.1 and report
I blame Windows for perpetuating this myth that "virtual memory" = "real memory + swap" since that's not at all what it means. Virtual memory is the address space. You're allocating part of the address space but you're not using it.
That is why I thought the user limit should be checked at the point I actually use it (not during MAP_NORESERVE), but certainly have little support for that :) Thanks for your help and comments, milan
- -Jeff
let me know if i should run anything else, thanks
On my 11.4 64-bit system, I can't reproduce your failure. I see:
jeffm@sled2:~> ./a.out Hello world Allocating 29Gb... Success: 0x7f7f79e57000 jeffm@sled2:~> uname -a Linux sled2 2.6.37.6-0.9-desktop #1 SMP PREEMPT 2011-10-19 22:33:27 +0200 x86_64 x86_64 x86_64 GNU/Linux
-Jeff
>> I am trying to resolve a failure of Google Native >> Client in Opensuse 64 11.4, discussed in Google Native >> Client forum: >> >> [1] >> https://groups.google.com/forum/#!topic/native-client-discuss/7DUFfi_BxqM >> >> >>
>> To repeat the issue in a nutshell:: >> --------------------------------------------- >> >> Google native client (Chrome) works on recent versions >> of at least Ubuntu [1], but fails on Opensuse 11.4 >> (with all latest updates up to Nov 4). This failure can >> be reproduced in chrome 14, 15, 16 (from >> http://dl.google.com/linux/chrome/rpm/stable/x86_64) >> and verified by loading >> >> [2] >> http://www.gonacl.com/dev/demos/sdk_examples/load_progress/load_progress.htm... >> >> >>
>> The problem / question for opensuse (kernel?) :: >> ------------------------------------------------------------------ >> >> >>
>> There is a long discussion in the above thread, to get to the >> point quickly: The Google guys identified an issue >> with mmap() with MAP_NORESERVE (see below). They >> believe it may be a bug or a kernel configuration >> issue(?) >> >> A Chrome Nacl person suggest the following code should >> print "Success" but it fails in my testing: >> >> #include <stdio.h> #include <sys/mman.h> >> >> int main(void) { void *addr; >> >> printf("Hello world\nAllocating 29Gb...\n"); addr = >> mmap((void *) NULL, 29 * (((size_t) 1) << 30), >> PROT_NONE, MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE, >> -1, 0); /* test 29 or other values */ if (MAP_FAILED == >> addr) { printf("FAILED\n"); } else { printf("Success: >> %p\n", addr); } return 0; } >> >> This prints FAILED on Opensuse 11.4 64 bit. >> >> I did some experiments. On my system: >> >> # cat /proc/meminfo MemTotal: 15948428 kB >> MemFree: 11270612 kB .... CommitLimit: 28945728 kB >> Committed_AS: 4918284 kB ... >> >> From running the test program above, it looks like >> *CommitLimit* is clearly used as upper limit of >> mmap(MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE), *no >> matter what vm.overcommit_memory* flag is used. >> >> In concrete terms: >> >> mmap((void *) NULL, 29 * (((size_t) 1) << 30), >> PROT_NONE, MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE, >> -1, 0); // always FAILS with value of 29 or higher >> >> mmap((void *) NULL, 28 * (((size_t) 1) << 30), >> PROT_NONE, MAP_ANONYMOUS | MAP_NORESERVE | MAP_PRIVATE, >> -1, 0); // always SUCCEEDS with value of 28 or lower >> >> No matter what setting of sysctl -w >> vm.overcommit_memory=0 # or 1 or 2 >> >> (This seems a Opensuse 11.4 bug in modes 0 and 1, as >> according to [2] anonymous private readonly should have >> 0 cost) >> >> Any comments or solutions or how to fix this? >> >> Thanks, >> >> Milan >> >> [1] >> http://www.mjmwired.net/kernel/Documentation/filesystems/proc.txt >> >>
>> [2] >> http://www.mjmwired.net/kernel/Documentation/vm/overcommit-accounting >> >> >> >> >>
>> ======================= PS: I am attaching a few pieces of info >> about my system that may be relevant: >> >> The hardware is AMD 4 core AMD Athlon II X4 610e and >> has 16Gb (sixteen) of memory, running very little (just >> KDE desktop at this point) >> >> # swapon -s Filename >> Type Size Used Priority /dev/sda1 partition >> 20971516 0 -1 >> >> # cat /proc/sys/vm/overcommit_memory 0 >> >> # ulimit -a core file size (blocks, -c) 0 data >> seg size (kbytes, -d) unlimited scheduling priority >> (-e) 0 file size (blocks, -f) unlimited >> pending signals (-i) 123980 max locked memory >> (kbytes, -l) 64 max memory size (kbytes, -m) 13556224 >> open files (-n) 1024 pipe size (512 bytes, >> -p) 8 POSIX message queues (bytes, -q) 819200 real-time >> priority (-r) 0 stack size (kbytes, -s) >> 8192 cpu time (seconds, -t) unlimited max user >> processes (-u) 123980 virtual memory >> (kbytes, -v) 29536000 file locks (-x) unlimited >> >> # df Filesystem 1K-blocks Used Available >> Use% Mounted on rootfs 82567856 >> 15558248 62815408 20% / devtmpfs 7934744 >> 244 7934500 1% /dev tmpfs 7974212 >> 1592 7972620 1% /dev/shm /dev/sda2 >> 82567856 15558248 62815408 20% / /dev/sda3 377510440 >> 90623252 267710692 26% /home
- -- Jeff Mahoney SUSE Labs -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.18 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBAgAGBQJOu/iYAAoJEB57S2MheeWywEAP/ROIa5GQo+wasRACfKNH6xJd /4E0MBxWbKBduorirabz786vIuZZ7xKzX0ZzakBh3xO0896SFTbzukJRPGWO+3tG hf1xrS+BiRQywIgS+d4xDLspLAgiXh2RlH5ANPaeE+jEy3+lCvDM/aH0UiOcGkGd nt9fQemP3QmoUSAK1xl+tXsHvK4yP5g/If/JDFiVhmNpSTRA3w8qqwL6PCDLnzH9 dL0XqN8e72a+xXCvkgPDLHFevVs2xkjSwJmtDpDnO9z1Bxj8lyWhpjl03mUu9jh9 0gxbT/soEr8YqBT9OtmX/o3OHZ3Ex7jHweEv10DmNg2H8z7jf8LMm/mTor4OXAQH PSoEwVfauBLhNMwWc+WYmBK++Pq79DIHR0jCe7BgKshr9QyL9fGzRqxPG8lKiXyZ oraBV7Ts9GLiWrTc4r3m3/I/tSAJJjYsqeJTwdiyTPNXsrDWowl5w3gjJRYmig0q ZIdFvBw3BoxvqkB3P4EwaC5HY41kqMquPmc1onvFW05yRxXAMahKsFu0Xn4/RG7R 85rf60IvD+vnXxw+y0Vu7qF4WdYV3sA/oqFrHQA4Q3bVP+6BdLwoVRVIjikjRwyL BURYGjFPcdPHoZs2rvaTIM9ssZXjHZyuvj/qTXQ5ML4li0CmWBxV3R0BC72vO/w2 foqHlyXZ0jrmJCcQR8dv =iqrE -----END PGP SIGNATURE-----
-- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org