[opensuse] I was getting dovecot errors: mmap() ... Cannot allocate memory
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I was getting this error: dovecot - - - imap(cer): Error: mmap() failed with file /home/cer/Mail/_Lists/.imap/one_mail_folder/dovecot.index.cache: Cannot allocate memory The file was just 38 MB, so not that large. I deleted the indexes, same problem. The problem only appeared when I was syncing email from my laptop to my desktop, with this (trimmed for clarity): imapsync --host1 LAPTOP --user1 USER -passfile1 .secret_imapsync \ --host2 DESKTOP --user2 USER --passfile2 .secret_imapsync \ --folderrec _Lists I googled a bit, and found a similar problem here: http://www.dovecot.org/list/dovecot/2012-August/137569.html In that case, the error was with "lmtp", not "imap". Running "doveconf", they see: default_vsz_limit = 256 M service lmtp { inet_listener lmtp { port = 24 } vsz_limit = 256 M } and they tell him to increase that "vsz_limit". So I run "doveconf" on my machine, and I get: default_vsz_limit = 256 M ... service imap { chroot = client_limit = 1 drop_priv_before_exec = no executable = imap extra_groups = group = idle_kill = 0 privileged_group = process_limit = 1024 process_min_avail = 0 protocol = imap service_count = 1 type = unix_listener login/imap { group = mode = 0666 user = } user = vsz_limit = 18446744073709551615 B } Notice that the limit is really huge! If the thing tries to reserve that ammount of memory, mmap will say "no way!" So I edit "/etc/dovecot/local.conf", and add this: service imap { # Most of the memory goes to mmap()ing files. You may need to increase this # limit if you have huge mailboxes. #vsz_limit = $default_vsz_limit # this results in vsz_limit = 18446744073709551615 B vsz_limit = 512 M # Max. number of IMAP processes (connections) #process_limit = 1024 } I restart dovecot, restart the imapsync process, and bingo, success. Now the problem is, where comes that huge "vsz_limit = 18446744073709551615 B" from? Is it a bug? See: Telcontar:/etc/dovecot # doveconf | grep vsz_limit default_vsz_limit = 256 M vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B vsz_limit = 18446744073709551615 B ... Everything is using that huge limit, and ignoring the "default_vsz_limit = 256 M" setting. I see that error has appeared a few times in my logs before, the earliest on 2012-10-05. - -- Cheers Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlOg3k4ACgkQtTMYHG2NR9VKNQCeP81ePrx/+2gBEXIJW87N5ARs RxEAnA2+Nl9PKIryYpgiBsNOhFAwTAR9 =meTi -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
El 17/06/14 20:33, Carlos E. R. escribió:
Now the problem is, where comes that huge "vsz_limit = 18446744073709551615 B" from?
Is it a bug?
Nope. that huge number is SIZE_MAX,, the maximum size you can request when creating a mapping in the virtual address space. -- Cristian "I don't know the key to success, but the key to failure is trying to please everybody." -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2014-06-18 02:43, Cristian Rodríguez wrote:
El 17/06/14 20:33, Carlos E. R. escribió:
Now the problem is, where comes that huge "vsz_limit = 18446744073709551615 B" from?
Is it a bug?
Nope. that huge number is SIZE_MAX,, the maximum size you can request when creating a mapping in the virtual address space.
Ok, yes, but that would be an mmap() limit, right? But dovecot is supposed to not request more than default_vsz_limit, which is just 256 MiB. Look, /etc/dovecot/conf.d/10-master.conf has this: # Default VSZ (virtual memory size) limit for service processes. This is mainly # intended to catch and kill processes that leak memory before they eat up # everything. #default_vsz_limit = 256M ... service imap { # Most of the memory goes to mmap()ing files. You may need to increase this # limit if you have huge mailboxes. #vsz_limit = $default_vsz_limit # Max. number of IMAP processes (connections) #process_limit = 1024 } which I assume to mean that the default is "vsz_limit = $default_vsz_limit", and there is no need to specify it, ie, 256 MiB. That's the setting the openSUSE package contains. But running "doveconf" I get the actual configuration of dovecot in use, which is: default_vsz_limit = 256 M .... service imap { ... vsz_limit = 18446744073709551615 B } So, what I'm saying is that dovecot is not honoring the default max size of 255 MiB, and probably it has to be explicitly specified. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlOg53MACgkQtTMYHG2NR9VeCQCfUn97hydVpufRuqbbQgQAIcne HwgAnitzUwkYkQdiFnGmCY6/fFwkz9Vy =1DER -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
El 17/06/14 21:12, Carlos E. R. escribió:
On 2014-06-18 02:43, Cristian Rodríguez wrote:
El 17/06/14 20:33, Carlos E. R. escribió:
Now the problem is, where comes that huge "vsz_limit = 18446744073709551615 B" from?
Is it a bug?
Nope. that huge number is SIZE_MAX,, the maximum size you can request when creating a mapping in the virtual address space.
Ok, yes, but that would be an mmap() limit, right?
Yes. If default_vsz_limit is a setting that is supposed to apply globally in case vsz_limit is not set per section, then there is a bug somewhere. In any case, you can probably set this to 0 in linux (that disables the limit, according to the documentation), it is up to the kernel work efficiently. note that this does not mean the application will actually use unlimited resources or that such allocations will be considered memory in-use... -- Cristian "I don't know the key to success, but the key to failure is trying to please everybody." -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-06-18 05:03, Cristian Rodríguez wrote:
El 17/06/14 21:12, Carlos E. R. escribió:
Ok, yes, but that would be an mmap() limit, right?
Yes.
If default_vsz_limit is a setting that is supposed to apply globally in case vsz_limit is not set per section, then there is a bug somewhere.
Mmm. I'll add a note to my To-Do list, report some time. ;-)
In any case, you can probably set this to 0 in linux (that disables the limit, according to the documentation), it is up to the kernel work efficiently. note that this does not mean the application will actually use unlimited resources or that such allocations will be considered memory in-use...
Yes... but that is not the problem here, but that dovecot doesn't heed its own default limit, unless there was a config change and it is not supposed to, in which case it is the config which is wrong. The configuration files hint/say somewhere that they impose a limit on mmap calls because they know that some components may become mad and request insane amounts of memory. So they impose a limit, and crash or kill the component when it reaches the limit. This is not happening. -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
El 18/06/14 05:31, Carlos E. R. escribió:
Yes... but that is not the problem here, but that dovecot doesn't heed its own default limit, unless there was a config change and it is not supposed to, in which case it is the config which is wrong.
Then there is a bug.
The configuration files hint/say somewhere that they impose a limit on mmap calls because they know that some components may become mad and request insane amounts of memory.
Requesting large amounts of virtual memory, does not mean memory will be used.. we are back again to the discussion we had a while ago about the journal memory usage..remember that one? So they impose a limit, and crash or
kill the component when it reaches the limit. This is not happening.
This looks like an horrible hack that you should not be relying on.. -- Cristian "I don't know the key to success, but the key to failure is trying to please everybody." -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2014-06-18 16:42, Cristian Rodríguez wrote:
El 18/06/14 05:31, Carlos E. R. escribió:
Yes... but that is not the problem here, but that dovecot doesn't heed its own default limit, unless there was a config change and it is not supposed to, in which case it is the config which is wrong.
Then there is a bug.
Looks that way.
The configuration files hint/say somewhere that they impose a limit on mmap calls because they know that some components may become mad and request insane amounts of memory.
Requesting large amounts of virtual memory, does not mean memory will be used.. we are back again to the discussion we had a while ago about the journal memory usage..remember that one?
Yes, I think so. I think that what happens is that, for some reason, the imap service process of dovecot requested a huge amount of memory, and mmap() said "no". I don't know how much it really requested, nor how much of that it used or was going to use - only that it requested "too much".
So they impose a limit, and crash or
kill the component when it reaches the limit. This is not happening.
This looks like an horrible hack that you should not be relying on..
But it is not mine, it is theirs ;-) -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
participants (3)
-
Carlos E. R.
-
Carlos E. R.
-
Cristian Rodríguez