[opensuse-kernel] Problem with memory allocation / management on SLES 11 SP4
Dear list, We are facing a strange behaviour regarding memory allocation for mysqld ( MySQL community release 5.7.21 ) on SLES 11 SP4. We have a database server ( tested on both virtualized and bare metal hardware ) with 24GB of RAM and 8GB of swap. This mysqld is configured for our special needs ( not many connections, but there could be 1 - 2 transactions that needs big caches and 512MB tmp table in memory, ... ). We are hosting it on RHEL 7 and SLES 11 SP4. On RHEL, everything is working smoothly, but on SLES, we are getting constant out of memory messages from mysqld (nor from kernel) and mysqld crashes. During investigation, i have noticed, that RHEL apparently allows to allocate more memory than actually available ( the mysqld uses roughly 20 - 40% of the allocated memory ). I first thought that this is a feature enabled in RHEL and disabled in SLES ( i have found out that the feature name is "memory overcommit" ), but it is set exactly the same on RHEL and on SLES, so the issue must be elsewhere (but anyway, i have tried to set vm.overcommit_ratio = 100, vm.overcommit_memory = 1, but without any change) . So, i would like to ask anyone with knowledge of linux kernel memory management and knowledge of SLES 11 kernel or mysqld, to give us any advice on why is SLES acting differently than RHEL. Some debug info: top, free, mysqld.log, /proc/meminfo and ulimit -a shortly before the crash: https://paste.opensuse.org/24626549 mysql "show variables like '%buffer%'; show variables like '%cache%';" https://paste.opensuse.org/58205484 mysql buffer pool and some other info: https://paste.opensuse.org/56264627 https://paste.opensuse.org/48434446 And some of the mysqld backtraces: https://paste.opensuse.org/88004779 The symptoms and steps to reproduce are: 1) Start mysqld 2) in top, VIRT immidiatelly raises to 21 GB, RES about 2 GB, %MEM is on 11% 3) In about 5 minutes, VIRT raises to 25 GB, RES to 4-5 GB and %MEM to about 19% 4) Now i start getting messages in mysqld log "[ERROR] /usr/sbin/mysqld: Out of memory (Needed 269312 bytes)", but "free" utility reports still 17 GB of ram free (and 0GB of swap used) 5) mysqld crashes On my last try, i got crash at: 19.3% MEM, 25G VIRT, 4.5G RES but, free reported: Mem: 24018M total, 7679M used, 16339M free, 104M buffers, Swap: 7993M total, 0M used, 7993M free, 2331M cached So, i am lost here. What exactly denies mysqld more memory and what are the possible configuration (kernel most probably, since RHEL works without problems) options for memory allocation? Thanks anyone for your time to think about this. I have already tried to solve this with some of the mysql experts, but with no outcome. Best regards Lukáš Krejza
participants (1)
-
Lukáš Krejza