Hi Vlastimil, Am 29.09.18 um 08:04 schrieb Vlastimil Babka:
On 9/29/18 7:53 AM, Vlastimil Babka wrote:
Tracing should help here, without recompiling the kernel, install trace-cmd and run:
trace-cmd record -T -e kmalloc -f bytes_alloc==2048
And after a while (while kmalloc-2048 grows), stop it and check the output of trace-cmd report.
Actually replace "trace-cmd report" with "trace-cmd hist" which nicely groups the events by processes and allocation sites and makes it obvious which were the frequent ones.
Thanks, this was really useful (why didn't I know about trace-cmd before? ;-) strolchi:~ # timeout -s 2 30 trace-cmd record -T -e kmalloc -f bytes_alloc==2048 Hit Ctrl^C to stop recording CPU0 data recorded at offset=0x5e6000 24576 bytes in size CPU1 data recorded at offset=0x5ec000 0 bytes in size CPU2 data recorded at offset=0x5ec000 4096 bytes in size CPU3 data recorded at offset=0x5ed000 0 bytes in size And I think I have a winner: %92.90 (1950) gkrellm kmalloc #144 | --- *kmalloc* | |--%100.00-- kmem_cache_alloc_trace # 144 | |--%60.42-- vmstat_start # 87 | seq_read | proc_reg_read | __vfs_read | vfs_read | ksys_read | do_syscall_64 | entry_SYSCALL_64_after_hwframe | |--%39.58-- cfg80211_sinfo_alloc_tid_stats # 57 sta_set_sinfo ieee80211_get_station | |--%50.88-- cfg80211_wireless_stats # 29 | wireless_dev_seq_show | seq_read | proc_reg_read | __vfs_read | vfs_read | ksys_read | do_syscall_64 | entry_SYSCALL_64_after_hwframe | |--%49.12-- cfg80211_wext_giwrate # 28 ioctl_standard_call wext_handle_ioctl sock_ioctl do_vfs_ioctl ksys_ioctl __x64_sys_ioctl do_syscall_64 entry_SYSCALL_64_after_hwframe At least /proc/net/wireless (used by gkrellm-wifi plugin) is leaking: seife@strolchi:~> sudo grep ^kmalloc-2048 /proc/slabinfo ;\ for i in `seq 1 10000`; do cat /proc/net/wireless > /dev/null ; done;\ sudo grep ^kmalloc-2048 /proc/slabinfo kmalloc-2048 98046 98046 2048 2 1 : tunables 24 12 8 : slabdata 49023 49023 0 kmalloc-2048 108002 108002 2048 2 1 : tunables 24 12 8 : slabdata 54001 54001 0 The same with 1000 "iwconfig" runs: kmalloc-2048 112020 112020 2048 2 1 : tunables 24 12 8 : slabdata 56010 56010 0 kmalloc-2048 114012 114012 2048 2 1 : tunables 24 12 8 : slabdata 57006 57006 0 I'll report that upstream. -- Stefan Seyfried "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled." -- Richard Feynman -- To unsubscribe, e-mail: opensuse-kernel+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kernel+owner@opensuse.org