More information from kmem below, attached as well to preserve columns. -Cameron
On 5/3/2010 at 01:47 PM, in message <20100503194736.GH3470@quack.suse.cz>, Jan Kara <jack@suse.cz> wrote: Hello,
On Mon 03-05-10 13:12:40, Cameron Seader wrote:
First, we're given an inode 'ainode', which should be the correct inode for the file we're looking at. (If it were incorrect, we would have gotten an error much earlier.)
If we have iget, we call iget. The 2.6.16.60-* kernels lack iget, I believe, so instead we do: So we are talking about SLE10 based kernels, right? In fact these kernels do have iget() but I guess you do not want to do all the writing by hand and want to use standard write path and thus you need open file descriptor for which you need a dentry...
fid.i32.ino = ainode; fid.i32.gen = 0; dp = afs_cacheSBp->s_export_op->fh_to_dentry(afs_cacheSBp, &fid, sizeof(fid), FILEID_INO32_GEN); filp = dentry_open(dp, mntget(afs_cacheMnt), O_RDWR); Hmm, so about which kernel are we speaking? fh_to_dentry has been introduced only in 2.6.24...
<snip>
It's not intended to be written to constantly, but in this case it probably is written to several times successively (due to certain parameters set a bit low, and the high load for these clients).
I believe this function was called about 644203 times in the core I'm looking at, which means that file was written to at least around 644000 times... I'm assuming at least most of those were right after another. OK, so these 644000 writes succeed and then you start getting ENOMEM? Does the machine have enough free memory? If not, output from /proc/meminfo and /proc/slabinfo could help to tell you where the memory has gone.
However, there would almost always be several reads of the same file between successive writes. (Again, in an 'open(); read(); close();' fashion) But they are probably all happening very quickly; I assume the cache for the stuff in this file is thrashing. Well, the cache could be thrashing but still you'll get ENOMEM only if the kernel cannot find enough memory to pull in a page you are writing to. And that should not happen unless the machine has real problems. My personal tip would be that your code leaks some memory (or reference or so) and thus kernel really gets out of memory after enough reading / writing...
Honza
crash> kmem -f NODE 0 ZONE NAME SIZE FREE MEM_MAP START_PADDR START_MAPNR 0 DMA 4096 3029 ffff810450b00000 0 0 AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES 0 4k ffff81000001b448 3 3 1 8k ffff81000001b460 1 2 2 16k ffff81000001b478 2 8 3 32k ffff81000001b490 5 40 4 64k ffff81000001b4a8 2 32 5 128k ffff81000001b4c0 2 64 6 256k ffff81000001b4d8 1 64 7 512k ffff81000001b4f0 0 0 8 1024k ffff81000001b508 1 256 9 2048k ffff81000001b520 1 512 10 4096k ffff81000001b538 2 2048 ZONE NAME SIZE FREE MEM_MAP START_PADDR START_MAPNR 1 DMA32 1044480 586009 ffff810450b38000 1000000 4096 AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES 0 4k ffff81000001bac8 8147 8147 1 8k ffff81000001bae0 8005 16010 2 16k ffff81000001baf8 6839 27356 3 32k ffff81000001bb10 5464 43712 4 64k ffff81000001bb28 4128 66048 5 128k ffff81000001bb40 2979 95328 6 256k ffff81000001bb58 1559 99776 7 512k ffff81000001bb70 818 104704 8 1024k ffff81000001bb88 286 73216 9 2048k ffff81000001bba0 63 32256 10 4096k ffff81000001bbb8 19 19456 ZONE NAME SIZE FREE MEM_MAP START_PADDR START_MAPNR 2 Normal 3538944 998117 ffff810454300000 100000000 1048576 AREA SIZE FREE_AREA_STRUCT BLOCKS PAGES 0 4k ffff81000001c148 100359 100359 1 8k ffff81000001c160 159675 319350 2 16k ffff81000001c178 68930 275720 3 32k ffff81000001c190 30774 246192 4 64k ffff81000001c1a8 3179 50864 5 128k ffff81000001c1c0 44 1408 6 256k ffff81000001c1d8 2 128 7 512k ffff81000001c1f0 0 0 8 1024k ffff81000001c208 0 0 9 2048k ffff81000001c220 0 0 10 4096k ffff81000001c238 4 4096 ZONE NAME SIZE FREE MEM_MAP START_PADDR START_MAPNR 3 HighMem 0 0 0 0 0 nr_free_pages: 1587155 (verified) crash> kmem -s CACHE NAME OBJSIZE ALLOCATED TOTAL SLABS SSIZE ffff81042bcfa840 nfs_direct_cache 184 0 0 0 4k ffff81042bd0b180 nfs_write_data 768 36 40 8 4k ffff81042bd0b800 nfs_read_data 768 32 35 7 4k ffff81042bd0c140 nfs_inode_cache 1024 160639 160644 40161 4k ffff81042bd0c7c0 nfs_page 128 31 180 6 4k ffff81042c3e1100 rpc_buffers 2048 8 8 4 4k ffff81042c3e1780 rpc_tasks 384 8 10 1 4k ffff81044ba040c0 rpc_inode_cache 832 52 52 13 4k ffff81044ba04740 afs_inode_cache 896 218657 218740 54685 4k ffff81044b887080 fib6_nodes 64 5 59 1 4k ffff81044b887700 ip6_dst_cache 320 4 12 1 4k ffff81044b894040 ndisc_cache 256 1 15 1 4k ffff81044b8946c0 RAWv6 896 11 12 3 4k ffff81044b8962c0 UDPv6 896 3 8 2 4k ffff81044b896940 tw_sock_TCPv6 192 0 0 0 4k ffff810450590280 request_sock_TCPv6 128 0 0 0 4k ffff810450590900 TCPv6 1664 1 4 1 8k ffff81044ced6240 ip_fib_alias 64 11 59 1 4k ffff81044ced68c0 ip_fib_hash 64 11 59 1 4k ffff81044c86a200 dm_events 40 16 92 1 4k ffff81044c86a880 dm_tio 24 0 0 0 4k ffff81044ed8e180 dm_io 40 0 0 0 4k ffff81044fc121c0 uhci_urb_priv 80 0 0 0 4k ffff81044fc12840 ext3_inode_cache 800 90438 90730 18146 4k ffff81044e828100 ext3_xattr 88 0 0 0 4k ffff81044ed8e800 journal_handle 24 0 144 1 4k ffff81044ee25140 journal_head 96 32 200 5 4k ffff81044ee257c0 revoke_table 16 10 202 1 4k ffff810450381080 revoke_record 32 0 0 0 4k ffff81044e828780 scsi_cmd_cache 384 6 20 2 4k ffff81044edb1740 sgpool-256 8192 32 32 32 8k ffff81044edc2080 sgpool-128 4096 32 32 32 4k ffff81044edc2700 sgpool-64 2048 32 32 16 4k ffff81044ee63040 sgpool-32 1024 32 32 8 4k ffff81044ee636c0 sgpool-16 512 32 32 4 4k ffff81044ee672c0 sgpool-8 256 32 45 3 4k ffff81044ee67940 scsi_io_context 112 0 0 0 4k ffff81044fe15280 UNIX 704 13 55 5 8k ffff81044fe15900 ip_mrt_cache 128 0 0 0 4k ffff81044f692240 tcp_bind_bucket 32 51 224 2 4k ffff81044f6928c0 inet_peer_cache 128 12 30 1 4k ffff81044f6fb200 secpath_cache 192 0 0 0 4k ffff81044f6fb880 xfrm_dst_cache 384 0 0 0 4k ffff81044f6b21c0 ip_dst_cache 384 93 110 11 4k ffff81044f6b2840 arp_cache 256 2 15 1 4k ffff81044f69b180 RAW 768 9 10 2 4k ffff81044f69b800 UDP 768 14 25 5 4k ffff81044f68b140 tw_sock_TCP 192 6 40 2 4k ffff81044f68b7c0 request_sock_TCP 128 0 0 0 4k ffff81044fe55100 TCP 1536 47 70 14 8k ffff81044fe55780 flow_cache 128 0 0 0 4k ffff81044fdd80c0 msi_cache 5760 11 11 11 8k ffff81044fdd8740 cfq_ioc_pool 168 36 115 5 4k ffff81044fdd3080 cfq_pool 160 34 96 4 4k ffff81044fdd3700 crq_pool 88 4 44 1 4k ffff81044fdcd040 deadline_drq 96 0 0 0 4k ffff81044fdcd6c0 as_arq 112 0 0 0 4k ffff8104503dc2c0 mqueue_inode_cache 896 1 4 1 4k ffff8104503dc940 isofs_inode_cache 640 0 0 0 4k ffff81044fd17280 minix_inode_cache 656 0 0 0 4k ffff81044fd17900 hugetlbfs_inode_cache 608 1 6 1 4k ffff81044fd14240 ext2_inode_cache 752 0 0 0 4k ffff81044fd148c0 ext2_xattr 88 0 0 0 4k ffff81044fd53200 dnotify_cache 40 0 0 0 4k ffff81044fd53880 dquot 256 0 0 0 4k ffff81044fd591c0 eventpoll_pwq 72 1 53 1 4k ffff81044fd59840 eventpoll_epi 192 1 20 1 4k ffff81044fd6b180 inotify_event_cache 40 0 0 0 4k ffff81044fd6b800 inotify_watch_cache 72 1 53 1 4k ffff81044fd72140 kioctx 384 0 0 0 4k ffff81044fd727c0 kiocb 256 0 0 0 4k ffff81044fd78100 fasync_cache 24 0 0 0 4k ffff81044fd78780 shmem_inode_cache 816 475 495 99 4k ffff8104502a70c0 posix_timers_cache 152 0 0 0 4k ffff8104502a7740 uid_cache 128 7 30 1 4k ffff810450381700 blkdev_ioc 56 36 134 2 4k ffff81045037e040 blkdev_queue 1608 26 30 6 8k ffff81045037e6c0 blkdev_requests 288 4 26 2 4k ffff81044f40d2c0 biovec-(256) 4096 256 256 256 4k ffff81044f40d940 biovec-128 2048 256 256 128 4k ffff81044f409280 biovec-64 1024 256 256 64 4k ffff81044f409900 biovec-16 256 256 270 18 4k ffff8104502f6240 biovec-4 64 256 295 5 4k ffff8104502f68c0 biovec-1 16 256 404 2 4k ffff81044f4aa200 bio 128 256 270 9 4k ffff81044f4aa880 sock_inode_cache 704 107 150 30 4k ffff81045025e1c0 skbuff_fclone_cache 512 3 21 3 4k ffff81045025e840 skbuff_head_cache 256 278 570 38 4k ffff8104501dd180 file_lock_cache 176 4 44 2 4k ffff8104501dd800 acpi_operand 72 1403 1484 28 4k ffff8104501da140 acpi_parse_ext 64 0 0 0 4k ffff8104501da7c0 acpi_parse 40 0 0 0 4k ffff810450a5d100 acpi_state 88 0 0 0 4k ffff810450a5d780 delayacct_cache 64 237 472 8 4k ffff810450a530c0 taskstats_cache 296 5 39 3 4k ffff810450a53740 proc_inode_cache 624 1137 1254 209 4k ffff810450a50080 sigqueue 160 2 24 1 4k ffff810450a50700 radix_tree_node 536 56468 65996 9428 4k ffff810450a44040 bdev_cache 832 26 32 8 4k ffff810450a446c0 sysfs_dir_cache 80 3892 3936 82 4k ffff810450a412c0 mnt_cache 256 9551 9600 640 4k ffff810450a41940 inode_cache 592 10489 10554 1759 4k ffff810450a3d280 dentry_cache 208 518661 540417 28443 4k ffff810450a3d900 filp 256 768 2355 157 4k ffff810450a38240 names_cache 4096 5 11 11 4k ffff810450a388c0 key_jar 192 14 40 2 4k ffff810450a35200 idr_layer_cache 528 238 238 34 4k ffff810450a35880 buffer_head 88 86231 96580 2195 4k ffff810450a331c0 mm_struct 832 62 126 14 8k ffff810450a33840 vm_area_struct 184 2248 3696 176 4k ffff810450a2f180 fs_cache 64 63 236 4 4k ffff810450a2f800 files_cache 896 64 96 24 4k ffff810450a2e140 signal_cache 768 131 160 32 4k ffff810450a2e7c0 sighand_cache 2112 130 165 55 8k ffff810450a2b100 task_struct 1936 236 264 132 4k ffff810450a2b780 anon_vma 24 800 1440 10 4k ffff810450a250c0 shared_policy_node 56 0 0 0 4k ffff810450a25740 numa_policy 24 63 144 1 4k ffff810450a0e700 size-131072(DMA) 131072 0 0 0 128k ffff810450a0e080 size-131072 131072 0 0 0 128k ffff810450a0d6c0 size-65536(DMA) 65536 0 0 0 64k ffff810450a0d040 size-65536 65536 0 0 0 64k ffff810450a0c940 size-32768(DMA) 32768 0 0 0 32k ffff810450a0c2c0 size-32768 32768 5 5 5 32k ffff810450a0b900 size-16384(DMA) 16384 0 0 0 16k ffff810450a0b280 size-16384 16384 7 7 7 16k ffff810450a0a8c0 size-8192(DMA) 8192 0 0 0 8k ffff810450a0a240 size-8192 8192 47 47 47 8k ffff810450a09880 size-4096(DMA) 4096 0 0 0 4k ffff810450a09200 size-4096 4096 170 172 172 4k ffff810450a08840 size-2048(DMA) 2048 0 0 0 4k ffff810450a081c0 size-2048 2048 501 528 264 4k ffff810450a07800 size-1024(DMA) 1024 0 0 0 4k ffff810450a07180 size-1024 1024 10163 10280 2570 4k ffff810450a067c0 size-512(DMA) 512 0 0 0 4k ffff810450a06140 size-512 512 3078 3168 396 4k ffff810450a05780 size-256(DMA) 256 0 0 0 4k ffff810450a05100 size-256 256 10835 10950 730 4k ffff810450a04740 size-128(DMA) 128 0 0 0 4k ffff810450a040c0 size-64(DMA) 64 0 0 0 4k ffff810450a02700 size-64 64 133127 144314 2446 4k ffff810450a02080 size-32(DMA) 32 0 0 0 4k ffff810450a006c0 size-128 128 50244 53250 1775 4k ffff810450a00040 size-32 32 10872 11536 103 4k ffffffff80368ca0 kmem_cache 1664 144 146 73 4k