[opensuse] 'du' explanation, please
I have two directories, one being a copy of the other: # l temp* temp: total 3860 drwxr-xr-x 2 root root 3846144 2015-10-11 10:29 ./ drwxr-xr-x 9 root root 40960 2015-10-18 10:06 ../ -rw-r--r-- 1 root root 3639 2015-10-11 09:45 file1 -rw-r--r-- 1 root root 4057 2015-10-11 09:45 file2 -rw-r--r-- 1 root root 3568 2015-10-11 09:45 file3 -rw-r--r-- 1 root root 4194 2015-10-11 10:12 file4 -rw-r--r-- 1 root root 3861 2015-10-11 10:12 file5 temp2: total 88 drwxr-xr-x 2 root root 40 2015-10-18 10:07 ./ drwxr-xr-x 9 root root 40960 2015-10-18 10:06 ../ -rw-r--r-- 1 root root 3639 2015-10-11 09:45 file1 -rw-r--r-- 1 root root 4057 2015-10-11 09:45 file2 -rw-r--r-- 1 root root 3568 2015-10-11 09:45 file3 -rw-r--r-- 1 root root 4194 2015-10-11 10:12 file4 -rw-r--r-- 1 root root 3861 2015-10-11 10:12 file5 with 'du', the sizes of temp and temp2 : # du -hsx temp temp2 3.8M temp 28K temp2 I know this question has been asked before, but I wasn't paying attention. -- Per Jessen, Zürich (6.9°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
18.10.2015 11:26, Per Jessen пишет:
I have two directories, one being a copy of the other:
# l temp* temp: total 3860 drwxr-xr-x 2 root root 3846144 2015-10-11 10:29 ./ drwxr-xr-x 9 root root 40960 2015-10-18 10:06 ../ -rw-r--r-- 1 root root 3639 2015-10-11 09:45 file1 -rw-r--r-- 1 root root 4057 2015-10-11 09:45 file2 -rw-r--r-- 1 root root 3568 2015-10-11 09:45 file3 -rw-r--r-- 1 root root 4194 2015-10-11 10:12 file4 -rw-r--r-- 1 root root 3861 2015-10-11 10:12 file5
temp2: total 88 drwxr-xr-x 2 root root 40 2015-10-18 10:07 ./ drwxr-xr-x 9 root root 40960 2015-10-18 10:06 ../ -rw-r--r-- 1 root root 3639 2015-10-11 09:45 file1 -rw-r--r-- 1 root root 4057 2015-10-11 09:45 file2 -rw-r--r-- 1 root root 3568 2015-10-11 09:45 file3 -rw-r--r-- 1 root root 4194 2015-10-11 10:12 file4 -rw-r--r-- 1 root root 3861 2015-10-11 10:12 file5
with 'du', the sizes of temp and temp2 :
# du -hsx temp temp2 3.8M temp 28K temp2
Well, size of *directory* is 3.8MB indeed. When files are deleted directories do not shrink (at least in those filesystems I am ware of) so if you once had millions of files there and then deleted them directory size remains big. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, Oct 18, 2015 at 1:36 AM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
Well, size of *directory* is 3.8MB indeed. When files are deleted directories do not shrink (at least in those filesystems I am ware of) so if you once had millions of files there and then deleted them directory size remains big.
I was about to respond when Andrei sent his message. He is correct. Some filesystems like Ext4 and UFS will not reclaim blocks when inode entries are removed. Overtime the directory size will grow. Don't worry about it. Some newer filesystems such as XFS will not grow with such behavior. Brandon Vincent -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkov wrote:
18.10.2015 11:26, Per Jessen пишет:
I have two directories, one being a copy of the other:
# l temp* temp: total 3860 drwxr-xr-x 2 root root 3846144 2015-10-11 10:29 ./ drwxr-xr-x 9 root root 40960 2015-10-18 10:06 ../ -rw-r--r-- 1 root root 3639 2015-10-11 09:45 file1 -rw-r--r-- 1 root root 4057 2015-10-11 09:45 file2 -rw-r--r-- 1 root root 3568 2015-10-11 09:45 file3 -rw-r--r-- 1 root root 4194 2015-10-11 10:12 file4 -rw-r--r-- 1 root root 3861 2015-10-11 10:12 file5
temp2: total 88 drwxr-xr-x 2 root root 40 2015-10-18 10:07 ./ drwxr-xr-x 9 root root 40960 2015-10-18 10:06 ../ -rw-r--r-- 1 root root 3639 2015-10-11 09:45 file1 -rw-r--r-- 1 root root 4057 2015-10-11 09:45 file2 -rw-r--r-- 1 root root 3568 2015-10-11 09:45 file3 -rw-r--r-- 1 root root 4194 2015-10-11 10:12 file4 -rw-r--r-- 1 root root 3861 2015-10-11 10:12 file5
with 'du', the sizes of temp and temp2 :
# du -hsx temp temp2 3.8M temp 28K temp2
Well, size of *directory* is 3.8MB indeed. When files are deleted directories do not shrink (at least in those filesystems I am ware of) so if you once had millions of files there and then deleted them directory size remains big.
Thanks Andrei, Brandon - does that directory then actually take up that amount of physical space? This directory (and many others) are essential fifo queues where many new files are written and deleted daily. -- Per Jessen, Zürich (7.1°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
18.10.2015 11:47, Per Jessen пишет:
Thanks Andrei, Brandon - does that directory then actually take up that amount of physical space?
Yes. du shows actual space comsumption on disk.
This directory (and many others) are essential fifo queues where many new files are written and deleted daily.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, Oct 18, 2015 at 1:47 AM, Per Jessen <per@computer.org> wrote:
Thanks Andrei, Brandon - does that directory then actually take up that amount of physical space? This directory (and many others) are essential fifo queues where many new files are written and deleted daily.
The directory is indeed taking up that space. The reason for leaving the directory entries intact although the files have been deleted is to reduce filesystem fragmentation. Filesystems that require the occasional defragmentation will usually free up filesystem blocks by removing directory entries. Brandon Vincent -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Brandon Vincent wrote:
On Sun, Oct 18, 2015 at 1:47 AM, Per Jessen <per@computer.org> wrote:
Thanks Andrei, Brandon - does that directory then actually take up that amount of physical space? This directory (and many others) are essential fifo queues where many new files are written and deleted daily.
The directory is indeed taking up that space. The reason for leaving the directory entries intact although the files have been deleted is to reduce filesystem fragmentation.
So as long as the directory entries are being reused, not much space is "wasted"? -- Per Jessen, Zürich (7.1°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, Oct 18, 2015 at 2:06 AM, Per Jessen <per@computer.org> wrote:
So as long as the directory entries are being reused, not much space is "wasted"?
I may have not been that clear. As you add and delete files into a directory, the size of the directory entry will grow on an Ext4 or similar filesystem. The space is wasted in a sense, but a couple of megabytes is usually not an issue to most. Brandon Vincent -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Brandon Vincent wrote:
On Sun, Oct 18, 2015 at 2:06 AM, Per Jessen <per@computer.org> wrote:
So as long as the directory entries are being reused, not much space is "wasted"?
I may have not been that clear. As you add and delete files into a directory, the size of the directory entry will grow on an Ext4 or similar filesystem. The space is wasted in a sense, but a couple of megabytes is usually not an issue to most.
No, that's true, that's not an issue here either - I was out looking for reason why directory access seems to have slowed down over time, and just happened on this apparent anomaly. -- Per Jessen, Zürich (7.1°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, Oct 18, 2015 at 2:16 AM, Per Jessen <per@computer.org> wrote:
No, that's true, that's not an issue here either - I was out looking for reason why directory access seems to have slowed down over time, and just happened on this apparent anomaly.
The biggest issue that will slow down directory access is having a lot of files in a directory. If the directory is empty, the extra allocated blocks/size won't really impact performance. Brandon Vincent -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
18.10.2015 12:20, Brandon Vincent пишет:
On Sun, Oct 18, 2015 at 2:16 AM, Per Jessen <per@computer.org> wrote:
No, that's true, that's not an issue here either - I was out looking for reason why directory access seems to have slowed down over time, and just happened on this apparent anomaly.
The biggest issue that will slow down directory access is having a lot of files in a directory. If the directory is empty, the extra allocated blocks/size won't really impact performance.
Never say never. Full directory scan may quite likely result in reading of full directory size to check every directory entry. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkov wrote:
18.10.2015 12:20, Brandon Vincent пишет:
On Sun, Oct 18, 2015 at 2:16 AM, Per Jessen <per@computer.org> wrote:
No, that's true, that's not an issue here either - I was out looking for reason why directory access seems to have slowed down over time, and just happened on this apparent anomaly.
The biggest issue that will slow down directory access is having a lot of files in a directory. If the directory is empty, the extra allocated blocks/size won't really impact performance.
Never say never. Full directory scan may quite likely result in reading of full directory size to check every directory entry.
The situation I have is as follows: I have a 600Gb filesystem mounted on /var - key directories under /var are: /var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails). /var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files. I am in the process of migrating some of the <elsewhere> directories to other systems using rsync. This typically causes a major slowdown, which results in large postfix queues building, ending up in processing delays of up to 24hours, sometimes more. When the system is not slowed down, it will easily put away 100.000 new emails per hour. I have been looking at adjusting vfs_cache_pressure, but I haven't found any good/better values. -- Per Jessen, Zürich (6.9°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, 18 Oct 2015 12:43, Per Jessen <per@...> wrote:
Andrei Borzenkov wrote:
18.10.2015 12:20, Brandon Vincent пишет:
On Sun, Oct 18, 2015 at 2:16 AM, Per Jessen <per@computer.org> wrote: .... The situation I have is as follows:
I have a 600Gb filesystem mounted on /var - key directories under /var are:
/var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails).
/var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files.
I am in the process of migrating some of the <elsewhere> directories to other systems using rsync. This typically causes a major slowdown, which results in large postfix queues building, ending up in processing delays of up to 24hours, sometimes more.
When the system is not slowed down, it will easily put away 100.000 new emails per hour. I have been looking at adjusting vfs_cache_pressure, but I haven't found any good/better values.
@Per, have you looked at "ionice", esp running rsync with ionice? That should reduce the impact that running rsync has on postfix queues. Short: "ionice -c3 <rysnc-cmd>" for idle-class or "ionice -c2 -n7 <rysnc-cmd>" for best-effort-class with lowest priority. Maybe you will have to set io-priority of postfix manually to get good or at least better results. "ionice -c2 -n1 -p <postfix-process-ids-septerated-by-space>" to give the postfix processes best-effort-class with second highest priority. AFAIKT, io-priority defaults to best-effort-class with level 4 priority. Have a nice sunday, - Yamaban.
Yamaban wrote:
On Sun, 18 Oct 2015 12:43, Per Jessen <per@...> wrote:
Andrei Borzenkov wrote:
18.10.2015 12:20, Brandon Vincent пишет:
On Sun, Oct 18, 2015 at 2:16 AM, Per Jessen <per@computer.org> wrote: .... The situation I have is as follows:
I have a 600Gb filesystem mounted on /var - key directories under /var are:
/var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails).
/var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files.
I am in the process of migrating some of the <elsewhere> directories to other systems using rsync. This typically causes a major slowdown, which results in large postfix queues building, ending up in processing delays of up to 24hours, sometimes more.
When the system is not slowed down, it will easily put away 100.000 new emails per hour. I have been looking at adjusting vfs_cache_pressure, but I haven't found any good/better values.
@Per, have you looked at "ionice", esp running rsync with ionice? That should reduce the impact that running rsync has on postfix queues.
Short: "ionice -c3 <rysnc-cmd>" for idle-class or "ionice -c2 -n7 <rysnc-cmd>" for best-effort-class with lowest priority.
Ah, I forgot about that - thanks Yamaban. Will let you know how it works out. It'll have to wait till Friday though, I can't upset production during the week.
Maybe you will have to set io-priority of postfix manually to get good or at least better results.
"ionice -c2 -n1 -p <postfix-process-ids-septerated-by-space>" to give the postfix processes best-effort-class with second highest priority.
I might try that on the virtual agents (that deliver mails). -- Per Jessen, Zürich (8.1°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 18/10/2015 12:43, Per Jessen a écrit :
I am in the process of migrating some of the <elsewhere> directories to other systems using rsync. This typically causes a major slowdown,
nice or ionice on rsync??? jdd -- When will a Label sign her!!? https://www.youtube.com/watch?t=94&v=BeMk3WRh8QI -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
jdd wrote:
Le 18/10/2015 12:43, Per Jessen a écrit :
I am in the process of migrating some of the <elsewhere> directories to other systems using rsync. This typically causes a major slowdown,
nice or ionice on rsync???
The box has plenty of cpu power, but ionice is worth trying. Amazing how one forgets about the most obvious. -- Per Jessen, Zürich (8.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 18/10/2015 16:24, Per Jessen a écrit :
jdd wrote:
Le 18/10/2015 12:43, Per Jessen a écrit :
I am in the process of migrating some of the <elsewhere> directories to other systems using rsync. This typically causes a major slowdown,
nice or ionice on rsync???
The box has plenty of cpu power, but ionice is worth trying. Amazing how one forgets about the most obvious.
the list is there for this use (among others :-) jdd -- When will a Label sign her!!? https://www.youtube.com/watch?t=94&v=BeMk3WRh8QI -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 18/10/2015 12:43, Per Jessen wrote:
The situation I have is as follows:
I have a 600Gb filesystem mounted on /var - key directories under /var are:
/var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails).
/var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files.
It would make sense to test several filesystem types with that workload. Years ago I would have said without hesitation "use reiserfs". Nowdays, I don't know which to suggest. Perhaps XFS. But your system is in production, so you can not do such changes... -- Saludos/Cheers, Carlos E.R. (Minas-Morgul - W10) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/18/2015 07:31 AM, Carlos E. R. wrote:
On 18/10/2015 12:43, Per Jessen wrote:
The situation I have is as follows:
I have a 600Gb filesystem mounted on /var - key directories under /var are:
/var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails).
/var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files.
It would make sense to test several filesystem types with that workload. Years ago I would have said without hesitation "use reiserfs". Nowdays, I don't know which to suggest. Perhaps XFS.
But your system is in production, so you can not do such changes...
I would have tried reiserfs too. Reiserfs did "tail packing", which really helped when you had large numbers of files smaller than your disk block size. I think BTRFS packs tails too, right? Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-10-18 17:46, Lew Wolfgang wrote:
I would have tried reiserfs too. Reiserfs did "tail packing", which really helped when you had large numbers of files smaller than your disk block size. I think BTRFS packs tails too, right?
I tested btrfs with many thousands of small files in the same directory, and it crashed the kernel, besides becoming slower every minute as it became fuller. I have not repeated the test with a recent kernel: maybe it has improved, dunno. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlYjwoQACgkQja8UbcUWM1zs4AD+KZybhxJMjovsgjEB8bMDMbdg T67m/KjogSR1iGJKQgEBAJ8R9VQYnG0E/FXzQfxgUtl8GVaFnt3AsXYrB8kX5Or/ =KsI9 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
On 18/10/2015 12:43, Per Jessen wrote:
The situation I have is as follows:
I have a 600Gb filesystem mounted on /var - key directories under /var are:
/var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails).
/var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files.
It would make sense to test several filesystem types with that workload. Years ago I would have said without hesitation "use reiserfs". Nowdays, I don't know which to suggest. Perhaps XFS.
But your system is in production, so you can not do such changes...
Quite so. Besides, JFS has served me well for this purpose for over ten years, surviving all kinds of accidents and abuse. This issue with slowdowns is fairly recent, maybe less then 6 months old. -- Per Jessen, Zürich (7.9°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, Oct 18, 2015 at 10:31 AM, Carlos E. R. <robin.listas@telefonica.net> wrote:
On 18/10/2015 12:43, Per Jessen wrote:
The situation I have is as follows:
I have a 600Gb filesystem mounted on /var - key directories under /var are:
/var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails).
/var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files.
It would make sense to test several filesystem types with that workload. Years ago I would have said without hesitation "use reiserfs". Nowdays, I don't know which to suggest. Perhaps XFS.
But your system is in production, so you can not do such changes...
XFS implemented a very cool journal sort algorithm a few years ago. I guess we all know the kernel does an elevator sort on i/o when it can thus if it needs to write data to sectors 100. 9900.200.9800.300.9700, it will sort the actual writes to the drive to be 100,200,300,9700,9800,9900. That can also happen at the drive level. Doing so can significantly cut down on seek times, especially if some of the writes are to the exact same track. For whatever reason processing the XFS Journal was a very slow process and the lower level elevator sorts weren't helping much. in the 2005-2010 timeframe XFS was known to extremely slow on large deletes. Thus deleting the kernel source tree could take a long time relative to other filesystems. The XFS team (led by Dave Chinner?) implemented a process / algorithm in XFS that allowed the filesystem code to elevator sort the journal before pushing it to the hard drive for the final write.I believe they also combine updates to inodes that are adjacent to each other into a single bigger i/o. Disk drives often work better with larger i/o's. The data shows on more or less single threaded apps it now runs at ext4 types of speeds, aybe a little slower. Pretty good, but not necessarily anything to write home about. On high thread count apps such as a busy mail server like Per is describing, as of 3 years ago XFS is supposed to be much faster than ext4, etc. This takes an hour to listen to and is 3 years old, but is well worth it if you want to see XFS compared to EXT4 and BTRFS. I don't recall JFS being in the benchmarks. https://www.youtube.com/watch?v=FegjLbCnoBw The benchmarks start about 20 minutes in if that is all you want to see. Greg -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
18.10.2015 13:43, Per Jessen пишет:
Andrei Borzenkov wrote:
18.10.2015 12:20, Brandon Vincent пишет:
On Sun, Oct 18, 2015 at 2:16 AM, Per Jessen <per@computer.org> wrote:
No, that's true, that's not an issue here either - I was out looking for reason why directory access seems to have slowed down over time, and just happened on this apparent anomaly.
The biggest issue that will slow down directory access is having a lot of files in a directory. If the directory is empty, the extra allocated blocks/size won't really impact performance.
Never say never. Full directory scan may quite likely result in reading of full directory size to check every directory entry.
The situation I have is as follows:
I have a 600Gb filesystem mounted on /var - key directories under /var are:
/var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails).
/var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files.
I am in the process of migrating some of the <elsewhere> directories to other systems using rsync. This typically causes a major slowdown, which results in large postfix queues building, ending up in processing delays of up to 24hours, sometimes more.
Slow down with new directory after migration or slow down during migration?
When the system is not slowed down, it will easily put away 100.000 new emails per hour. I have been looking at adjusting vfs_cache_pressure, but I haven't found any good/better values.
-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Andrei Borzenkov wrote:
18.10.2015 13:43, Per Jessen пишет:
Andrei Borzenkov wrote:
18.10.2015 12:20, Brandon Vincent пишет:
On Sun, Oct 18, 2015 at 2:16 AM, Per Jessen <per@computer.org> wrote:
No, that's true, that's not an issue here either - I was out looking for reason why directory access seems to have slowed down over time, and just happened on this apparent anomaly.
The biggest issue that will slow down directory access is having a lot of files in a directory. If the directory is empty, the extra allocated blocks/size won't really impact performance.
Never say never. Full directory scan may quite likely result in reading of full directory size to check every directory entry.
The situation I have is as follows:
I have a 600Gb filesystem mounted on /var - key directories under /var are:
/var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails).
/var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files.
I am in the process of migrating some of the <elsewhere> directories to other systems using rsync. This typically causes a major slowdown, which results in large postfix queues building, ending up in processing delays of up to 24hours, sometimes more.
Slow down with new directory after migration or slow down during migration?
During migration/rsync. The new directory is on a different machine. -- Per Jessen, Zürich (7.9°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-10-18 18:34, Per Jessen wrote:
Slow down with new directory after migration or slow down during migration?
During migration/rsync. The new directory is on a different machine.
What about kernel cache exhaustion syndrome? (my naming) - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlYj80YACgkQja8UbcUWM1wX/wEAlyHfinBVXA3kb2R5nRMc/3CG o8g6JdDHODT9c1TtbiIBAKBqwgNynEUwKtwJ/IjqdwsZMBMGQ7IhBOKkjhn4GI+k =oFAC -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256
On 2015-10-18 18:34, Per Jessen wrote:
Slow down with new directory after migration or slow down during migration?
During migration/rsync. The new directory is on a different machine.
What about kernel cache exhaustion syndrome? (my naming)
Yeah, something like that - I seem to remember having such an issue before, but it's 7-8 years ago on a different machine. This one has 10Gb RAM, of which about half is used for filesystem cache. I've been looking a dentry in /proc/slabinfo, but I don't really know what to do with that number. -- Per Jessen, Zürich (6.4°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per Jessen wrote:
No, that's true, that's not an issue here either - I was out looking for reason why directory access seems to have slowed down over time, and just happened on this apparent anomaly.
I don't suppose you had a look at the fragmentation level as I last mentioned? < Does JFS have any utilities to tell you how fragmented the
files, directories and free space is? ... 13% is getting close to space exhaustion -- 10-15% is usually the lower limit, though in practice I find it's best to keep drives below 75% usage for continued fast performance (not that I always do that, mind you, but when possible) -- if it is a mostly "read-only" drive, then lower amounts of free space are usually fine...
Older file systems usually limited the amount of user-usable space to 85-90% of capacity, because various algorithms -- like finding free space, or trying to map space contiguously, slow down when you get below 10-15% of space.
I have a 600Gb filesystem mounted on /var - key directories under /var are:
/var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails).
/var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files.
So you are creating/destroying what? 10k files/day or 100k files/day? what? If you've been running that file system @ 85-90% full for some time and lots of file creates and destroys, most old file systems aren't good about consolidating space (like not shrinking directories that would free up 3mg of space). Few file systems have any sort of defragment utility, and for what we are talking about here, (defragmented free space), I don't know of any linux file systems that have such -- but most could be helped by dumping the disk contents to another disk, then, since some file systems aren't good about cleaning up and consilidating free space -- (leaving random bits of allocated space laying around...), it might be most prudent to re-mkfs your file system. rsync isn't the best tool to copy off files -- you might try 'cp -au', for example, -- will only copy files with newer times (unless your target machine is far away over a slow link). If you really need to slow rsync down, besides doing both: 1) nice -19 ionice -c3, you could limit rsync to only run on 1 of the cpu's (normally since it has a client and sender they'll more often than not fall onto separate cores. So: 2) sudo nice -19 ionice -c3 taskset -c1 rsync.... 3) There's also the "--bwlimit" switch to rsync, but I doubt that bw is your problem, -- likely rsync is having to alot of seeking and comparing of metadata. Beyond that .. it gets hairier to control things. doable, but hairier. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 18/10/2015 21:48, Linda Walsh a écrit :
for what we are talking about here, (defragmented free space),
small files do not fragment. Some file system (reiser, ntfs, others...) do not even use disk space but only store data in inodes jdd -- When will a Label sign her!!? https://www.youtube.com/watch?t=94&v=BeMk3WRh8QI -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-10-18 23:19, jdd wrote:
Le 18/10/2015 21:48, Linda Walsh a écrit :
for what we are talking about here, (defragmented free space),
small files do not fragment. Some file system (reiser, ntfs, others...) do not even use disk space but only store data in inodes
She refers to the structure used to list or collect free space. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlYkD2sACgkQja8UbcUWM1yWVgEAoIazq7dDs6sRCCiTT7A+tBYH eEv3Ms7tIBhXyolMS+kBAI5x34z4gyEBKwwiL67/BFqlugIySxnGb3rpQfNn3WPX =VcVa -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
jdd wrote:
Le 18/10/2015 21:48, Linda Walsh a écrit :
for what we are talking about here, (defragmented free space),
small files do not fragment. Some file system (reiser, ntfs, others...) do not even use disk space but only store data in inodes
NTFS doesn't store files in inodes! NTFS doesn't even have inodes! It stores the names and meta-information in 1-4 gigantic, mostly unmovable "MFT" areas (Master File Table). In that NTFS stores all the meta-information about the DATA associated with a filename (including *pointers* to one or more data streams that are generally allocated in 4KB-clusters. To see the fragmentation, go get "DiskView" from "https://technet.microsoft.com/en-us/sysinternals" (it's under the File and Disk Utilities). As for file systems that have no data storage other than mixed in with the file metadata?... Can you list 1 example. AFAIK Reiserfs only did this for *small files*. XFS puts some data in small inodes as well -- including small directories. But most file systems allocate data out of some "free storage area". And that free storage can get fragmented over time with any file system. FWIW, My 'C-Drive(NTFS/Win7)' has 1,512,823 files in 2,394,681 fragments with 38.45% free space left. According to windows's disk defragmenter, the current files are 1% fragmented. (Guess that means ~15,000 files have unnecessary fragmentation). But you can zoom in to the block level and find exactly where a file is, and if it is split into more than 1 fragment. Don't know where you get your information, but you might want to re-examine some of the sources, as they are provably inaccurate. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Le 20/10/2015 05:05, Linda Walsh a écrit :
than mixed in with the file metadata?... Can you list 1 example. AFAIK Reiserfs only did this for *small files*.
yes, it's what I tied to say. this is the OP problem: many small files (mails) jdd -- When will a Label sign her!!? https://www.youtube.com/watch?t=94&v=BeMk3WRh8QI -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-10-20 05:05, Linda Walsh wrote:
jdd wrote:
Le 18/10/2015 21:48, Linda Walsh a écrit :
small files do not fragment. Some file system (reiser, ntfs, others...) do not even use disk space but only store data in inodes
As for file systems that have no data storage other than mixed in with the file metadata?... Can you list 1 example. AFAIK Reiserfs only did this for *small files*.
Notice the word "small" at the start of the quote ;-) - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlYmP2IACgkQja8UbcUWM1zjHgEAjevnyHFfg+MX8sf7rH8pbbo9 veTWL43iBcCo9sZ/lY0A/0OcFuXAzLzVbGa+8nVzcAoJ4+zkqxQ5mMXIV1udnVzy =J5dG -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/19/2015 11:05 PM, Linda Walsh wrote:
Don't know where you get your information, but you might want to re-examine some of the sources, as they are provably inaccurate.
What I wonder about ... True b-tree systems such as ReiserFS follow a b-tree allocation *AND REALLOCATION* policy. Yes file systems that use a model where the directories are just a specially marked file (harks back to the old V6/V7 FS! WOW KISS) are by their very nature going to have problems such as gaps for deleted files and quite probably linear searches. Perhaps that's why Postfix and others use a slightly hierarchical "indexing" scheme. Many email systems are high throughput and this seems to be one tactic Postfix uses. If the directories are part of a b-tree then a whole host of problems about allocation and searching go away. Hmm. Other problems arise, but that's another matter. <quote src='http://cs.uns.edu.ar/~jechaiz/sosd/clases/slides/07-FileSystemsExtra4_Reiser.pdf"> "The simplest method of implementing a directory is to use a linear list of file names with pointers to the data blocks ... The real disadvantage of a linear list of directory entries is the linear search to find a file. ... Directory information is used frequently, and a slow implementation of access to it would be noticed by users. </quote> There's more, but its worth reading the slide deck. I don't know about the other B-tree file systems. Ext3 did store directories in a linear table but there was also the 'dir_index' option. I presume this is available for ext4 as well. O don't know if its purely indexing or how it affects creation/deletion if the directory structure underneath is still linear. I do now that there are problems with dynamic rebalancing of b-trees when they exist on dis; in short there can be delays when they get re-written due to balancing. ReiserFS apparently solved some of these, but I see the occasional delay on my BtrFS ones. I don't use XFS so can't comment. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Linda Walsh wrote:
Per Jessen wrote:
No, that's true, that's not an issue here either - I was out looking for reason why directory access seems to have slowed down over time, and just happened on this apparent anomaly.
I don't suppose you had a look at the fragmentation level as I last mentioned?
No, afaik there are no such utilities for JFS.
I have a 600Gb filesystem mounted on /var - key directories under /var are:
/var/spool/postfix-in/{incoming,active} with 2 levels of hashing. On a busy day or after delays, each subdir might easily reach 10.000 files. 99% small files of less than 100Kb. (emails).
/var/spool/elsewhere/dir{0,1,2,3...1000}/maildirs - each such maildir might have 100.000 files.
So you are creating/destroying what? 10k files/day or 100k files/day? what?
It varies quite a bit, but at least 200.000 new written, and the same deleted. Last week we had a couple of days with 800-900.000 a day.
If you've been running that file system @ 85-90% full for some time and lots of file creates and destroys, most old file systems aren't good about consolidating space (like not shrinking directories that would free up 3mg of space).
It's probably running at 85% full for a year or more. As I think I said, it's essentially a fifo buffer for lots of small files. The "working set" is about 4 days worth of traffic, but the filesystem has lots of other stuff. (mysql for instance).
but most could be helped by dumping the disk contents to another disk, then, since some file systems aren't good about cleaning up and consilidating free space -- (leaving random bits of allocated space laying around...), it might be most prudent to re-mkfs your file system.
Yes that is an interesting idea.
rsync isn't the best tool to copy off files -- you might try 'cp -au', for example, -- will only copy files with newer times (unless your target machine is far away over a slow link).
The target is about 40cm away over GigE :-)
If you really need to slow rsync down, besides doing both: 1) nice -19 ionice -c3, you could limit rsync to only run on 1 of the cpu's (normally since it has a client and sender they'll more often than not fall onto separate cores.
So: 2) sudo nice -19 ionice -c3 taskset -c1 rsync....
3) There's also the "--bwlimit" switch to rsync, but I doubt that bw is your problem, -- likely rsync is having to alot of seeking and comparing of metadata.
Yeah, I don't see bandwidth being the issue either. I'm also not sure if rsync is really the culprit here, but it's worth a try to see what it does. Thanks for all your input! /Per -- Per Jessen, Zürich (6.5°C) http://www.dns24.ch/ - free dynamic DNS, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 On 2015-10-19 10:02, Per Jessen wrote:
Yeah, I don't see bandwidth being the issue either. I'm also not sure if rsync is really the culprit here, but it's worth a try to see what it does. Thanks for all your input!
rsync has a phase during which it needs to compare, at least, all files stamps and attributes, and find out what is new or changed and needs to be copied. It is a similar load to a recursive "find"; heavier, I think. As your directories are large, it is a lot of head seeking, probably. An SSD would be faster, but would probably die under that load. And depending on your switches or the circumstances, rsync also compares checksums. An improvement can be to use the rsync daemon on whichever machine is "remote". I believe it runs faster. Another would be using a mirror (raid), and isolating one side to image it, for a limited time. - -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" (Minas Tirith)) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlYk+hYACgkQja8UbcUWM1xDMAD+N8zioQwN7buICnlduomk81tk 7glTFjHrZoWi4/Ddu/0BAJVv6VfjD6MlrBumxZSHZZVC8NN6EPOhTWfYzYA/xhfW =UXVr -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
18.10.2015 12:06, Per Jessen пишет:
Brandon Vincent wrote:
On Sun, Oct 18, 2015 at 1:47 AM, Per Jessen <per@computer.org> wrote:
Thanks Andrei, Brandon - does that directory then actually take up that amount of physical space? This directory (and many others) are essential fifo queues where many new files are written and deleted daily.
The directory is indeed taking up that space. The reason for leaving the directory entries intact although the files have been deleted is to reduce filesystem fragmentation.
So as long as the directory entries are being reused, not much space is "wasted"?
That's correct, directory slots that belonged to deleted entries will be reused. There could be issues with fragmentation if file name lengths vary significantly, but if all file names are of the same length it should not grow. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/18/2015 05:20 AM, Andrei Borzenkov wrote:
So as long as the directory entries are being reused, not much space is "wasted"?
That's correct, directory slots that belonged to deleted entries will be reused. There could be issues with fragmentation if file name lengths vary significantly, but if all file names are of the same length it should not grow.
I don't know about Linux file systems, but the OS/2 HPFS would minimize fragmentation by finding the smallest free block that would hold the file. As files were removed, the space was added to adjacent free blocks. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 10/18/2015 04:36 AM, Andrei Borzenkov wrote:
Well, size of *directory* is 3.8MB indeed. When files are deleted directories do not shrink (at least in those filesystems I am ware of) so if you once had millions of files there and then deleted them directory size remains big.
So, if you had a directory with files that use over half the partition space and moved those to a new directory, then du for the 2 directories would total more than the partition size? BTW, I remember the days of exceeding my quota on a Novell Netware server, because I had so many files. The admin suggested I delete some but I told them I couldn't as they were company records with important information that had to be maintained for years. He upped my quota on a couple of occasions. ;-) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Sun, Oct 18, 2015 at 5:21 AM, James Knott <james.knott@rogers.com> wrote:
So, if you had a directory with files that use over half the partition space and moved those to a new directory, then du for the 2 directories would total more than the partition size?
Not quite. The size of the directory being reported is referring to the size of the directory entry (the filename to inode mapping) and not the cumulative total of the files under said directory. The directory size is never going to get over a couple of megabytes. Brandon Vincent -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (11)
-
Andrei Borzenkov -
Anton Aylward -
Brandon Vincent -
Carlos E. R. -
Greg Freemyer -
James Knott -
jdd -
Lew Wolfgang -
Linda Walsh -
Per Jessen -
Yamaban