[opensuse] Re: faster way to get total dir size besides du?
Jim Henderson wrote:
On Tue, 06 Mar 2012 04:38:21 +0100, Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2012-03-06 04:07, David Haller wrote:
- AFAIK no filesystem I know of stores the sizes of a directory's contents somewhere in the directory's metadata
FAT does, kind of. The directory is a record that contains file names, attributes, sizes, and starting record. You just have to load the directory record and sum the sizes of all files: one single read one disc record operation (I don't remember how many files per record). If the directory is big, then there have to be more reads. The operation is very fast. If there are subdirectories, then it is slower (recursive calls).
That's different than storing the size of the contents in the directory entry's metadata - the summing of each subordinate file is what takes the time. What David is trying to say is that no filesystem that he knows of (nor that I know of for that matter) stores the total of the subordinate objects.
ie, for a structure of:
/ /usr /usr/file1 /usr/file2 /usr/file3
/usr doesn't store the sum of the sizes of file1, file2, and file3. It has to be calculated on the fly.
But that does neither contradict Carlos' point nor does it address his argument. /usr might not store the sum of the sizes, but each size for each file. To repeat: -- Both for FAT and for Linux filesystems, you have to read the directory contents. -- Only for Linux filesystems, you have to stat every file. -- Both for FAT and for Linux filesystems, you have to sum the sizes. Summing sizes is fast. Carlos' argument is, by not needing to stat every file, disk usage determination on FAT is much faster, because stat(2) is an expensive operation when it has to be done on many files. Actually, I don't know if stat(2) is really the culprit for slow du; but it's a good bet: I'm quite sure that summing some numbers ain't the issue. Computers are good in adding numbers, much better than in accessing disks... Mind you, I'm not telling that this behaviour would convince me to use FAT beyond an USB stick. ;-) I just want to contribute what I see as a misunderstanding. ;-) Joachim -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Joachim Schrod Email: jschrod@acm.org Roedermark, Germany -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, 06 Mar 2012 18:27:44 +0100, Joachim Schrod wrote:
But that does neither contradict Carlos' point nor does it address his argument. /usr might not store the sum of the sizes, but each size for each file.
OIC, I had read Carlos' comment as stating that it's easy and fast to do that enumeration, because you just have to sum the size of the files - I wasn't aware that files needed to have stat() called against them on some filesystems in order to get the size. Jim -- Jim Henderson Please keep on-topic replies on the list so everyone benefits -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-03-06 18:27, Joachim Schrod wrote:
Jim Henderson wrote:
On Tue, 06 Mar 2012 04:38:21 +0100, Carlos E. R. wrote:
Summing sizes is fast. Carlos' argument is, by not needing to stat every file, disk usage determination on FAT is much faster, because stat(2) is an expensive operation when it has to be done on many files.
That's my meaning :-)
Mind you, I'm not telling that this behaviour would convince me to use FAT beyond an USB stick. ;-) I just want to contribute what I see as a misunderstanding. ;-)
Absolutely. Each filesystem has its design goals and compromises. And FAT has currently some not very nice patches, like the addition for long names. - -- Cheers / Saludos, Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAk9WgxYACgkQIvFNjefEBxovvwCdG8FHAm22l9/fLTNrJdLiMwUA YVsAn3b0Jp0gDiCNjqIuYaBtmxx6QWQY =ddJc -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, 06 Mar 2012 22:35:18 +0100, "Carlos E. R." <robin.listas@telefonica.net> wrote:
And FAT has currently some not very nice patches, like the addition for long names.
FAT is a dinosaur nearly from the CP/M days and should have died long ago! But it is very easy to implement and thus will live on and on on consumer devices like cameras, mp3 players, thumb drives and other flash memory. And when you thought that finally FAT would die because the memory device exceeds the FAT limits, MS comes along with ExFAT and you're bound to live with it for a couple of decades more ... Philipp -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-03-08 23:40, Philipp Thomas wrote:
On Tue, 06 Mar 2012 22:35:18 +0100, "Carlos E. R." <> wrote:
And FAT has currently some not very nice patches, like the addition for long names.
FAT is a dinosaur nearly from the CP/M days and should have died long ago!
Being old is not a reason to not being useful. On the contrary, it says good things. If it were that bad it would have disapeared long ago.
the memory device exceeds the FAT limits, MS comes along with ExFAT and you're bound to live with it for a couple of decades more ...
No, ExFAT is patented technology, you will not see it that often. And windows 8 comes with another different filesystem I don't even know the name of. Ah, yes, "Protogon", says the wikipedia. [...] reminds me of reiserfs... <http://en.wikipedia.org/wiki/ReFS#ReFS> - -- Cheers / Saludos, Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAk9ZOm8ACgkQIvFNjefEBxoCJgCfckraTsa0C3Usd1+u7PI1CuKs F0QAoMI3UziS6quIzvmk8Xzg4x2/3Cqr =3Ynw -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Fri, 09 Mar 2012 00:02:07 +0100, "Carlos E. R." <robin.listas@telefonica.net> wrote:
No, ExFAT is patented technology,
MS also has some patents regarding FAT. And no, it doesn't have any merit besides being easy to implement. What do you expect from a file system that was part of Quick and Dirty Operating System and then renamed to DOS while reinterpreting the 'D'?
you will not see it that often.
You will, as any SDXD card carries it and the number of devices that support CF is rapidly shrinking. Philipp -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-03-09 20:38, Philipp Thomas wrote:
On Fri, 09 Mar 2012 00:02:07 +0100, "Carlos E. R." <robin.listas@telefonica.net> wrote:
No, ExFAT is patented technology,
MS also has some patents regarding FAT.
The patents protecting exfat are stronger. They learnt from their previous mistakes. It will avoid SUSE from distributing it.
And no, it doesn't have any merit besides being easy to implement. What do you expect from a file system that was part of Quick and Dirty Operating System and then renamed to DOS while reinterpreting the 'D'?
It has the merit that is very easy to code, with very small cpu requirements, and is already supported universally. If I recall correctly, the new standard bios booting system, uefi, requires a FAT partition on the boot disk on which the boot code is placed, so you are going to see FAT for many years ahead. >:-P
you will not see it that often.
You will, as any SDXD card carries it and the number of devices that support CF is rapidly shrinking.
I really hope not. - -- Cheers / Saludos, Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAk9aYbQACgkQIvFNjefEBxrY2wCgsu0FoRT4ft2UPIurWmj4+8ya EEQAn2BCdtFHBhPJPg3cEaZftMOpVxJR =TY5J -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Friday 09 March 2012 21:01:56 Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2012-03-09 20:38, Philipp Thomas wrote:
On Fri, 09 Mar 2012 00:02:07 +0100, "Carlos E. R."
<robin.listas@telefonica.net> wrote:
No, ExFAT is patented technology,
MS also has some patents regarding FAT.
The patents protecting exfat are stronger. They learnt from their previous mistakes. It will avoid SUSE from distributing it.
The two patents mentioned in the wikipedia article on exfat are file name lookups using a hash - a method for quickly determining if a file name exists by searching for a hash of the name Storing files contiguously on disk Seriously. Storing files in one chunk is patented. if there were such a thing as a frivolous patent claim, as there are frivolous law suits, Microsoft would be up to their ears in fines. There is no way on earth that those patents would stand up in court. Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 2012-03-09 21:17, Anders Johansson wrote:
if there were such a thing as a frivolous patent claim, as there are frivolous law suits, Microsoft would be up to their ears in fines. There is no way on earth that those patents would stand up in court.
I doubt that they are that dumb, and I doubt that a company such as SUSE will risk it, till somebody else does. I hope that you are right, but I doubt it. - -- Cheers / Saludos, Carlos E. R. (from 11.4 x86_64 "Celadon" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/ iEYEARECAAYFAk9aZ4cACgkQIvFNjefEBxpSVwCgre2CVFQZBo0jkgl2gBHsBpyM f9YAnir3VJA97NjvVtjXtccH9usiFxQT =+buo -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Friday 09 March 2012 21:26:47 Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2012-03-09 21:17, Anders Johansson wrote:
if there were such a thing as a frivolous patent claim, as there are frivolous law suits, Microsoft would be up to their ears in fines. There is no way on earth that those patents would stand up in court.
I doubt that they are that dumb, and I doubt that a company such as SUSE will risk it, till somebody else does.
I hope that you are right, but I doubt it.
There is no doubt that those two patents are ludicrous and easily dismissed. I don't know what else they have, I "hope" it's more because if that is all they have they really are that dumb. The risk you mention is another thing. It's one thing to be technically right, it's another to convince 12 good windows users and true that you are right, and in the mean time you have to pay the lawyers. This is the only reason anyone pays any heed to all that silliness Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Friday 09 March 2012 21:17:43 I wrote:
Storing files contiguously on disk
I forgot to mention the most hilarious aspect of this. The full title of the patent is CONTIGUOUS FILE ALLOCATION IN AN EXTENSIBLE FILE SYSTEM They had the nerve to even mention one prior art file system in the title itself. Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 09/03/12 20:27, Anders Johansson wrote:
On Friday 09 March 2012 21:17:43 I wrote:
Storing files contiguously on disk
I forgot to mention the most hilarious aspect of this. The full title of the patent is
CONTIGUOUS FILE ALLOCATION IN AN EXTENSIBLE FILE SYSTEM
They had the nerve to even mention one prior art file system in the title itself.
Anders
Their lawyers would argue that Extensible != Extended See https://en.wikipedia.org/wiki/Ext3 Bob -- Bob Williams System: Linux 3.1.9-1.4-desktop Distro: openSUSE 12.1 (x86_64) with KDE Development Platform: 4.7.2 (4.7.2) "release 5" Uptime: 18:00pm up 3 days 0:29, 3 users, load average: 0.82, 0.45, 0.31 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Friday 09 March 2012 22:48:19 Bob Williams wrote:
On 09/03/12 20:27, Anders Johansson wrote:
On Friday 09 March 2012 21:17:43 I wrote:
Storing files contiguously on disk
I forgot to mention the most hilarious aspect of this. The full title of the patent is
CONTIGUOUS FILE ALLOCATION IN AN EXTENSIBLE FILE SYSTEM
They had the nerve to even mention one prior art file system in the title itself.
Anders
Their lawyers would argue that Extensible != Extended
Well, I would argue that Extensible == Extensible. See http://en.wikipedia.org/wiki/Xfs Yes, the wikipedia article doesn't mention it for whatever reason, but that is actually what XFS stands for Anders -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 09/03/12 22:55, Anders Johansson wrote:
On Friday 09 March 2012 22:48:19 Bob Williams wrote:
On 09/03/12 20:27, Anders Johansson wrote:
On Friday 09 March 2012 21:17:43 I wrote:
Storing files contiguously on disk
I forgot to mention the most hilarious aspect of this. The full title of the patent is
CONTIGUOUS FILE ALLOCATION IN AN EXTENSIBLE FILE SYSTEM
They had the nerve to even mention one prior art file system in the title itself.
Anders
Their lawyers would argue that Extensible != Extended
Well, I would argue that Extensible == Extensible.
See http://en.wikipedia.org/wiki/Xfs
Yes, the wikipedia article doesn't mention it for whatever reason, but that is actually what XFS stands for
Anders
You're right. I'd forgotten about XFS. Bob -- Bob Williams System: Linux 3.1.9-1.4-desktop Distro: openSUSE 12.1 (x86_64) with KDE Development Platform: 4.7.2 (4.7.2) "release 5" Uptime: 18:00pm up 3 days 0:29, 3 users, load average: 0.82, 0.45, 0.31 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (6)
-
Anders Johansson -
Bob Williams -
Carlos E. R. -
Jim Henderson -
Joachim Schrod -
Philipp Thomas