[opensuse] Question (just one) about moving from ReiserFS to JFS
I've been using ReiserFS quite happily for many years now and I think it is one of those software packages that was an excellent/genius design; it has needed very little maintenance over the years. The PowerThatBe, however, think that it is 'unsupported' (actually it isn't, it just doesn't have the crowds that theext4 does since it don't need much work, which to my mind reflects badly on ext4) and so are depreciating it, especially with Leap-15. ReiserFS does have the problem that it is single threaded, so I don't get the best performance when I have so many file system running it :-( I don't see reiser4 being merged in in teh foreseeable future. So I'm looking to migrate to another obscure file system. JFS. I've used JFS extensively for many years (hmm over a decade) on IBM AIX machines. (Where I always used it on top of LVM.) Like ReiserFS and XFS and BtrFS it beaks away from the archaic model that goes back to the pre-networking UNIX V6 days on the PDP-11 where the inode space and the data space blocks were preallocated and fixed at MKFS time. This idiocy is preserved even with ext4FS! All these late model file systems are based on B-tree models so there is no reason why ext4FS should have this constraint. My one question is about concurrency. In some ways it a dumb question. The Linux implementation of JFS was done by IBM engineers so presumably it reflects the way the IBM AIX implementation worked, but I can't find an explicit answer in the extensive AIX documentation I have either. Is JFS multi-threaded? Please: no answers that are "Not invented here" & other prejudices. Particularly Ext4, BtrFS and XFS. "Comparisons are odious". I'm also not interested in building my own kernel https://reiser4.wiki.kernel.org/index.php/Reiser4_Howto And please, this is a desktop, not a file server, so advantages such as dealing with 16Tb files or file systems are of little importance, but the handling of lots of smaller files (such as ReiserFS does very well) *IS* important. That is why I'm asking about multi-threading. That's the ReiserFS limitation for me. ==================== -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
So I'm looking to migrate to another obscure file system. JFS.
My favourite. Except for test systems with ext4, everything here is JFS. Has been since ... forever. JFS v1.1 maybe.
I've used JFS extensively for many years (hmm over a decade) on IBM AIX machines.
The Linux variant of JFS is based on the JFS that was re-developed/-written for OS/2.
My one question is about concurrency. In some ways it a dumb question. The Linux implementation of JFS was done by IBM engineers so presumably it reflects the way the IBM AIX implementation worked, but I can't find an explicit answer in the extensive AIX documentation I have either.
Is JFS multi-threaded?
What does that mean?
And please, this is a desktop, not a file server, so advantages such as dealing with 16Tb files or file systems are of little importance, but the handling of lots of smaller files (such as ReiserFS does very well) *IS* important. That is why I'm asking about multi-threading. That's the ReiserFS limitation for me.
Sounds to me like you just ought to try it out. You're not making a strategic decision for the future, you can always change to something else. JFS is a mature filesystem that just works and works and works. IIRC, Backblaze (storage provider) used to swear by JFS. -- Per Jessen, Zürich (3.4°C) http://www.dns24.ch/ - your free DNS host, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/15/2018 10:27 AM, Per Jessen wrote:
Sounds to me like you just ought to try it out. You're not making a strategic decision for the future, you can always change to something else.
JFS is a mature filesystem that just works and works and works. IIRC, Backblaze (storage provider) used to swear by JFS.
Why would one select JFS before XFS for large RAID data storage? Arrays larger than 50-TB, for example. Regards, Lew -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 15/01/18 18:33, Lew Wolfgang wrote:
On 01/15/2018 10:27 AM, Per Jessen wrote:
Sounds to me like you just ought to try it out. You're not making a strategic decision for the future, you can always change to something else.
JFS is a mature filesystem that just works and works and works. IIRC, Backblaze (storage provider) used to swear by JFS.
Why would one select JFS before XFS for large RAID data storage? Arrays larger than 50-TB, for example.
Because that's not the problem. ReiserFS is *very* good at dealing with lots of small files - something which *most* filesystems have problems. If you have a 50TB filesystem FULL of files with an average size of, say, 4K, what will that do to XFS performance? Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2018-01-15 20:22, Wol's lists wrote:
On 15/01/18 18:33, Lew Wolfgang wrote:
On 01/15/2018 10:27 AM, Per Jessen wrote:
Sounds to me like you just ought to try it out. You're not making a strategic decision for the future, you can always change to something else.
JFS is a mature filesystem that just works and works and works. IIRC, Backblaze (storage provider) used to swear by JFS.
Why would one select JFS before XFS for large RAID data storage? Arrays larger than 50-TB, for example.
Because that's not the problem.
ReiserFS is *very* good at dealing with lots of small files - something which *most* filesystems have problems.
If you have a 50TB filesystem FULL of files with an average size of, say, 4K, what will that do to XFS performance?
XFS can cope with that, but worse than Reiserfs. Which makes me wonder: How difficult is for a user to use R4? -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" (Minas Tirith))
On 15/01/18 02:28 PM, Carlos E. R. wrote:
Which makes me wonder: How difficult is for a user to use R4?
I looked into that. The patches for the kernel are up to date, you just need to get the source tree and compile your own kernel - and keep that up to date. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2018-01-16 03:25, Anton Aylward wrote:
On 15/01/18 02:28 PM, Carlos E. R. wrote:
Which makes me wonder: How difficult is for a user to use R4?
I looked into that. The patches for the kernel are up to date, you just need to get the source tree and compile your own kernel - and keep that up to date.
That's a difficulty, because I have /usr/src/ on a reiserfs partition. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On 16/01/18 04:58 AM, Carlos E. R. wrote:
That's a difficulty, because I have /usr/src/ on a reiserfs partition.
Well you'll just have to convert that to JFS then, won't you :-) -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2018-01-16 15:41, Anton Aylward wrote:
On 16/01/18 04:58 AM, Carlos E. R. wrote:
That's a difficulty, because I have /usr/src/ on a reiserfs partition.
Well you'll just have to convert that to JFS then, won't you :-)
I don't plan on converting >;-) -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" (Minas Tirith))
On 15/01/18 02:22 PM, Wol's lists wrote:
On 15/01/18 18:33, Lew Wolfgang wrote:
On 01/15/2018 10:27 AM, Per Jessen wrote:
Sounds to me like you just ought to try it out. You're not making a strategic decision for the future, you can always change to something else.
JFS is a mature filesystem that just works and works and works. IIRC, Backblaze (storage provider) used to swear by JFS.
Why would one select JFS before XFS for large RAID data storage? Arrays larger than 50-TB, for example.
Because that's not the problem.
ReiserFS is *very* good at dealing with lots of small files - something which *most* filesystems have problems.
If you have a 50TB filesystem FULL of files with an average size of, say, 4K, what will that do to XFS performance?
I looked at the distribution of file sizes for my /etc (which is on a ext4 'cos I can figure out the constraints there beforehand). As you can see, the bulk of the file will fit in one sector and a significant number can be packed or tail packed File Size # of files 0 9 1 1 4 2 8 8 16 28 32 57 64 119 128 200 256 231 512 248 1024 258 2048 235 4096 123 8192 144 16384 49 32768 85 65536 10 131072 6 262144 2 524288 4 1048576 1 2097152 1 4194304 1 The numbers for my ~anton and my ~anton/Documents partitions are even more dramatic. Packing (sharing a block) and inode packing are really going to be effective. Why is this relevant? ===================== Service providers such as the ones I use, Dreamhost, implement share account and virtual machines on systems where users are going to have a lot of files like this. The machines hosting the databases for the web services will be separate. Does JFS do any form or 'packing'? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/15/2018 01:27 PM, Per Jessen wrote:
The Linux variant of JFS is based on the JFS that was re-developed/-written for OS/2.
The version of JFS on Linux is JFS2, which appeared on OS/2 before AIX. In the SCO owns Linux nonsense lawsuit, SCO tried to claim they owned the JFS in Linux, because it was installed on AIX, IBM's Unix, and moved illegally to Linux, ignoring the fact that it was on OS/2 first. I recall when JFS became available for OS/2, as I was working at IBM Canada, providing 3rd level OS/2 support at the time. BTW, I still have a bunch of Linux CDs, published by IBM back then. Caldera was one of the included distros. I was on an internal mailing list for various CDs. In addition to Linux, I received many others for OS/2, Windows, DOS, AIX, mainframe and DB2, probably well over a hundred CDs in total. They pretty much fill a fairly large box. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Op maandag 15 januari 2018 20:05:36 CET schreef James Knott:
On 01/15/2018 01:27 PM, Per Jessen wrote:
The Linux variant of JFS is based on the JFS that was re-developed/-written for OS/2.
The version of JFS on Linux is JFS2, which appeared on OS/2 before AIX. In the SCO owns Linux nonsense lawsuit, SCO tried to claim they owned the JFS in Linux, because it was installed on AIX, IBM's Unix, and moved illegally to Linux, ignoring the fact that it was on OS/2 first. I recall when JFS became available for OS/2, as I was working at IBM Canada, providing 3rd level OS/2 support at the time.
BTW, I still have a bunch of Linux CDs, published by IBM back then. Caldera was one of the included distros. I was on an internal mailing list for various CDs. In addition to Linux, I received many others for OS/2, Windows, DOS, AIX, mainframe and DB2, probably well over a hundred CDs in total. They pretty much fill a fairly large box.
Djeez, I never thought I'd see the name Caldera again. Don't remember why, but my choice between S.u.S.E. and Caldera was won by S.u.S.E. . Caldera was awesome though. -- Gertjan Lettink, a.k.a. Knurpht openSUSE Board Member openSUSE Forums Team -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 15/01/18 01:27 PM, Per Jessen wrote:
Anton Aylward wrote:
So I'm looking to migrate to another obscure file system. JFS.
My favourite. Except for test systems with ext4, everything here is JFS. Has been since ... forever. JFS v1.1 maybe.
I've used JFS extensively for many years (hmm over a decade) on IBM AIX machines.
The Linux variant of JFS is based on the JFS that was re-developed/-written for OS/2.
As I understand it, that code base was then fed back into the development of later model JFS for AIX and the Linux version is derived from that, concurrently with the Linux-for-AIX version.
My one question is about concurrency. [...] Is JFS multi-threaded?
What does that mean?
Well, by comparison, ReiserFS isn't. So if I have multiple ReiserFS partitions and am running, say, Thunderbird storing on one, and it wakes up occasion ad gets mail and stores it there, and I'm watching a movie that is on another partition that is also ReiserFS, while a compile of a tree is going on in the background and the source and destination are each on another partition and the binaries and libraries are on a third partition and they are all ReiserFS, then there's only one thread and they are blocking each other and overall performance sucks.
JFS is a mature filesystem that just works and works and works. IIRC, Backblaze (storage provider) used to swear by JFS.
I'm converting my "movies" and a few others and we'll see how it goes. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 15/01/18 01:27 PM, Per Jessen wrote:
Anton Aylward wrote:
So I'm looking to migrate to another obscure file system. JFS.
My favourite. Except for test systems with ext4, everything here is JFS. Has been since ... forever. JFS v1.1 maybe.
I have a serious problem and it seems related to JFS I have converted a couple of partitions to JFS, mostly large files, PDF document movies/presentations. They were mounted bt not being accessed. I have firefox running and encounter a site that doesn't want to work with FF, needs chromium. Again, a presentation. What I normally do in these circumstances is copy the URL, start chromium in other window inder KDE (I keep a couple space just for this purpose) and view the rpesentation there. I've done this many times. Today I try it and my machine freezes. The disk light comes on and stays on. Mouse is unresponsive. I hot key ctl-altF1 and log in as root. Machine is unresponsive and I get "login timeout after 60 seconds" before getting in as root. The 'w' command tells me the load factor is in the 30s I reboot the machine and try this all again. Same result. Frustrating. So I reboot and this time I unmount the two JFS partitions. I then start FF, copy the URL, start chromium, paste the URL ... and all works fine! I suppose, ultimately, the question is not simply is it a problem with JSF, but is it a problem with the scheduler? Is it an interaction with the other FS drivers? As advised by a page on JFS setup I have # cat /sys/block/sda/queue/scheduler noop [deadline] cfq Has anyone observed this? I'm not finding illumination with google but perhaps my fu is off today as well. Any ideas? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Tue, 16 Jan 2018 17:02:47 -0500 Anton Aylward <opensuse@antonaylward.com> wrote:
On 15/01/18 01:27 PM, Per Jessen wrote:
Anton Aylward wrote:
So I'm looking to migrate to another obscure file system. JFS.
My favourite. Except for test systems with ext4, everything here is JFS. Has been since ... forever. JFS v1.1 maybe.
I have a serious problem and it seems related to JFS
I have converted a couple of partitions to JFS, mostly large files, PDF document movies/presentations.
They were mounted bt not being accessed.
I have firefox running and encounter a site that doesn't want to work with FF, needs chromium. Again, a presentation. What I normally do in these circumstances is copy the URL, start chromium in other window inder KDE (I keep a couple space just for this purpose) and view the rpesentation there.
I've done this many times.
Today I try it and my machine freezes. The disk light comes on and stays on. Mouse is unresponsive.
I hot key ctl-altF1 and log in as root. Machine is unresponsive and I get "login timeout after 60 seconds" before getting in as root. The 'w' command tells me the load factor is in the 30s
I reboot the machine and try this all again. Same result. Frustrating.
So I reboot and this time I unmount the two JFS partitions. I then start FF, copy the URL, start chromium, paste the URL ... and all works fine!
I suppose, ultimately, the question is not simply is it a problem with JSF, but is it a problem with the scheduler? Is it an interaction with the other FS drivers?
As advised by a page on JFS setup I have
# cat /sys/block/sda/queue/scheduler noop [deadline] cfq
Has anyone observed this? I'm not finding illumination with google but perhaps my fu is off today as well. Any ideas?
Maybe version numbers for system, kernel, FF, chromium, JFS etc might help. Also what process is causing that load factor? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 16/01/18 07:59 PM, Dave Howorth wrote:
So I reboot and this time I unmount the two JFS partitions. I then start FF, copy the URL, start chromium, paste the URL ... and all works fine!
I suppose, ultimately, the question is not simply is it a problem with JSF, but is it a problem with the scheduler? Is it an interaction with the other FS drivers?
As advised by a page on JFS setup I have
# cat /sys/block/sda/queue/scheduler noop [deadline] cfq
Has anyone observed this? I'm not finding illumination with google but perhaps my fu is off today as well. Any ideas? Maybe version numbers for system, kernel, FF, chromium, JFS etc might help. Also what process is causing that load factor?
Kernel 4.14.13-4 of Jan 12 from the kernel Stable repository FF 52.2.0 chromium 63.0.3239.108-104.44.1 from openSUSE-Leap-42.2-Update I'm presuming the jfs driver itself is from /lib/modules/ I run strings on that and don't find any version number The jfsutil is 1.1.15-27.1 # fsck.jfs -V fsck.jfs version 1.1.15, 04-Mar-2011 when I run search it seems that's the version for tumbleweed and 42.3 as well. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
On 16/01/18 07:59 PM, Dave Howorth wrote:
So I reboot and this time I unmount the two JFS partitions. I then start FF, copy the URL, start chromium, paste the URL ... and all works fine!
I suppose, ultimately, the question is not simply is it a problem with JSF, but is it a problem with the scheduler? Is it an interaction with the other FS drivers?
As advised by a page on JFS setup I have
# cat /sys/block/sda/queue/scheduler noop [deadline] cfq
Has anyone observed this? I'm not finding illumination with google but perhaps my fu is off today as well. Any ideas? Maybe version numbers for system, kernel, FF, chromium, JFS etc might help. Also what process is causing that load factor?
Kernel 4.14.13-4 of Jan 12 from the kernel Stable repository FF 52.2.0 chromium 63.0.3239.108-104.44.1 from openSUSE-Leap-42.2-Update
I'm presuming the jfs driver itself is from /lib/modules/ I run strings on that and don't find any version number
The jfsutil is 1.1.15-27.1 # fsck.jfs -V fsck.jfs version 1.1.15, 04-Mar-2011
when I run search it seems that's the version for tumbleweed and 42.3 as well.
Yes, 1.15 is the latest version. I have no idea what the problem might be, but given that it is reproducable, you ought to have a good chance of diagnosing it. -- Per Jessen, Zürich (4.2°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 00:59, Dave Howorth wrote:
Maybe version numbers for system, kernel, FF, chromium, JFS etc might help. Also what process is causing that load factor?
How does one find that out? (the load factor, that is.) My SUSE laptop takes ages to boot, and as part of the boot process it fires up xosview. That tells me that the load factor is often maybe 6, a bit lower, whatever. On a twin core processor, as you can guess, it takes ages for the laptop to become responsive. My gentoo desktop almost never goes over 2 during boot, and on three cores, that's fine. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On Wed, 17 Jan 2018 14:11:03 +0000 Wols Lists <antlists@youngman.org.uk> wrote:
On 17/01/18 00:59, Dave Howorth wrote:
Maybe version numbers for system, kernel, FF, chromium, JFS etc might help. Also what process is causing that load factor?
How does one find that out? (the load factor, that is.)
Generally, I just use top. There may be better ways. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 09:54 AM, Dave Howorth wrote:
On Wed, 17 Jan 2018 14:11:03 +0000 Wols Lists <antlists@youngman.org.uk> wrote:
On 17/01/18 00:59, Dave Howorth wrote:
Maybe version numbers for system, kernel, FF, chromium, JFS etc might help. Also what process is causing that load factor?
How does one find that out? (the load factor, that is.)
Generally, I just use top. There may be better ways.
The most minimalist way I know of is the command 'w' However all of this is assuming that you have a functioning terminal, that you can log in somewhere, either as ctl-alt-F1 or as a remote terminal on a tty port. When I said the system was frozen I meant that. One time I recovered from a situationist like this before I used JSF I found a load factor of 23 on recovery; it must have been higher when I was locked out. The load fact as I'm working now is load average: 0.49, 0.53, 0.57 That's pretty typical in my day with T'bird, a couple or 4 konsole windows/sessions, Konqueror as a file manager with 10 tabs, and two Firefox windows each with a couple of hundred tabs and few hundred more as subject-specific 'groups'. In my case, when I met this freeze, I hit ctl-alt-F1 and waited. I may as well have put on my winter duds and gone across the road to Tims for a take out before I got a login prompt back. YAWN! So I entered "r" wait ... "o" .... wait "o" ... wait "t" return. long wait. Then instead of a prompt for password I got a message telling me that login had timed out after 60 seconds. And it took a lot longer than 60 seconds for that to occur! Same when I tried again. The only way out was a reboot. The next time I try this experiment, this time it will be with CFQ scheduling, [It was "DEADLINE" before, as recommended by some of the docco on JFS] I will have a login to root ready and waiting :-) How responsive that will be, well, we'll see, oh, on scheduling ... https://wiki.archlinux.org/index.php/JFS#Deadline_I.2FO_scheduler https://bbs.archlinux.org/viewtopic.php?id=52338 https://en.wikipedia.org/wiki/Deadline_scheduler fascinating! -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Anton Aylward wrote:
In my case, when I met this freeze, I hit ctl-alt-F1 and waited. I may as well have put on my winter duds and gone across the road to Tims for a take out before I got a login prompt back. YAWN!
So I entered "r" wait ... "o" .... wait "o" ... wait "t" return. long wait.
Then instead of a prompt for password I got a message telling me that login had timed out after 60 seconds.
I have seen that before, I just don't recall the exact circumstances.
The next time I try this experiment, this time it will be with CFQ scheduling, [It was "DEADLINE" before, as recommended by some of the docco on JFS] [I will have a login to root ready and waiting :-) How responsive that will be, well, we'll see,
I doubt if changing the scheduling will produce any noticable effect on a desktop machine. It doesn't produce much on our storage servers (elevator=deadline), but they're not heavily loaded. Just my opinion. -- Per Jessen, Zürich (3.1°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 14:54, Dave Howorth wrote:
On Wed, 17 Jan 2018 14:11:03 +0000 Wols Lists <antlists@youngman.org.uk> wrote:
On 17/01/18 00:59, Dave Howorth wrote:
Maybe version numbers for system, kernel, FF, chromium, JFS etc might help. Also what process is causing that load factor?
How does one find that out? (the load factor, that is.)
Generally, I just use top. There may be better ways.
What's the option that tells me what's causing the load? As I understand it, "load" is the number of processes waiting in the queue, and top prioritizes telling me which programs are running, ie not the ones waiting. So if top tells me, I don't know how to recognise it ... Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 10:47 AM, Wols Lists wrote:
On 17/01/18 14:54, Dave Howorth wrote:
On Wed, 17 Jan 2018 14:11:03 +0000 Wols Lists <antlists@youngman.org.uk> wrote:
On 17/01/18 00:59, Dave Howorth wrote:
Maybe version numbers for system, kernel, FF, chromium, JFS etc might help. Also what process is causing that load factor?
How does one find that out? (the load factor, that is.)
Generally, I just use top. There may be better ways.
What's the option that tells me what's causing the load?
As I understand it, "load" is the number of processes waiting in the queue, and top prioritizes telling me which programs are running, ie not the ones waiting.
So if top tells me, I don't know how to recognise it ...
'top" or "htop" tells you a lot of things depending on how you configure it. The load average is not the figures you get from the columns - those are instantaneous measurements of single processes. The load average is "over time" and a logarithmic scale of the last few minutes and further back. It is not about a single process, it is not just CPU, it is not just memory demand, it is not just IO demands; it takes into account all of them. Hence a intensely swappy/thrashing system where there is no work being done by the processes since the kernel/virtualmemory system is taking up everything and blocking every process will have a high load factor. The figures in those columns -- if you can get 'top' to run and update at all in this circumstance - are going to be meaningless. Yes, 'top' does display the load averages, if you configure it that way. but rely on the lighter weight 'w' or 'uptime' main:~ # uptime 11:05am up 1:08, 5 users, load average: 0.84, 0.78, 0.72 As a set of three, you can tell if load is increasing or decreasing, which is useful. You need to get to know your system. I know that a LA of more than 5 means my system is getting very busy. For you system the threshold might be higher or lower. I've used systems that are still usable with a LA in the 20s. YMMV. observe your system over the longer term and various activities. I'd have to look at the current sources, but when I was maintaining the V7 kernel the code that computed the LA was the only bit of the kernel that used floating point and was exponentially-damped moving sums sample. I'm told the Linux version uses fixed point. The important thing to note is that these figures can include blocked/unitneruptable process, that is processes taking zero CPU; they are blocked because they are waiting for IO complete. That my be programmed IO or it my be virtual memory IO. When my system freezes as I mentioned at the start of this sub-thread, yes my disk light comes on and stays on. Somehow there is a memory/IO clash between the processes. I don't know what it is. Think: "thrashing" in the extreme. Back to "top' The columns really represent what you might get from 'pidstat'. Adding the '-d" parameter is a bit more useful. But what you really want is 'iostat' Oh look! I can drill down specifically on the JFS devices that I'm not even accessing! -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 16:31, Anton Aylward wrote:
The load average is "over time" and a logarithmic scale of the last few minutes and further back. It is not about a single process, it is not just CPU, it is not just memory demand, it is not just IO demands; it takes into account all of them. Hence a intensely swappy/thrashing system where there is no work being done by the processes since the kernel/virtualmemory system is taking up everything and blocking every process will have a high load factor. The figures in those columns -- if you can get 'top' to run and update at all in this circumstance - are going to be meaningless.
Well, xosview tells me the load average, and the system is horribly slow, and this is boot, so I'm swearing at it ... Bear in mind I'm running 64-bit SUSE in 3gig of RAM, could that be some of it? Unfortunately, it's a tosh, so when I tried to upgrade it with a 4gig stick (making it 4+2) the new ram promptly caused it to refuse to boot - well, it booted to console in gentoo, but both SUSE and Windows bombed :-( And seeing as I'd already returned the ram twice, I didn't feel like returning it a 3rd time - so I've got a 4GB DDR3L SODIMM that's probably perfectly okay, just that my tosh doesn't like it (toshes apparently are *extremely* sensitive to what ram they do or don't like). Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 12:01 PM, Wols Lists wrote:
Well, xosview tells me the load average, and the system is horribly slow, and this is boot, so I'm swearing at it ...
Yes, but xosview runs on top of X and tor un X you need X up and to run X you nee the system up and responsive ... that's not lightweight! Whereas 'w', 'uptime', 'iostat', 'pidstat' are all text tools. less demanding
Bear in mind I'm running 64-bit SUSE in 3gig of RAM, could that be some of it?
It could be, but then I'm running with just 4G of DDR2.
Unfortunately, it's a tosh, so when I tried to upgrade it with a 4gig stick (making it 4+2) the new ram promptly caused it to refuse to boot - well, it booted to console in gentoo, but both SUSE and Windows bombed :-(
Maybe a memory size conflict. Try it with JUST the 4G stick.
And seeing as I'd already returned the ram twice, I didn't feel like returning it a 3rd time - so I've got a 4GB DDR3L SODIMM that's probably perfectly okay, just that my tosh doesn't like it (toshes apparently are *extremely* sensitive to what ram they do or don't like).
It might be that it is so sensitive that you have other problems. I suppose someone is going to ask why don't you sell it and get a 'real' computer, so I might as be the one. http://dilbert.com/strip/1995-06-24 -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2018-01-17 18:47, Anton Aylward wrote:
I suppose someone is going to ask why don't you sell it and get a 'real' computer, so I might as be the one. http://dilbert.com/strip/1995-06-24
I think I may lack part of the American or Unix culture to fully understand that joke :-) It does make me giggle, but I'm sure I don't fully understand it. Could you explain, please? :-) -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Den 2018-01-17 kl. 19:23, skrev Carlos E. R.:
On 2018-01-17 18:47, Anton Aylward wrote:
I suppose someone is going to ask why don't you sell it and get a 'real' computer, so I might as be the one. http://dilbert.com/strip/1995-06-24 I think I may lack part of the American or Unix culture to fully understand that joke :-)
It does make me giggle, but I'm sure I don't fully understand it.
Could you explain, please? :-)
I'll bite. in include/math-emu/single.h #if _FP_W_TYPE_SIZE < 32 #error "Here's a nickel kid. Go buy yourself a real computer." #endif Unfortunately Linus left out the log history when he imported to git in 2005. I probably have it somewhere in a bitkeeper repo on some old ancient HD. git log include/math-emu/single.h commit 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 (tag: v2.6.12-rc2) Author: Linus Torvalds <torvalds@ppc970.osdl.org> Date: Sat Apr 16 15:20:36 2005 -0700 Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip! Cheers, -- /bengan -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Wednesday, 2018-01-17 at 20:38 +0100, Bengt Gördén wrote:
Den 2018-01-17 kl. 19:23, skrev Carlos E. R.:
On 2018-01-17 18:47, Anton Aylward wrote:
I suppose someone is going to ask why don't you sell it and get a 'real' computer, so I might as be the one. http://dilbert.com/strip/1995-06-24 I think I may lack part of the American or Unix culture to fully understand that joke :-)
It does make me giggle, but I'm sure I don't fully understand it.
Could you explain, please? :-)
I'll bite.
in include/math-emu/single.h
#if _FP_W_TYPE_SIZE < 32 #error "Here's a nickel kid. Go buy yourself a real computer." #endif
Wow!
Unfortunately Linus left out the log history when he imported to git in 2005. I probably have it somewhere in a bitkeeper repo on some old ancient HD.
git log include/math-emu/single.h
commit 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 (tag: v2.6.12-rc2) Author: Linus Torvalds <torvalds@ppc970.osdl.org> Date: Sat Apr 16 15:20:36 2005 -0700
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
Wow. :-)
Cheers,
Thanks :-) - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpfxQIACgkQtTMYHG2NR9VYGwCfdoYEsHzLSvWqgN4iWhU34A8w M4IAn0AaNwNj7kiEoCvECsE8Y/117/o0 =Yjzm -----END PGP SIGNATURE-----
On 17/01/18 02:38 PM, Bengt Gördén wrote:
Den 2018-01-17 kl. 19:23, skrev Carlos E. R.:
On 2018-01-17 18:47, Anton Aylward wrote:
I suppose someone is going to ask why don't you sell it and get a 'real' computer, so I might as be the one. http://dilbert.com/strip/1995-06-24 I think I may lack part of the American or Unix culture to fully understand that joke :-)
It does make me giggle, but I'm sure I don't fully understand it.
Could you explain, please? :-)
I'll bite.
in include/math-emu/single.h
#if _FP_W_TYPE_SIZE < 32 #error "Here's a nickel kid. Go buy yourself a real computer." #endif
Unfortunately Linus left out the log history when he imported to git in 2005. I probably have it somewhere in a bitkeeper repo on some old ancient HD.
git log include/math-emu/single.h
commit 1da177e4c3f41524e886b7f1b8a0c1fc7321cac2 (tag: v2.6.12-rc2) Author: Linus Torvalds <torvalds@ppc970.osdl.org> Date: Sat Apr 16 15:20:36 2005 -0700
Linux-2.6.12-rc2 Initial git repository build. I'm not bothering with the full history, even though we have it. We can create a separate "historical" git archive of that later if we want to, and in the meantime it's about 3.2GB when imported into git - space that would just make the early git days unnecessarily complicated, when we don't have a lot of good infrastructure for it. Let it rip!
There's a footnote: https://wiert.me/2015/06/24/20-years-ago-today-heres-a-nickel-kid-go-buy-you... 20 years ago, we were in the autumn of the 16-bit computing era, the heydays of the 486 family of processors. While Delphi 95 still being 16-bit, but Windows 95 trying to bridge the gap between 16-bit DOS / Windows 3.x programs and the 32-bit Windows NT world. Linux and 68000 both were far ahead in the 32-bit world. Not quite: as a comment points out In 1995 your machine would have been already a Pentium, unless you were going for a cheap one. But it took really a loooong to switch from 16 bit computing to 32 bit, since 32 bit processors had been available for years. The problem was not processors – Intel CPUs were already fairly capable ones with some features unmatched by competition, especially its security model never fully used by any OS – it was software support that took too long to move ahead. Can you say SCO UNIX? How about Convergent? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 17:47, Anton Aylward wrote:
It might be that it is so sensitive that you have other problems.
I suppose someone is going to ask why don't you sell it and get a 'real' computer, so I might as be the one. http://dilbert.com/strip/1995-06-24
You mean one of these? http://www.programguiden.se/ithistoria/8.html (the colour picture, second, not the black-and-white first one). I have fond memories of working on them. I also remember those large disk packs - 300 MEG! Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 17:47, Anton Aylward wrote:
Unfortunately, it's a tosh, so when I tried to upgrade it with a 4gig stick (making it 4+2) the new ram promptly caused it to refuse to boot - well, it booted to console in gentoo, but both SUSE and Windows bombed:-(
Maybe a memory size conflict. Try it with JUST the 4G stick.
Don't you think I didn't? :-) A web search led me to conclude that if you didn't buy the memory that Toshiba recommend, it's unlikely to work. Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 02:26 PM, Wol's lists wrote:
On 17/01/18 17:47, Anton Aylward wrote:
Unfortunately, it's a tosh, so when I tried to upgrade it with a 4gig stick (making it 4+2) the new ram promptly caused it to refuse to boot - well, it booted to console in gentoo, but both SUSE and Windows bombed:-(
Maybe a memory size conflict. Try it with JUST the 4G stick.
Don't you think I didn't? :-)
A web search led me to conclude that if you didn't buy the memory that Toshiba recommend, it's unlikely to work.
I know that there are problems with Intel demands for memory and AMD demands for memory on mobos. I can't recall what tey are but it means the AMD style is a lot cheaper. Something to do with buffering IIR. One obvious question: if the tosh was to be so problematic why did you get it? was it a great deal or was it a good idea at the time? -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 22:47, Anton Aylward wrote:
On 17/01/18 02:26 PM, Wol's lists wrote:
On 17/01/18 17:47, Anton Aylward wrote:
Unfortunately, it's a tosh, so when I tried to upgrade it with a 4gig stick (making it 4+2) the new ram promptly caused it to refuse to boot - well, it booted to console in gentoo, but both SUSE and Windows bombed:-(
Maybe a memory size conflict. Try it with JUST the 4G stick.
Don't you think I didn't? :-)
A web search led me to conclude that if you didn't buy the memory that Toshiba recommend, it's unlikely to work.
I know that there are problems with Intel demands for memory and AMD demands for memory on mobos. I can't recall what tey are but it means the AMD style is a lot cheaper. Something to do with buffering IIR.
One obvious question: if the tosh was to be so problematic why did you get it? was it a great deal or was it a good idea at the time?
It was a good idea at the time ... Bear in mind, this upgrade was an attempt to give an aging computer a new lease of life - it was probably at least four (if not more) years old. It got a new battery at about the same time ... (And yes, with the new battery and bigger hard disk, it still is a nice machine. It's just that with more ram I would probably be able to hammer it more. Oh, and yes, it is an Intel chip - I prefer AMD but I think AMD laptops were like hens' teeth at the time.) Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2018-01-17 18:01, Wols Lists wrote:
Well, xosview tells me the load average, and the system is horribly slow, and this is boot, so I'm swearing at it ...
Bear in mind I'm running 64-bit SUSE in 3gig of RAM, could that be some of it? Unfortunately, it's a tosh, so when I tried to upgrade it with a 4gig stick (making it 4+2) the new ram promptly caused it to refuse to boot - well, it booted to console in gentoo, but both SUSE and Windows bombed :-(
And seeing as I'd already returned the ram twice, I didn't feel like returning it a 3rd time - so I've got a 4GB DDR3L SODIMM that's probably perfectly okay, just that my tosh doesn't like it (toshes apparently are *extremely* sensitive to what ram they do or don't like).
Try replacing the HD with an SSD. I did. System is much faster, mostly due to swap speed (seek times, my educated guess). -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Wols Lists wrote:
On 17/01/18 16:31, Anton Aylward wrote:
The load average is "over time" and a logarithmic scale of the last few minutes and further back. It is not about a single process, it is not just CPU, it is not just memory demand, it is not just IO demands; it takes into account all of them. Hence a intensely swappy/thrashing system where there is no work being done by the processes since the kernel/virtualmemory system is taking up everything and blocking every process will have a high load factor. The figures in those columns -- if you can get 'top' to run and update at all in this circumstance - are going to be meaningless.
Well, xosview tells me the load average, and the system is horribly slow, and this is boot, so I'm swearing at it ...
Bear in mind I'm running 64-bit SUSE in 3gig of RAM, could that be some of it?
3Gb should be fine, but 1Gb more wouldn't go amiss. Check to sww how much swap space it's normally using - but for the startup, I wouldn't expect 3Gb to cause a problem (unless you're starting up a few concurrent firefoxes and libreoffices).
Unfortunately, it's a tosh, so when I tried to upgrade it with a 4gig stick (making it 4+2) the new ram promptly caused it to refuse to boot - well, it booted to console in gentoo, but both SUSE and Windows bombed :-(
Pretty odd - still, if there was a memory config problem, it wouldn't have booted either of them.
And seeing as I'd already returned the ram twice, I didn't feel like returning it a 3rd time - so I've got a 4GB DDR3L SODIMM that's probably perfectly okay, just that my tosh doesn't like it (toshes apparently are *extremely* sensitive to what ram they do or don't like).
My current laptop is also a toshiba, also 3Gb I think, but I'm not on 42.3 on that. -- Per Jessen, Zürich (3.6°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 17/01/18 18:28, Per Jessen wrote:
My current laptop is also a toshiba, also 3Gb I think, but I'm not on 42.3 on that.
Dunno how old mine is, but a good few years. It was a pretty modern system with Win7 home 32-bit as originally supplied. Satellite L670. I think most 32-bit systems came with 3GB (despite being a 64-bit CPU) - I was going to stick the redundant 1GB stick in my father-in-law's 2GB machine, but it's not redundant ... I can't remember quite why, but I decided to upgrade it, bought a 2TB drive and 4GB SODIMM, and had a go. The drive upgrade was a cinch, the ram, well, I gave up ... (actually, I think it was because I was running out of disk space. The new C: and D: drives are each individually larger than the entire old hard drive :-) So the laptop now boots Win10, SUSE, and a not-fully-installed gentoo (as in, systemd seems to be a second-class citizen on gentoo - although it's supposedly an "all about choice" distro there seem to be a *lot* of systemd-haters) because I couldn't get systemd networking to work. And of course, with no network, getting past bootstrap to a fully functional system is rather difficult :-) Cheers, Wol -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Wol's lists wrote:
On 17/01/18 18:28, Per Jessen wrote:
My current laptop is also a toshiba, also 3Gb I think, but I'm not on 42.3 on that.
Dunno how old mine is, but a good few years. It was a pretty modern system with Win7 home 32-bit as originally supplied. Satellite L670. I think most 32-bit systems came with 3GB (despite being a 64-bit CPU) -
Haha, mine's an L675, also a good few years old. It's running openSUSE 12.3 though. -- Per Jessen, Zürich (4.6°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 16/01/18 05:02 PM, Anton Aylward wrote:
I suppose, ultimately, the question is not simply is it a problem with JSF, but is it a problem with the scheduler? Is it an interaction with the other FS drivers?
As advised by a page on JFS setup I have
# cat /sys/block/sda/queue/scheduler noop [deadline] cfq
Today I set the scheduler to CFQ and tried again. The system did not hang. Running both FF and Chromium the system was slow redrawing content when switching from one to the other, opening new tabs to paste in URL from the other. But it didn't hang. I'm sure if I had a dedicated fast graphics board that redraw issue would be overcome. I'm going to continue converting to JFS. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 15.01.2018 18:50, Anton Aylward wrote:
Is JFS multi-threaded?
I think so: # lvcreate -L 1G -n jfstest system # mkfs.jfs /dev/system/jfstest # mount /dev/system/jfstest /mnt # ps -ef |grep jfs root 5485 2 0 19:30 ? 00:00:00 [jfsIO] root 5486 2 0 19:30 ? 00:00:00 [jfsCommit] root 5487 2 0 19:30 ? 00:00:00 [jfsCommit] root 5488 2 0 19:30 ? 00:00:00 [jfsCommit] root 5489 2 0 19:30 ? 00:00:00 [jfsCommit] root 5490 2 0 19:30 ? 00:00:00 [jfsSync] I have 4 CPUs and there are some kernel threads for jfs. If i mount another jfs filesystem, the number of kernel threads remains the same. How JFS works with parallel workload, you will have to try yourself.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday, 2018-01-15 at 12:50 -0500, Anton Aylward wrote:
I've been using ReiserFS quite happily for many years now and I think it is one of those software packages that was an excellent/genius design; it has needed very little maintenance over the years. The PowerThatBe, however, think that it is 'unsupported' (actually it isn't, it just doesn't have the crowds that theext4 does since it don't need much work, which to my mind reflects badly on ext4) and so are depreciating it, especially with Leap-15.
I just joined my 5 remaining reiserfs running partitions into a single one: three of them were contiguous. I put the contents of 4 of them in the same partitions and did bind mounts on the directories. The 5th I converted to ext4. I keep thus a single large reiserfs partitions because it has a lot of small files. For instance, one of the partitions had more than 600000 files in 1.5 GB. - -- Cheers, Carlos E. R. (from openSUSE 42.2 x86_64 "Malachite" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iEYEARECAAYFAlpjkiIACgkQtTMYHG2NR9XUAQCfRWmSB1WGMY8tlbEfB4W8Rsgd LH0AniU7nzcuxBIzw+LIGBkkj2xZNk+C =y5h+ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 20/01/18 02:01 PM, Carlos E. R. wrote:
I just joined my 5 remaining reiserfs running partitions into a single one: three of them were contiguous. I put the contents of 4 of them in the same partitions and did bind mounts on the directories. The 5th I converted to ext4.
I keep thus a single large reiserfs partitions because it has a lot of small files. For instance, one of the partitions had more than 600000 files in 1.5 GB.
I have a one-line script that tells me the distribution of file sizes, in base two increments. If anyone wasn't to adjust for other granularity or a fixed array of size, I'd be pleased to see the result. Here is the one liner: find . -xdev -type f -print0 | xargs -0 ls -l | \ awk '{size[int(log($5)/log(2))]++}\ END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | \ sort -n My remaining reiserFS are df -h | egrep "reiser" vgmain-vHome reiserfs 8.0G 5.3G 2.8G 66% /home vgmain-vCamera reiserfs 5.0G 213M 4.8G 5% /home/anton/Photographs vgmain-vDownloads reiserfs 5.0G 405M 4.7G 8% /home/anton/Downloads vgmain-vDocuments reiserfs 5.0G 761M 4.3G 15% /home/anton/Documents vgmain-vResumes reiserfs 5.0G 91M 5.0G 2% /home/anton/Documents/Resumes vgmain-vMail reiserfs 5.0G 3.0G 2.1G 60% /home/anton/Mail vgmain-vPhotoByYear reiserfs 32G 5.5G 27G 17% /home/anton/Photographs/ByYear vgmain-PhotoYear2014 reiserfs 5.0G 4.6G 477M 91% /home/anton/Photographs/ByYear/2014 You'd expect /home to have lots of small files under the hidden "dot" configs: # cd /home/anton ;find . -xdev -type f -print0 | xargs -0 ls -l | awk '{size[int(log($5)/log(2))]++}END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | sort -n 0 300 1 30 2 18 4 25 8 50 16 214 32 538 64 1211 128 1713 256 2233 512 5747 1024 11128 2048 6340 4096 8212 8192 5645 16384 4131 32768 2819 65536 2668 131072 904 262144 413 524288 262 1048576 149 2097152 104 4194304 58 8388608 25 16777216 16 33554432 10 67108864 1 134217728 1 268435456 1 Now here's the interesting one. Under my photogrpqhy-by-year there are the daya files, tar and jpg, and the sidecar files that Darktable uses. .RAF are the RAW, .xmp are the corresonding sidear files. -rw-r--r-- 1 anton anton 5.3M Mar 21 2017 2017Mar21-1605_DSCF0979.JPG -rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1605_DSCF0979.JPG.xmp -rw-r--r-- 1 anton anton 25M Mar 21 2017 2017Mar21-1605_DSCF0979.RAF -rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1605_DSCF0979.RAF.xmp -rw-r--r-- 1 anton anton 5.4M Mar 21 2017 2017Mar21-1605_DSCF0980.JPG -rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1605_DSCF0980.JPG.xmp -rw-r--r-- 1 anton anton 25M Mar 21 2017 2017Mar21-1605_DSCF0980.RAF -rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1605_DSCF0980.RAF.xmp -rw-r--r-- 1 anton anton 5.2M Mar 21 2017 2017Mar21-1606_DSCF0981.JPG -rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1606_DSCF0981.JPG.xmp -rw-r--r-- 1 anton anton 25M Mar 21 2017 2017Mar21-1606_DSCF0981.RAF -rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1606_DSCF0981.RAF.xmp -rw-r--r-- 1 anton anton 5.3M Mar 21 2017 2017Mar21-1607_DSCF0982.JPG -rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1607_DSCF0982.JPG.xmp -rw-r--r-- 1 anton anton 25M Mar 21 2017 2017Mar21-1607_DSCF0982.RAF -rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1607_DSCF0982.RAF.xmp It makes me wonder if there is space at the end of each of the data files for the sidecar file. That directory has a file size distribution of 1024 8 4194304 4 16777216 4 Overall, my 2017 photo size distribution is 1024 207 2048 22 4096 2 8192 5 131072 2 524288 6 1048576 4 4194304 125 16777216 137 and that spread seems the same year to year though the overall number may differ. So yes, if I'm using a file system with 4K blocks there is a lot of opportunity for tail packing. or conversely, there is a lot of space wasted on file systems that don't have tail packing. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 2018-01-23 04:32, Anton Aylward wrote:
On 20/01/18 02:01 PM, Carlos E. R. wrote:
I just joined my 5 remaining reiserfs running partitions into a single one: three of them were contiguous. I put the contents of 4 of them in the same partitions and did bind mounts on the directories. The 5th I converted to ext4.
I keep thus a single large reiserfs partitions because it has a lot of small files. For instance, one of the partitions had more than 600000 files in 1.5 GB.
I have a one-line script that tells me the distribution of file sizes, in base two increments. If anyone wasn't to adjust for other granularity or a fixed array of size, I'd be pleased to see the result.
Here is the one liner:
find . -xdev -type f -print0 | xargs -0 ls -l | \ awk '{size[int(log($5)/log(2))]++}\ END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | \ sort -n
Interesting! Telcontar:/data/Lareiserfs # find . -xdev -type f -print0 | xargs -0 ls -l | awk '{size[int(log($5)/log(2))]++} END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | sort -n 0 144977 1 28 2 242 4 166 8 3054 16 3082 32 2492 64 6260 128 38090 256 85895 512 44763 1024 416248 <== 2048 258586 <== 4096 70770 8192 40845 16384 27172 32768 18731 65536 10004 131072 6379 262144 4353 524288 2954 1048576 1225 2097152 531 4194304 174 8388608 78 16777216 36 33554432 8 134217728 1 Telcontar:/data/Lareiserfs #
My remaining reiserFS are
df -h | egrep "reiser" vgmain-vHome reiserfs 8.0G 5.3G 2.8G 66% /home vgmain-vCamera reiserfs 5.0G 213M 4.8G 5% /home/anton/Photographs vgmain-vDownloads reiserfs 5.0G 405M 4.7G 8% /home/anton/Downloads vgmain-vDocuments reiserfs 5.0G 761M 4.3G 15% /home/anton/Documents vgmain-vResumes reiserfs 5.0G 91M 5.0G 2% /home/anton/Documents/Resumes vgmain-vMail reiserfs 5.0G 3.0G 2.1G 60% /home/anton/Mail vgmain-vPhotoByYear reiserfs 32G 5.5G 27G 17% /home/anton/Photographs/ByYear vgmain-PhotoYear2014 reiserfs 5.0G 4.6G 477M 91% /home/anton/Photographs/ByYear/2014
Telcontar:/data/Lareiserfs # df -h | egrep "reiser" /dev/sdc13 64G 16G 49G 24% /data/Lareiserfs Telcontar:/data/Lareiserfs # There are others but they are the root partitions of old test installs. ...
-rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1606_DSCF0981.RAF.xmp -rw-r--r-- 1 anton anton 5.3M Mar 21 2017 2017Mar21-1607_DSCF0982.JPG -rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1607_DSCF0982.JPG.xmp -rw-r--r-- 1 anton anton 25M Mar 21 2017 2017Mar21-1607_DSCF0982.RAF -rw-r--r-- 1 anton anton 1.4K Oct 23 18:45 2017Mar21-1607_DSCF0982.RAF.xmp
It makes me wonder if there is space at the end of each of the data files for the sidecar file.
No, those are unused. As I understand, reiserfs packs small files on the directory structure without using data sectors. It doesn't use the empty space on larger files.
So yes, if I'm using a file system with 4K blocks there is a lot of opportunity for tail packing. or conversely, there is a lot of space wasted on file systems that don't have tail packing.
Now, what is the filesystem block size? I forgot how to find out (I'm tired today, recuperating form yesterday). Not necessarily the same size as the disk hardware or partition sector size, I understand. For ext3 google says it is tunefs. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Anton Aylward wrote:
On 20/01/18 02:01 PM, Carlos E. R. wrote:
I just joined my 5 remaining reiserfs running partitions into a single one: three of them were contiguous. I put the contents of 4 of them in the same partitions and did bind mounts on the directories. The 5th I converted to ext4.
I keep thus a single large reiserfs partitions because it has a lot of small files. For instance, one of the partitions had more than 600000 files in 1.5 GB.
I have a one-line script that tells me the distribution of file sizes, in base two increments. If anyone wasn't to adjust for other granularity or a fixed array of size, I'd be pleased to see the result.
Here is the one liner:
find . -xdev -type f -print0 | xargs -0 ls -l | \ awk '{size[int(log($5)/log(2))]++}\ END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | \ sort -n
<nitpicking> four lines in a one-liner? </nitpicking> You could shorten it to a three lines: find . -xdev -type f -ls | awk '{size[int(log($7)/log(2))]++}\ END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | \ sort -n :-) -- Per Jessen, Zürich (7.2°C) http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 23/01/18 08:47 AM, Per Jessen wrote:
<nitpicking> four lines in a one-liner? </nitpicking>
Pick your own nits! When I *run* it it is a one liner! But Thunderbird was going wrap it, so I decided to wrap it more intelligibly. -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Per, et al -- ...and then Per Jessen said... % % Anton Aylward wrote: % ... % > find . -xdev -type f -print0 | xargs -0 ls -l | \ % > awk '{size[int(log($5)/log(2))]++}\ % > END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | \ % > sort -n % % <nitpicking> % four lines in a one-liner? % </nitpicking> *grin* % % You could shorten it to a three lines: % % find . -xdev -type f -ls | awk '{size[int(log($7)/log(2))]++}\ % END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | \ % sort -n That was a nice touch, since we're playing there. I don't use -ls much since there are so crazy many fields, and I really should try the printf options I've just learned here. But ... The nice thing about -print0 is that it gets past ugly characters. Your version will break if you have a filename with (ew) a newline embedded, right? HAND :-D -- David T-G See http://justpickone.org/davidtg/email/ See http://justpickone.org/davidtg/tofu.txt -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 01/23/2018 06:28 PM, David T-G wrote:
% Anton Aylward wrote: % find . -xdev -type f -ls | awk '{size[int(log($7)/log(2))]++}\ % END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | \ % sort -n
That was a nice touch, since we're playing there. I don't use -ls much since there are so crazy many fields, and I really should try the printf options I've just learned here. But ...
The nice thing about -print0 is that it gets past ugly characters. Your version will break if you have a filename with (ew) a newline embedded, right?
you don't need the file names at all, just the file size: $ find . -xdev -type f -printf '%s\n' \ | awk '{size[int(log($1)/log(2))]++} \ END {for (i in size) printf("%10d %3d\n", 2^i, size[i])}' \ | sort -n ;-) Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
Bernhard Voelker wrote:
On 01/23/2018 06:28 PM, David T-G wrote:
% Anton Aylward wrote: % find . -xdev -type f -ls | awk '{size[int(log($7)/log(2))]++}\ % END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | \ % sort -n
That was a nice touch, since we're playing there. I don't use -ls much since there are so crazy many fields, and I really should try the printf options I've just learned here. But ...
The nice thing about -print0 is that it gets past ugly characters. Your version will break if you have a filename with (ew) a newline embedded, right?
you don't need the file names at all, just the file size:
$ find . -xdev -type f -printf '%s\n' \ | awk '{size[int(log($1)/log(2))]++} \ END {for (i in size) printf("%10d %3d\n", 2^i, size[i])}' \ | sort -n
Nice one, Berny! -- Per Jessen, Zürich (5.5°C) http://www.hostsuisse.com/ - dedicated server rental in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
David T-G wrote:
Per, et al --
...and then Per Jessen said... % % Anton Aylward wrote: % ... % > find . -xdev -type f -print0 | xargs -0 ls -l | \ % > awk '{size[int(log($5)/log(2))]++}\ % > END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | \ % > sort -n % % <nitpicking> % four lines in a one-liner? % </nitpicking>
*grin*
% % You could shorten it to a three lines: % % find . -xdev -type f -ls | awk '{size[int(log($7)/log(2))]++}\ % END{for (i in size) printf("%10d %3d\n", 2^i, size[i])}' | \ % sort -n
That was a nice touch, since we're playing there. I don't use -ls much since there are so crazy many fields, and I really should try the printf options I've just learned here. But ...
The nice thing about -print0 is that it gets past ugly characters. Your version will break if you have a filename with (ew) a newline embedded, right?
I was wondering about that too, but I thought awk might be able to deal with it. -- Per Jessen, Zürich (5.4°C) http://www.hostsuisse.com/ - virtual servers, made in Switzerland. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 23/01/18 12:28 PM, David T-G wrote:
The nice thing about -print0 is that it gets past ugly characters. Your version will break if you have a filename with (ew) a newline embedded, right?
A LOT of things will break, then. The real solution, like it is to so many of life's problwems, is Well Don't Do that, then! -- A: Yes. > Q: Are you sure? >> A: Because it reverses the logical flow of conversation. >>> Q: Why is top posting frowned upon? -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (13)
-
Anton Aylward -
Bengt Gördén -
Bernhard Voelker -
Carlos E. R. -
Dave Howorth -
David T-G -
Florian Gleixner -
James Knott -
Knurpht - Gertjan Lettink -
Lew Wolfgang -
Per Jessen -
Wol's lists -
Wols Lists