A year or so ago I started a discussion about a problem I had with the disk cache getting full, and then there being long delays when it was emptied. The solution at that time (openSUSE 12.1) was to periodically run:
echo 1 > /proc/sys/vm/drop_caches
At the time, this solved everything. For some reason I cannot understand, I think this problem has returned. The cache seems to be filling up - but the command above does not help.
I looked through the archives but could not find the message from the person who suggested an X app that monitored the various cache. I found this tool very useful at the time. But I have since forgotten it. I usually record this kind of information, but not that time. Anyone recall the name of the tool?
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:
The cache seems to be filling up
That is GOOD! The kernel uses memory not used by applications to cache things regarding file system reads/writes. I don't see anything wrong with that. In fact, I personally don't have fast SSDs, but rather bought 20G of memory. The processes are only using ~2-8G, so everything else can be - and in fact - is used as cache. I doubt that for my case an SSD could be faster than RAM. I admit this is an extreme attitude, but it adheres to the very old yet still valid statement: "to make a UNIX system faster, add more RAM". It's not that UNIXoide and thus Linux kernels need more memory, but if it's there, then they'll use it very efficiently.
Therefore: simply trust your kernel (unless you have modified the cache settings into an disadvantageous range).
Regarding writing to the drop_cache pseudo file: I use it only when I want to measure the performance to simulate a "cold" cache, i.e., when the program and data needed for a process is not yet in memory but has to be read from disk. A subsequent run may be faster in magnitudes, and therefore can't be taken for comparison.
Have a nice day, Berny
Le 22/09/2014 09:07, Bernhard Voelker a écrit :
don't have fast SSDs, but rather bought 20G of memory. The
not any mpobo accept that :-(
jdd
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker mail@bernhard-voelker.de wrote:
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:
The cache seems to be filling up
That is GOOD!
It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.
On 09/22/2014 09:20 AM, Andrei Borzenkov wrote:
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker mail@bernhard-voelker.de wrote:
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:
The cache seems to be filling up
That is GOOD!
It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.
Well, in that case, you'll have to wait for the data to be sync'ed to disk anyway. ;-)
Have a nice day, Berny
On Mon, Sep 22, 2014 at 12:10 PM, Bernhard Voelker mail@bernhard-voelker.de wrote:
On 09/22/2014 09:20 AM, Andrei Borzenkov wrote:
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker mail@bernhard-voelker.de wrote:
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:
The cache seems to be filling up
That is GOOD!
It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.
Well, in that case, you'll have to wait for the data to be sync'ed to disk anyway. ;-)
Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system.
So while every single application will never write faster, each application could be more responsive with smaller amount of cached data.
What Linux still lacks is rate-limiting of program writes to match output device speed.
On 22/09/14 09:50, Andrei Borzenkov wrote:
Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system.
IMO this is a result of bad design or a bug in the application causing the block - I use several 3d graphics/rendering applications which generate very large amounts of intermediate data (much more than can be written to disk at the speed it's generated) and none of them block other applications, even though they may be unresponsive themselves due to waiting for their own write-out to finish...
Dx
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Monday, 2014-09-22 at 10:34 +0100, Dylan wrote:
On 22/09/14 09:50, Andrei Borzenkov wrote:
Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system.
IMO this is a result of bad design or a bug in the application causing the block - I use several 3d graphics/rendering applications which generate very large amounts of intermediate data (much more than can be written to disk at the speed it's generated) and none of them block other applications, even though they may be unresponsive themselves due to waiting for their own write-out to finish...
Maybe they use "sync" writes.
- -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Monday, 2014-09-22 at 12:50 +0400, Andrei Borzenkov wrote:
On Mon, Sep 22, 2014 at 12:10 PM, Bernhard Voelker <> wrote:
On 09/22/2014 09:20 AM, Andrei Borzenkov wrote:
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker <> wrote:
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:
The cache seems to be filling up
That is GOOD!
It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.
Well, in that case, you'll have to wait for the data to be sync'ed to disk anyway. ;-)
Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system.
AHHHHH!!!!!!!
SO THAT'S WHY!!
That's why writing big files, or many files, to slow usb flash devices makes the entire system crawl!
So while every single application will never write faster, each application could be more responsive with smaller amount of cached data.
What Linux still lacks is rate-limiting of program writes to match output device speed.
Can we limit, externally, the amount of cache an application can use?
Say I want to start a large file copy to usb stick, like a dd to create an usb installation media. How do I tell it not to use a big write cache?
Or a many file copy to a usb device, using, say, mc.
Or instead of limiting the cache per application, do so per device. If the usb is on /dev/sdh, limit the cache on /dev/sdh.
Is this possible at all?
- -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
On September 22, 2014 7:18:19 AM EDT, "Carlos E. R." carlos.e.r@opensuse.org wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Monday, 2014-09-22 at 12:50 +0400, Andrei Borzenkov wrote:
On Mon, Sep 22, 2014 at 12:10 PM, Bernhard Voelker <> wrote:
On 09/22/2014 09:20 AM, Andrei Borzenkov wrote:
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker <> wrote:
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:
The cache seems to be filling up
That is GOOD!
It depends. It is good when cache is filled up with data waiting to
be
read. It is bad when it is filled up with data waiting to be
written
out to slow device.
Well, in that case, you'll have to wait for the data to be sync'ed
to
disk anyway. ;-)
Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application)
has
to wait while data is drained. And that blocks not only write-out,
but
also reading of data. Resulting in very long delays and
non-responsive
system.
AHHHHH!!!!!!!
SO THAT'S WHY!!
That's why writing big files, or many files, to slow usb flash devices makes the entire system crawl!
So while every single application will never write faster, each application could be more responsive with smaller amount of cached data.
What Linux still lacks is rate-limiting of program writes to match output device speed.
Can we limit, externally, the amount of cache an application can use?
Say I want to start a large file copy to usb stick, like a dd to create an usb installation media. How do I tell it not to use a big write cache?
Well for dd anyway you can bypass the cache all together. Adding oflag=direct should do it.
Or a many file copy to a usb device, using, say, mc.
Or instead of limiting the cache per application, do so per device. If the usb is on /dev/sdh, limit the cache on /dev/sdh.
Is this possible at all?
No idea on your other questions.
Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux)
iEYEARECAAYFAlQgBXsACgkQtTMYHG2NR9UMiQCfafHNXMhYFVc/H/CyKbQtWoPU kikAn0WQHPj/3j9zS7PTDfovyZQq3VTb =lLCJ -----END PGP SIGNATURE-----
On 2014-09-22 14:49, Greg Freemyer wrote:
Well for dd anyway you can bypass the cache all together. Adding oflag=direct should do it.
Ok, so that flag and a reasonable bs to compensate.
What about "dsync"? Or "nocache"?
Maybe I'll try them to time the results later.
Or a many file copy to a usb device, using, say, mc.
Or instead of limiting the cache per application, do so per device. If the usb is on /dev/sdh, limit the cache on /dev/sdh.
Is this possible at all?
No idea on your other questions.
Pity...
On September 22, 2014 9:18:44 AM EDT, "Carlos E. R." robin.listas@telefonica.net wrote:
On 2014-09-22 14:49, Greg Freemyer wrote:
Well for dd anyway you can bypass the cache all together. Adding
oflag=direct should do it.
Ok, so that flag and a reasonable bs to compensate.
What about "dsync"? Or "nocache"?
Dsync just makes sure the data is written to disk asap, but it leaves a copy in cache. Not useful for writing an iso to a flash.
Nocache probably works, but it first uses the kernel disk cache mechanism, then flushes it. That may or may not solve your issue.
Direct eliminates the use of the kernel cache altogether; that seems best for that use case.
Maybe I'll try them to time the results later.
On 2014-09-22 16:09, Greg Freemyer wrote:
On September 22, 2014 9:18:44 AM EDT, "Carlos E. R." <> wrote:
On 2014-09-22 14:49, Greg Freemyer wrote:
Well for dd anyway you can bypass the cache all together. Adding
oflag=direct should do it.
Ok, so that flag and a reasonable bs to compensate.
What about "dsync"? Or "nocache"?
Dsync just makes sure the data is written to disk asap, but it leaves a copy in cache. Not useful for writing an iso to a flash.
Ok, I see. Right.
Nocache probably works, but it first uses the kernel disk cache mechanism, then flushes it. That may or may not solve your issue.
Direct eliminates the use of the kernel cache altogether; that seems best for that use case.
Maybe I'll try them to time the results later.
I did:
Telcontar:/data/storage_b/Isos/isos_13.1 # echo ; echo "*** direct" ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=direct ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=direct ; echo ; echo "*** dsync" ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=dsync ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=dsync ; echo ; echo "*** nocache" ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=nocache ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=nocache
*** direct
43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 906.21 s, 5.0 MB/s
real 15m6.214s user 0m0.001s sys 0m5.762s
43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 880.037 s, 5.2 MB/s
real 14m40.058s user 0m0.000s sys 0m4.320s
*** dsync
43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 928.262 s, 4.9 MB/s
real 15m28.304s user 0m0.002s sys 0m8.159s
43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 923.614 s, 5.0 MB/s
real 15m23.633s user 0m0.002s sys 0m8.098s
*** nocache
43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 910.864 s, 5.0 MB/s
real 15m10.882s user 0m0.000s sys 0m7.767s
43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 916.208 s, 5.0 MB/s
real 15m16.239s user 0m0.002s sys 0m8.085s Telcontar:/data/storage_b/Isos/isos_13.1 #
They all took the same time to write to the usb stick.
There is something else, in the case of usb flash media, when writing files: there are regions, such as the filesystem metadata structures, that should better be cached, and written when needed, to minimize actual writes on the same area (even if the flash firmware distributes the actual writes).
dsync is ruled out.
Which would be better for that metadata consideration, direct or nocache?
It does not matter for this case of "burning" an iso, but it would be different when copying files. I'm considering using dd when copying big video files to my flash sticks, my laptop becomes some unresponsive when doing that.
As long as we don't know of a way to disable cache for an application or partition or disk...
Le 22/09/2014 22:42, Carlos E. R. a écrit :
They all took the same time to write to the usb stick.
does it make the computer more usable during write? (I think of my mass iso copy, I will need it when 13.2 is avail)
thanks jdd
On 2014-09-22 23:11, jdd wrote:
Le 22/09/2014 22:42, Carlos E. R. a écrit :
They all took the same time to write to the usb stick.
does it make the computer more usable during write? (I think of my mass iso copy, I will need it when 13.2 is avail)
I was using the computer all the time (6 * 15 min), and I didn't notice any slag at all. Although this is not a recent machine, it is quite powerful. Maybe the laptop would be more impacted, dunno.
Or I can try to burn more than one stick simultaneously. Maybe I can rig that...
Le 22/09/2014 23:25, Carlos E. R. a écrit :
I was using the computer all the time (6 * 15 min), and I didn't notice any slag at all. Although this is not a recent machine, it is quite powerful. Maybe the laptop would be more impacted, dunno. Or I can try to burn more than one stick simultaneously. Maybe I can rig that...
I will try and report :-)
jdd
El 22/09/14 a las #4, Carlos E. R. escribió:
SO THAT'S WHY!!
That's why writing big files, or many files, to slow usb flash devices makes the entire system crawl!
No, this is a known limitation/bug in the kernel..it has been around for quite a while.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
В Mon, 22 Sep 2014 13:18:19 +0200 (CEST) "Carlos E. R." carlos.e.r@opensuse.org пишет:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Monday, 2014-09-22 at 12:50 +0400, Andrei Borzenkov wrote:
On Mon, Sep 22, 2014 at 12:10 PM, Bernhard Voelker <> wrote:
On 09/22/2014 09:20 AM, Andrei Borzenkov wrote:
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker <> wrote:
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:
The cache seems to be filling up
That is GOOD!
It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.
Well, in that case, you'll have to wait for the data to be sync'ed to disk anyway. ;-)
Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system.
AHHHHH!!!!!!!
SO THAT'S WHY!!
That's why writing big files, or many files, to slow usb flash devices makes the entire system crawl!
So while every single application will never write faster, each application could be more responsive with smaller amount of cached data.
What Linux still lacks is rate-limiting of program writes to match output device speed.
Can we limit, externally, the amount of cache an application can use?
Memory control group looks like it can limit total amount of filesystem cache used by CG. Of course, this needs some way to start process in own control group and set its values ...
Otherwise sysctl vm.dirty_bytes or vm.dirty_ratio may help.
Unfortunately, in both cases they are just workaround with obvious drawbacks. They limit amount of dirty data globally, while in ideal world process writing to fast device should be able to dirty much more memory than process writing to slow device. I have seen discussion of patches floating around but do not know if anything was completed.
Say I want to start a large file copy to usb stick, like a dd to create an usb installation media. How do I tell it not to use a big write cache?
Or a many file copy to a usb device, using, say, mc.
Or instead of limiting the cache per application, do so per device. If the usb is on /dev/sdh, limit the cache on /dev/sdh.
Yes, I think last patch I have seen did something similar.
Is this possible at all?
Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)
-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux)
iEYEARECAAYFAlQgBXsACgkQtTMYHG2NR9UMiQCfafHNXMhYFVc/H/CyKbQtWoPU kikAn0WQHPj/3j9zS7PTDfovyZQq3VTb =lLCJ -----END PGP SIGNATURE-----
Le 22/09/2014 09:20, Andrei Borzenkov a écrit :
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker mail@bernhard-voelker.de wrote:
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:
The cache seems to be filling up
That is GOOD!
It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.
the kernel is supposed to take care of all this
jdd
On September 22, 2014 2:31:55 AM EDT, Roger Oberholtzer roger.oberholtzer@gmail.com wrote:
A year or so ago I started a discussion about a problem I had with the disk cache getting full, and then there being long delays when it was emptied. The solution at that time (openSUSE 12.1) was to periodically run:
echo 1 > /proc/sys/vm/drop_cachesAt the time, this solved everything. For some reason I cannot understand, I think this problem has returned. The cache seems to be filling up - but the command above does not help.
I looked through the archives but could not find the message from the person who suggested an X app that monitored the various cache. I found this tool very useful at the time. But I have since forgotten it. I usually record this kind of information, but not that time. Anyone recall the name of the tool?
Roger,
Here's the thread from last year.
http://opensuse.markmail.org/thread/try3zux24jzzcj2a
The others that want to understand your use case should read it. I recall it as an interesting thread.
Fyi: I love the markmail search site. Opensuse.markmail.org has the last couple years of opensuse emails in it for a lot of the main mailing lists.
I assume you figure out a way to search just that thread, but I'm not sure.
Greg
Roger,
I just read the thread from last year (it is very long).
1) the only x app mentioned is xosview
2) the drop_caches thing you imply was not used long term, just as a troubleshooting aid
3) in the end you said posix_fadvise() did the trick.
This search will find the relatively few emails in the thread that discuss posix_advise()
http://opensuse.markmail.org/search/?q=posix_fadvise+xfs
Greg
-
Andrei Borzenkov -
Bernhard Voelker -
Carlos E. R. -
Carlos E. R. -
Cristian Rodríguez -
Dylan -
Greg Freemyer -
jdd -
Roger Oberholtzer