[opensuse] disk cache

Roger Oberholtzer

22 Sep 2014 22 Sep '14

06:31

A year or so ago I started a discussion about a problem I had with the disk cache getting full, and then there being long delays when it was emptied. The solution at that time (openSUSE 12.1) was to periodically run: echo 1 > /proc/sys/vm/drop_caches At the time, this solved everything. For some reason I cannot understand, I think this problem has returned. The cache seems to be filling up - but the command above does not help. I looked through the archives but could not find the message from the person who suggested an X app that monitored the various cache. I found this tool very useful at the time. But I have since forgotten it. I usually record this kind of information, but not that time. Anyone recall the name of the tool? -- Roger Oberholtzer -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Show replies by date

Bernhard Voelker

22 Sep 22 Sep

07:07

On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:

...

The cache seems to be filling up

That is GOOD! The kernel uses memory not used by applications to cache things regarding file system reads/writes. I don't see anything wrong with that. In fact, I personally don't have fast SSDs, but rather bought 20G of memory. The processes are only using ~2-8G, so everything else can be - and in fact - is used as cache. I doubt that for my case an SSD could be faster than RAM. I admit this is an extreme attitude, but it adheres to the very old yet still valid statement: "to make a UNIX system faster, add more RAM". It's not that UNIXoide and thus Linux kernels need more memory, but if it's there, then they'll use it very efficiently. Therefore: simply trust your kernel (unless you have modified the cache settings into an disadvantageous range). Regarding writing to the drop_cache pseudo file: I use it only when I want to measure the performance to simulate a "cold" cache, i.e., when the program and data needed for a process is not yet in memory but has to be read from disk. A subsequent run may be faster in magnitudes, and therefore can't be taken for comparison. Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

jdd

07:20

Le 22/09/2014 09:07, Bernhard Voelker a écrit :

...

don't have fast SSDs, but rather bought 20G of memory. The

not any mpobo accept that :-( jdd -- http://www.dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Andrei Borzenkov

07:20

On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker wrote:

...

On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:

...
The cache seems to be filling up

That is GOOD!

It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Bernhard Voelker

08:10

On 09/22/2014 09:20 AM, Andrei Borzenkov wrote:

...

On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker wrote:

...
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:

...
The cache seems to be filling up

That is GOOD!

It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.

Well, in that case, you'll have to wait for the data to be sync'ed to disk anyway. ;-) Have a nice day, Berny -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Andrei Borzenkov

08:50

On Mon, Sep 22, 2014 at 12:10 PM, Bernhard Voelker wrote:

...

On 09/22/2014 09:20 AM, Andrei Borzenkov wrote:

...
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker wrote:

...
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:

...
The cache seems to be filling up

That is GOOD!

It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.

Well, in that case, you'll have to wait for the data to be sync'ed to disk anyway. ;-)

Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system. So while every single application will never write faster, each application could be more responsive with smaller amount of cached data. What Linux still lacks is rate-limiting of program writes to match output device speed. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Dylan

09:34

On 22/09/14 09:50, Andrei Borzenkov wrote:

...

Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system.

IMO this is a result of bad design or a bug in the application causing the block - I use several 3d graphics/rendering applications which generate very large amounts of intermediate data (much more than can be written to disk at the speed it's generated) and none of them block other applications, even though they may be unresponsive themselves due to waiting for their own write-out to finish... Dx -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R.

11:22

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday, 2014-09-22 at 10:34 +0100, Dylan wrote:

...

On 22/09/14 09:50, Andrei Borzenkov wrote:

...
Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system.

IMO this is a result of bad design or a bug in the application causing the block - I use several 3d graphics/rendering applications which generate very large amounts of intermediate data (much more than can be written to disk at the speed it's generated) and none of them block other applications, even though they may be unresponsive themselves due to waiting for their own write-out to finish...

Maybe they use "sync" writes. - -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlQgBnEACgkQtTMYHG2NR9X4BgCaA1rgZlPWcX5Cf6OJI2TB/NJ/ fssAmgL7WXibvFiG2awY78RqKPYUOTW1 =Mfj1 -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R.

11:18

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Monday, 2014-09-22 at 12:50 +0400, Andrei Borzenkov wrote:

...

On Mon, Sep 22, 2014 at 12:10 PM, Bernhard Voelker <> wrote:

...
On 09/22/2014 09:20 AM, Andrei Borzenkov wrote:

...
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker <> wrote:

...
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:

...

...
...
...
...
The cache seems to be filling up

That is GOOD!

It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.

Well, in that case, you'll have to wait for the data to be sync'ed to disk anyway. ;-)

Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system.

AHHHHH!!!!!!! SO THAT'S WHY!! That's why writing big files, or many files, to slow usb flash devices makes the entire system crawl!

...

So while every single application will never write faster, each application could be more responsive with smaller amount of cached data.

What Linux still lacks is rate-limiting of program writes to match output device speed.

Can we limit, externally, the amount of cache an application can use? Say I want to start a large file copy to usb stick, like a dd to create an usb installation media. How do I tell it not to use a big write cache? Or a many file copy to a usb device, using, say, mc. Or instead of limiting the cache per application, do so per device. If the usb is on /dev/sdh, limit the cache on /dev/sdh. Is this possible at all? - -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlQgBXsACgkQtTMYHG2NR9UMiQCfafHNXMhYFVc/H/CyKbQtWoPU kikAn0WQHPj/3j9zS7PTDfovyZQq3VTb =lLCJ -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Greg Freemyer

12:49

On September 22, 2014 7:18:19 AM EDT, "Carlos E. R." wrote:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Monday, 2014-09-22 at 12:50 +0400, Andrei Borzenkov wrote:

...
On Mon, Sep 22, 2014 at 12:10 PM, Bernhard Voelker <> wrote:

...
On 09/22/2014 09:20 AM, Andrei Borzenkov wrote:

...
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker <> wrote:

...
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:

...
...
...
...
...
The cache seems to be filling up

That is GOOD!

It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.

Well, in that case, you'll have to wait for the data to be sync'ed to disk anyway. ;-)

Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system.

AHHHHH!!!!!!!

SO THAT'S WHY!!

That's why writing big files, or many files, to slow usb flash devices makes the entire system crawl!

...
So while every single application will never write faster, each application could be more responsive with smaller amount of cached data.

What Linux still lacks is rate-limiting of program writes to match output device speed.

Can we limit, externally, the amount of cache an application can use?

Say I want to start a large file copy to usb stick, like a dd to create an usb installation media. How do I tell it not to use a big write cache?

Well for dd anyway you can bypass the cache all together. Adding oflag=direct should do it.

...

Or a many file copy to a usb device, using, say, mc.

Or instead of limiting the cache per application, do so per device. If the usb is on /dev/sdh, limit the cache on /dev/sdh.

Is this possible at all?

No idea on your other questions.

...

- -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux)

iEYEARECAAYFAlQgBXsACgkQtTMYHG2NR9UMiQCfafHNXMhYFVc/H/CyKbQtWoPU kikAn0WQHPj/3j9zS7PTDfovyZQq3VTb =lLCJ -----END PGP SIGNATURE-----

-- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R.

13:18

On 2014-09-22 14:49, Greg Freemyer wrote:

...

...
Well for dd anyway you can bypass the cache all together. Adding oflag=direct should do it.

Ok, so that flag and a reasonable bs to compensate. What about "dsync"? Or "nocache"? Maybe I'll try them to time the results later.

...

Or a many file copy to a usb device, using, say, mc.

Or instead of limiting the cache per application, do so per device. If the usb is on /dev/sdh, limit the cache on /dev/sdh.

Is this possible at all?

...
No idea on your other questions.

Pity... -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

Greg Freemyer

14:09

On September 22, 2014 9:18:44 AM EDT, "Carlos E. R." wrote:

...

On 2014-09-22 14:49, Greg Freemyer wrote:

...
...
Well for dd anyway you can bypass the cache all together. Adding oflag=direct should do it.

Ok, so that flag and a reasonable bs to compensate.

What about "dsync"? Or "nocache"?

Dsync just makes sure the data is written to disk asap, but it leaves a copy in cache. Not useful for writing an iso to a flash. Nocache probably works, but it first uses the kernel disk cache mechanism, then flushes it. That may or may not solve your issue. Direct eliminates the use of the kernel cache altogether; that seems best for that use case.

...

Maybe I'll try them to time the results later.

-- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R.

20:42

On 2014-09-22 16:09, Greg Freemyer wrote:

...

On September 22, 2014 9:18:44 AM EDT, "Carlos E. R." <> wrote:

...
On 2014-09-22 14:49, Greg Freemyer wrote:

...
...
Well for dd anyway you can bypass the cache all together. Adding oflag=direct should do it.

Ok, so that flag and a reasonable bs to compensate.

What about "dsync"? Or "nocache"?

Dsync just makes sure the data is written to disk asap, but it leaves a copy in cache. Not useful for writing an iso to a flash.

Ok, I see. Right.

...

Nocache probably works, but it first uses the kernel disk cache mechanism, then flushes it. That may or may not solve your issue.

Direct eliminates the use of the kernel cache altogether; that seems best for that use case.

...
Maybe I'll try them to time the results later.

I did: Telcontar:/data/storage_b/Isos/isos_13.1 # echo ; echo "*** direct" ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=direct ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=direct ; echo ; echo "*** dsync" ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=dsync ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=dsync ; echo ; echo "*** nocache" ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=nocache ; echo ; time dd if=openSUSE-13.1-DVD-x86_64.iso of=/dev/sde bs=100M oflag=nocache *** direct 43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 906.21 s, 5.0 MB/s real 15m6.214s user 0m0.001s sys 0m5.762s 43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 880.037 s, 5.2 MB/s real 14m40.058s user 0m0.000s sys 0m4.320s *** dsync 43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 928.262 s, 4.9 MB/s real 15m28.304s user 0m0.002s sys 0m8.159s 43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 923.614 s, 5.0 MB/s real 15m23.633s user 0m0.002s sys 0m8.098s *** nocache 43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 910.864 s, 5.0 MB/s real 15m10.882s user 0m0.000s sys 0m7.767s 43+1 records in 43+1 records out 4572839936 bytes (4.6 GB) copied, 916.208 s, 5.0 MB/s real 15m16.239s user 0m0.002s sys 0m8.085s Telcontar:/data/storage_b/Isos/isos_13.1 # They all took the same time to write to the usb stick. There is something else, in the case of usb flash media, when writing files: there are regions, such as the filesystem metadata structures, that should better be cached, and written when needed, to minimize actual writes on the same area (even if the flash firmware distributes the actual writes). dsync is ruled out. Which would be better for that metadata consideration, direct or nocache? It does not matter for this case of "burning" an iso, but it would be different when copying files. I'm considering using dd when copying big video files to my flash sticks, my laptop becomes some unresponsive when doing that. As long as we don't know of a way to disable cache for an application or partition or disk... -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

jdd

21:11

Le 22/09/2014 22:42, Carlos E. R. a écrit :

...

They all took the same time to write to the usb stick.

does it make the computer more usable during write? (I think of my mass iso copy, I will need it when 13.2 is avail) thanks jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Carlos E. R.

21:25

On 2014-09-22 23:11, jdd wrote:

...

Le 22/09/2014 22:42, Carlos E. R. a écrit :

...
They all took the same time to write to the usb stick.

does it make the computer more usable during write? (I think of my mass iso copy, I will need it when 13.2 is avail)

I was using the computer all the time (6 * 15 min), and I didn't notice any slag at all. Although this is not a recent machine, it is quite powerful. Maybe the laptop would be more impacted, dunno. Or I can try to burn more than one stick simultaneously. Maybe I can rig that... -- Cheers / Saludos, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

jdd

23 Sep 23 Sep

06:14

Le 22/09/2014 23:25, Carlos E. R. a écrit :

...

I was using the computer all the time (6 * 15 min), and I didn't notice any slag at all. Although this is not a recent machine, it is quite powerful. Maybe the laptop would be more impacted, dunno. Or I can try to burn more than one stick simultaneously. Maybe I can rig that... I will try and report :-)

jdd -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Cristian Rodríguez

22 Sep 22 Sep

21:26

El 22/09/14 a las #4, Carlos E. R. escribió:

...

SO THAT'S WHY!!

That's why writing big files, or many files, to slow usb flash devices makes the entire system crawl!

No, this is a known limitation/bug in the kernel..it has been around for quite a while. -- Cristian "I don't know the key to success, but the key to failure is trying to please everybody." -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Andrei Borzenkov

23 Sep 23 Sep

03:03

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 В Mon, 22 Sep 2014 13:18:19 +0200 (CEST) "Carlos E. R." пишет:

...

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

On Monday, 2014-09-22 at 12:50 +0400, Andrei Borzenkov wrote:

...
On Mon, Sep 22, 2014 at 12:10 PM, Bernhard Voelker <> wrote:

...
On 09/22/2014 09:20 AM, Andrei Borzenkov wrote:

...
On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker <> wrote:

...
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:

...
...
...
...
...
The cache seems to be filling up

That is GOOD!

It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.

Well, in that case, you'll have to wait for the data to be sync'ed to disk anyway. ;-)

Yes, but currently single application is able to fill in RAM with unwritten data, so that the whole system (every other application) has to wait while data is drained. And that blocks not only write-out, but also reading of data. Resulting in very long delays and non-responsive system.

AHHHHH!!!!!!!

SO THAT'S WHY!!

That's why writing big files, or many files, to slow usb flash devices makes the entire system crawl!

...
So while every single application will never write faster, each application could be more responsive with smaller amount of cached data.

What Linux still lacks is rate-limiting of program writes to match output device speed.

Can we limit, externally, the amount of cache an application can use?

Memory control group looks like it can limit total amount of filesystem cache used by CG. Of course, this needs some way to start process in own control group and set its values ... Otherwise sysctl vm.dirty_bytes or vm.dirty_ratio may help. Unfortunately, in both cases they are just workaround with obvious drawbacks. They limit amount of dirty data globally, while in ideal world process writing to fast device should be able to dirty much more memory than process writing to slow device. I have seen discussion of patches floating around but do not know if anything was completed.

...

Say I want to start a large file copy to usb stick, like a dd to create an usb installation media. How do I tell it not to use a big write cache?

Or a many file copy to a usb device, using, say, mc.

Or instead of limiting the cache per application, do so per device. If the usb is on /dev/sdh, limit the cache on /dev/sdh.

Yes, I think last patch I have seen did something similar.

...

Is this possible at all?

- -- Cheers, Carlos E. R. (from 13.1 x86_64 "Bottle" at Telcontar)

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux)

iEYEARECAAYFAlQgBXsACgkQtTMYHG2NR9UMiQCfafHNXMhYFVc/H/CyKbQtWoPU kikAn0WQHPj/3j9zS7PTDfovyZQq3VTb =lLCJ -----END PGP SIGNATURE-----

-----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlQg4wcACgkQR6LMutpd94yjLQCffUyXT7siGYwJC3GHzbtfgXNv t2QAoIA+01ETxmGMGw6t9yd7PC+a6fWf =gecl -----END PGP SIGNATURE-----

jdd

22 Sep 22 Sep

09:03

Le 22/09/2014 09:20, Andrei Borzenkov a écrit :

...

On Mon, Sep 22, 2014 at 11:07 AM, Bernhard Voelker wrote:

...
On 09/22/2014 08:31 AM, Roger Oberholtzer wrote:

...
The cache seems to be filling up

That is GOOD!

It depends. It is good when cache is filled up with data waiting to be read. It is bad when it is filled up with data waiting to be written out to slow device.

the kernel is supposed to take care of all this jdd -- http://www.dodin.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Greg Freemyer

11:38

On September 22, 2014 2:31:55 AM EDT, Roger Oberholtzer wrote:

...

A year or so ago I started a discussion about a problem I had with the disk cache getting full, and then there being long delays when it was emptied. The solution at that time (openSUSE 12.1) was to periodically run:

echo 1 > /proc/sys/vm/drop_caches

At the time, this solved everything. For some reason I cannot understand, I think this problem has returned. The cache seems to be filling up - but the command above does not help.

I looked through the archives but could not find the message from the person who suggested an X app that monitored the various cache. I found this tool very useful at the time. But I have since forgotten it. I usually record this kind of information, but not that time. Anyone recall the name of the tool?

Roger, Here's the thread from last year. http://opensuse.markmail.org/thread/try3zux24jzzcj2a The others that want to understand your use case should read it. I recall it as an interesting thread. Fyi: I love the markmail search site. Opensuse.markmail.org has the last couple years of opensuse emails in it for a lot of the main mailing lists. I assume you figure out a way to search just that thread, but I'm not sure. Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

Greg Freemyer

12:45

Roger, I just read the thread from last year (it is very long). 1) the only x app mentioned is xosview 2) the drop_caches thing you imply was not used long term, just as a troubleshooting aid 3) in the end you said posix_fadvise() did the trick. This search will find the relatively few emails in the thread that discuss posix_advise() http://opensuse.markmail.org/search/?q=posix_fadvise+xfs Greg -- Sent from my Android phone with K-9 Mail. Please excuse my brevity. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org

3515

Age (days ago)

3516

Last active (days ago)

List overview

Download

20 comments

9 participants

participants (9)

Andrei Borzenkov
Bernhard Voelker
Carlos E. R.
Carlos E. R.
Cristian Rodríguez
Dylan
Greg Freemyer
jdd
Roger Oberholtzer