Linda Walsh said the following on 05/07/2013 04:10 AM:
Anton Aylward wrote:
Linda Walsh said the following on 05/06/2013 09:18 PM:
(which is a darn good reason not to move /tmp to being RAM based, as it gets used for large files sometimes, and .. oops. there goes your memory!...;-)
Or NOT as the case may be.
Sorry -- its not. a 1TB file still takes 1TB out of the backing store whether it is on disk or memory.
Ah, another fringe case raises its head! To my mind a 1TB temp file tells me something is wrong, perhaps something has run away and is producing prodigious output.. Carlos mentions overloading shared resources as being unsociable (perhaps that's what cgroups are for but that's another matter - obviously you're not using cgroups to limit that runaway process that is generating the 1TB temp file) but I would think that consuming all of /tmp with a runaway -- or is it just poorly designed -- application is unsociable as well. Or are you being ridiculous just for the sake of the argument?
Secondly, Linux uses a demand paged virtual memory so you're never going to run out of memory, for whatever value of 'never' applies. And it does apply here. If that memory is needed by a process it can be paged out to swap.
--- As someone else pointed out, it's ridiculous to have much virtual memory these days -- it's a waste of disk space -- the kernel only uses it, *significantly*, in pathological situations just before the OOM-killer is invoked.
I think you have the argument backward. First, swap is there for the cases where VM does overflow. Yes, if you have a well constrained system that isn't doing things which demand memory, and you aren't running so many processes that some have to swap out so others can run, they you can in fact do without swap or end up not using swap. Good luck to you. My little 1G box from the Closet of Anxieties isn't in that fortunate situation :-( So if you have a sort program or similar that is chundling over a large data set, backwards and forward, and needs working tables, with it needs lots of memory or, it uses a temp file as intermediate store. If the latter, its nice if that intermediate store is fast. Unless you're using cgroups to limit things then this is another fringe case. We can always come up with fringe cases. I can optimise for this particular fringe case by giving it more memory; perhaps 'in core' if the design uses in core tables and sort buffers, perhaps as shm if its an old design that uses external files. Either I have enough real memory to accommodate it either way or I don't. If I don't have enough real memory then it doesn't matter whether it is running in core or with a tmpfs temp file - its going to use VM and something is going to hit swap. Any more fringe cases? Perhaps we can optimise by spending money on a high end graphics card rather than memory? Oh right, that's a fringe case as well.
System designers will all admit, that optimal behavior is decided by the application and its writer.
I'm glad you go to that, Linda. I'm glad you're saying that "optimal" depends on the "application".
Anyfile that can be changed by other processes and rewritten is not one you want to open using memory mapping. Can you imagine the performance penalty if someone or a library tried to do a memory move or 'garbage collection' on something that was mapped to disk? Ouch!
Yes, I'm glad you noticed that I mentioned SPECIFICALLY that mapping was done with executables and libraries. Oh, and yes, we do write to such files quite often. Perhaps you'd care to run a 'zypper ps' after doing a 'zypper up' to see an example of your "ouch".
I think you're worrying about something that isn't a concern.
It's a major concern. If some idiot steals /tmp for their own selfish purposes, it will kill many apps.
Oh, you mean like creating the 1TB file in /tmp you mentioned above. Yes that would be idiotic, and if you're using a shared system its idiotic not to use something like cgroups of other resource management mechanism to let someone steal that much from /tmp. Because in that and similar cases it really doesn't matter whether /tmp is a disk or a tmpfs. In that case worrying about that chose isn't a concern, like I said. Worrying about idiot or abusing users and resource exhaustion in a shared environment is your concern. Heck, maybe that idiot is creating the 1TB file in /home/idiot/Documents/ rather than /tmp, and worry about the disk/tmpfs choice for /tmp isn't a concern but resource exhaustion still is.
/var/run is for long-lived tmp files -- /tmp is for short lived tmp files... If I am copying files from one system to another and need a tmp dir -- I use the one for short-lived files.
Right. So /tmp is for the short lived intermediate and buffer files for the sort program or the phases of the compiler; and in those fringe cases I can make the case that a tmpfs implementation is quite sensible; get the program to run faster so it completes and removes them faster. When I copy files from one system to another I use rsync - that's another thread, though. Does it use a temp file?
Now can you tell me how /tmp mapped to your swap will handle a 1-2 or 2-3 TB file? Do you really have that much swap allocated?
Are we talking my dinky little 1G memory system running on a 200G drive from the Closet of Anxieties? Hopefully you have a system that is geared for the kind of applications you choose to run on it. How many people have a /tmp file on disk that that can handle your 2-3 TB file? That's the question. Because if you're in the context where such files are the norm then you're going to design a system around that use-case.
Now if we're talking about my memory starved box from The Closet Of Anxieties ... no, its still a NOT, because I'm not doing anything on such a box that involves big files. Its only a small box...
I wouldn't call a 48G server memory starved, exactly, but the case can always be made for more. yet it tops out at 192G capacity.
That doesn't come close to holding files in the TB range.
I think you're making my point for me, Linda. Recall, my first line was Or NOT as the case may be All the arguments against tmpfs has been specific use cases. If you're not designing and administering your system to address the business use cases then you re failing. Arguing about disk vs tmpfs is so missing the point. You said above
System designers will all admit, that optimal behavior is decided by the application and its writer.
The application is there to meet the business use case. If that necessitates you finding a way to accommodate 2-3 TB temporary, transient files AT AN ACCEPTABLE PERFORMANCE than so be it. If Linux can't hack it then call IBM and ask if they have a machine that can cope with the use case. There's an old Zen teaching about the finger pointing at the moon ... The Enlightened look where the finger is pointing; the novice focuses on the finger. -- Sometimes I think your train of though is carrying a shipment of toxic waste. -- Ozy & Millie, Monday, August 28, 2000 -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org