I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder. Does anybody know of a way to do that? TIA! --doug
File a feature request at bugs.kde.org. :) On Monday 09 August 2004 15:15, Doug McGarrett wrote:
I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
Does anybody know of a way to do that? TIA! --doug
* Doug McGarrett
I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
procmail man procmail man procmailex man procmailsc man procmailrc mailto:procmail-request@lists.RWTH-Aachen.DE?subject=subscribe -- Patrick Shanahan Registered Linux User #207535 http://wahoo.no-ip.org @ http://counter.li.org HOG # US1244711 Photo Album: http://wahoo.no-ip.org/photos
Patrick, Doug, On Monday 09 August 2004 13:31, Patrick Shanahan wrote:
* Doug McGarrett
[08-09-04 15:13]: I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
procmail
man procmail man procmailex man procmailsc man procmailrc
Indeed! As I suspected, it's all been done already. Here are a couple of resources from which one can find all sorts of procmail tutorials, code and recipies: - http://rhols66.adsl.netsonic.fi/era/procmail/links.html - http://www.uwasa.fi/~ts/info/proctips.html Procmail is very ... fancy, and there's a commensurate amount of coverage on the Web.
mailto:procmail-request@lists.RWTH-Aachen.DE?subject=subscribe
-- Patrick Shanahan
Randall Schulz
I don't want to have to take a university degree in mail programs. Also, I want a GUI program, not something I have to run as a terminal routine. It appears, from first looks, that procmail would require both. If I'm all wet, please advise. For those who came upon Unix way back when, that may be OK, but I'm a relative newbie, and I really don't wish to start "way back when" and catch up, somehow. It shouldn't be necessary in this modern day to do so. I don't write Perl. _ Years_ ago I wrote some Basic and some Pascal. That's about it. (And the damned Basic I wrote won't run on Linux!) I guess I'll just have to do it the hard way and move the files by hand. But I will take someone's advice to send a message to bugs.kde.org. --doug On Monday 09 August 2004 18:05, Randall R Schulz wrote:
Patrick, Doug,
On Monday 09 August 2004 13:31, Patrick Shanahan wrote:
* Doug McGarrett
[08-09-04 15:13]: I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
procmail
man procmail man procmailex man procmailsc man procmailrc
Indeed! As I suspected, it's all been done already.
Here are a couple of resources from which one can find all sorts of procmail tutorials, code and recipies:
- http://rhols66.adsl.netsonic.fi/era/procmail/links.html - http://www.uwasa.fi/~ts/info/proctips.html
Procmail is very ... fancy, and there's a commensurate amount of coverage on the Web.
mailto:procmail-request@lists.RWTH-Aachen.DE?subject=subscribe
-- Patrick Shanahan
Randall Schulz
On Monday 09 August 2004 08:25 pm, Doug McGarrett wrote:
I don't want to have to take a university degree in mail programs. Also, I want a GUI program, not something I have to run as a terminal routine. It appears, from first looks, that procmail would require both. If I'm all wet, please advise.
You're all wet. Procmail is one flexible piece of software.
For those who came upon Unix way back when, that may be OK, but I'm a relative newbie, and I really don't wish to start "way back when" and catch up, somehow. It shouldn't be necessary in this modern day to do so.
Well, just sit back with windows then and click away.
I don't write Perl. _ Years_ ago I wrote some Basic and some Pascal. That's about it. (And the damned Basic I wrote won't run on Linux!)
Not needed.
I guess I'll just have to do it the hard way and move the files by hand. But I will take someone's advice to send a message to bugs.kde.org.
I don't think your attitude is going to get you a good answer to your problem.
--doug
On Monday 09 August 2004 18:05, Randall R Schulz wrote:
Patrick, Doug,
On Monday 09 August 2004 13:31, Patrick Shanahan wrote:
* Doug McGarrett
[08-09-04 15:13]: I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
procmail
man procmail man procmailex man procmailsc man procmailrc
Indeed! As I suspected, it's all been done already.
Here are a couple of resources from which one can find all sorts of procmail tutorials, code and recipies:
- http://rhols66.adsl.netsonic.fi/era/procmail/links.html - http://www.uwasa.fi/~ts/info/proctips.html
Procmail is very ... fancy, and there's a commensurate amount of coverage on the Web.
mailto:procmail-request@lists.RWTH-Aachen.DE?subject=subscribe
-- Patrick Shanahan
Randall Schulz
-- +----------------------------------------------------------------------------+ + Bruce S. Marshall bmarsh@bmarsh.com Bellaire, MI 08/09/04 20:59 + +----------------------------------------------------------------------------+ "You might be a high-tech Red-neck if: you need a checklist to turn on the TV"
* Doug McGarrett
[08-09-04 15:13]: I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder
<LOTS DELETED> I'm not a kmail afficionado, but I've used evolution, mozilla mail and currently it's offshoot thunderbird. All I have to do is Right-Click on the attachment and do a "Save As", it also has excellent Spam filtering as does recent versions of mozilla mail, that's one of the chief reasons why I chose it. In evolution it's exactly the same. If you handle large numbers of such emails a day, I could see how it would be a nuisance to have to do that for each email. Eudora would probably run under dosemu or wine, it's so long back I can't remember if it's a DOS or Windows program. I think I would be more peeved if people were bombarding me with attachments, I'd rather have them point me to a URL or some such in case I was interested in what they sent. I did get peeved when people sent me a .doc file as a one-off with details that warranted no more than an email, well, people do strange things and we suffer. Regards Sid. -- Sid Boyce .... Hamradio G3VBV and keen Flyer =====LINUX ONLY USED HERE=====
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mon August 9 2004 9:59 pm, Sid Boyce wrote:
* Doug McGarrett
[08-09-04 15:13]: I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder
<LOTS DELETED> I'm not a kmail afficionado, but I've used evolution, mozilla mail and currently it's offshoot thunderbird. All I have to do is Right-Click on the attachment and do a "Save As",
- ------------SNIP_________ Your reply got me thinking, I do a lot of right clicking (filing messages for future reference) and agree it is a real pain. I don't know of a way to automatically save an attachment, but a button can be added to the toolbar that (I haven't tried it) should allow easier saving of attachments. Check the Kmail->Settings->Configure Toolbars menu and you will see a "Save Attachments" button is available. Hope this is useful. - -- dh -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.2 (GNU/Linux) iD8DBQFBGOplBwgxlylUsJARAsi/AJ0SSEaQ6w+nf6iZeCKAK1HUbZiA9QCbBGho jLwWcB7BUQbHoLzUP9DKF8g= =OQX8 -----END PGP SIGNATURE-----
dh wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Mon August 9 2004 9:59 pm, Sid Boyce wrote:
* Doug McGarrett
[08-09-04 15:13]: >I like KMail, and have had no problems with it. (I'm not doing >complicated filters, etc.) And thank heavens it _doesn't_ look >like a Microsoft routine! But I can't find a way to implement >a feature that Eudora has: a way to automatically copy an >attachment on an incoming message to a folder. It would be >even cooler if I could specify the filename extension of the >incoming attachment and only copy that file to a specified >folder. i.e., if the incoming message was a .doc file, it would >automatically copy it to the >/home/doug/Documents folder > > <LOTS DELETED> I'm not a kmail afficionado, but I've used evolution, mozilla mail and currently it's offshoot thunderbird. All I have to do is Right-Click on the attachment and do a "Save As",
- ------------SNIP_________
Your reply got me thinking, I do a lot of right clicking (filing messages for future reference) and agree it is a real pain.
I don't know of a way to automatically save an attachment, but a button can be added to the toolbar that (I haven't tried it) should allow easier saving of attachments.
Check the Kmail->Settings->Configure Toolbars menu and you will see a "Save Attachments" button is available.
Hope this is useful.
- --
So it seems we are getting close to "If it can't be done in Linux, it can't be done". Regards Sid. -- Sid Boyce .... Hamradio G3VBV and keen Flyer =====LINUX ONLY USED HERE=====
Sid, On Tuesday 10 August 2004 08:37, Sid Boyce wrote:
...
So it seems we are getting close to "If it can't be done in Linux, it can't be done".
Regards Sid.
-- Sid Boyce
Everything is SMOP: A Simple Matter of Programming. But that's small comfort to the non-programmer, I suppose. RRS
* Doug McGarrett
I don't want to have to take a university degree in mail programs. Also, I want a GUI program, not something I have to run as a terminal routine. It appears, from first looks, that procmail would require both. If I'm all wet, please advise.
consider yourself advised.
For those who came upon Unix way back when, that may be OK, but I'm a relative newbie, and I really don't wish to start "way back when" and catch up, somehow. It shouldn't be necessary in this modern day to do so.
and it is not.
I don't write Perl. _ Years_ ago I wrote some Basic and some Pascal. That's about it. (And the damned Basic I wrote won't run on Linux!)
linux has *much* better tools than basic, but perl is good.
I guess I'll just have to do it the hard way and move the files by hand. But I will take someone's advice to send a message to bugs.kde.org.
If the desire/need is great enough you will expend the necessary energy. It is easy to write recipies for procmail and the site referenced is a great starting point. Linux *is* about choice, your choice. The procmail quick start: http://www.ii.com/internet/robots/procmail/qs/ Please, trim your quotes. references: Please read the FAQs: suse-linux-e-faq@suse.com The Etiquette of the SuSE-Linux Mailing-List. http://www.dhaller.de/linux/etiquette-e.html Netiquette Guidelines, rfc 1855 http://www.dtcc.edu/cs/rfc1855.html http://www.faqs.org/rfcs/rfc1855.html http://www.gweep.ca/~edmonds/usenet/ml-etiquette.html http://www.netmeister.org/news/learn2quote.html http://herbert.the-little-red-haired-girl.org/en/quoting/ Accepted proper email writing/quoting/etc. http://www.camtp.uni-mb.si/books/jargon/html/email-style.html http://www.catb.org/~esr/jargon/html/M/McQuary-limit.html http://www.catb.org/~esr/jargon/html/S/signal-to-noise-ratio.html -- Patrick Shanahan Registered Linux User #207535 http://wahoo.no-ip.org @ http://counter.li.org HOG # US1244711 Photo Album: http://wahoo.no-ip.org/photos
Procmail is not a mail reading program, it's a processor. You might instead want to look at something like mimedefang. These things work with your MTA (such as sendmail) rather than your mail reader. The mail reader really should not be "doing things" to your emails at all. The things should be done as the mails are received to your system. I would be really surprised if there were not some GUI to Procmail, though I have always found it easier to edit the .procmailrc directly. On Monday 09 August 2004 19:25, Doug McGarrett wrote:
I don't want to have to take a university degree in mail programs. Also, I want a GUI program, not something I have to run as a terminal routine.
Misty, On Tuesday 10 August 2004 07:38, Misty Stanley-Jones wrote:
Procmail is not a mail reading program, it's a processor. You might instead want to look at something like mimedefang. These things work with your MTA (such as sendmail) rather than your mail reader. The mail reader really should not be "doing things" to your emails at all. The things should be done as the mails are received to your system.
I cannot really agree with that. Treatment of the messages themselves are, from the user's perspective, fully at the discretion of the end users. There are many things I like to do to messages, and KMail is great at handling them: - Remove my ISP's spam headers from false positives - Strip list name tags (the only thing worse than Reply-To munging) - Normalize Re: (remove duplicates, mostly) - Strip References headers from posting made with the Reply command that actually start new topic threads KMail's ability to add, rewrite and remove headers is very powerful. Its ability to pass messages to or through external programs essentially removes all limits from how mail can be handled and provides the basic hooks for using something like procmail or other mail processing tools. I agree with the OP about attachments: Eudora's way of handling them (extracting them upon receipt and storing them separately) is preferable, but that's just a personal preference, presumably. Webmin + Usermin (http://www.webmin.com/ -- the SuSE 9.1 distribution includes version 1.130 while the webmin.com site has version 1.150) can do some procmail configuration, apparently, but it appears to be as daunting to learn as procmail, if not more. However, there are lots and LOTS of pretty pictures... And it's a Web / browser interface, not a classic GUI, if that matters.
I would be really surprised if there were not some GUI to Procmail, though I have always found it easier to edit the .procmailrc directly.
On Monday 09 August 2004 19:25, Doug McGarrett wrote:
I don't want to have to take a university degree in mail programs. Also, I want a GUI program, not something I have to run as a terminal routine.
Randall Schulz
On Monday 09 August 2004 21:15, Doug McGarrett wrote:
I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
Does anybody know of a way to do that? TIA! --doug
There's probably a command line tool you could use the existing filter with. Not done it, but there should be something out there. -- Steve Boddy
On Monday 09 August 2004 21:32, Stephen Boddy wrote:
On Monday 09 August 2004 21:15, Doug McGarrett wrote:
I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
Does anybody know of a way to do that? TIA! --doug
There's probably a command line tool you could use the existing filter with. Not done it, but there should be something out there. -- Steve Boddy
To answer myself; from http://docs.kde.org/en/3.2/kdepim/kmail/filters.html: ---8<--- execute command This will execute a program, but will not modify the message. Specify the full path to the program you want to execute. KMail will block until the program returns. You can feed the program with the parts of the mail: %0, %1, etc. will stand for files representing the message parts. For common messages %0 is the text, %1 the first attachment and so on. Additionally, the whole message is fed into the program's stdin. In addition, every occurrence of %{foo} is replaced by the content of the foo header. Warning This currently only works if the message has at least one attachment. No, not even %0 will work in the general case! Tip You can enter arbitrarily complex shell commands here, since KMail uses a sub shell to execute the command line. Therefore, even this command will work (within its limits): uudecode -o $(mktemp kmail-uudecoded.XXXXXX) && echo $'\a' ---8<--- That should be enough to get you going. If not there's a good discussion and references to tools here: http://zgp.org/linux-elitists/20010902130403.S32544@zork.net.html Finally, I recommend taking a look at Python email library. It would be perhaps an few hours work to get something usable, even learning as you go. Add a couple of options for mimetype and destination, and it'll totally meet your requirements. HTH -- Steve Boddy
Doug, I've wanted such a feature, too (I'm also a former Eudora user, and about the only thing I could ever find to criticize about it is its lack of topic threading, though that's a big omission). In particular, I don't like my mailboxes burdened with encoded attachment data (I use mbox format, not maildir, which strikes me as an insanely bloated way to store email). When I was setting up my Linux as a primary desktop, the absence of a direct way to do this was enough to make me shrug and decide to live with what KMail provides directly. However, your message has led me to reconsider this. KMail does have the necessary capabilities to implement an attachment separator. Specifically, the filter action "pipe through" allows both extraction of attachments (decoded or raw) as well as removal of attachment data. For all I know, someone has done this already, but if not, it's a relatively straightforward programming task. It could readily be done in Perl, e.g., or even as a shell script, given the necessary MIME commands, as well as in many other languages available in a Linux environment. I can't drop what I'm doing now to try this, but I may take it up over the next days or weeks. If I don't hear of an alternative or of someone else working on this task, I'll report back here on what I come up with. If I produce something, I'll share it. ... Does the KDE project have a contributed software section? Randall Schulz On Monday 09 August 2004 13:15, Doug McGarrett wrote:
I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
Does anybody know of a way to do that? TIA! --doug
On Monday 09 August 2004 22:36, Randall R Schulz wrote:
I use mbox format, not maildir, which strikes me as an insanely bloated way to store email
This statement requires elucidation mbox and maildir store the exact same information, except that mbox *adds* an additional "From" header to signify envelope sender. So mbox actually stores more data. Assuming you are using a sane file system - like reiserfs for example - that is capable of filling up disk completely and not leave little empty "tails" for half filled blocks, á la ext2/3, it will consume *less* disk space. So how is it bloated It is also faster and safer (although I seem to recall some problems with storing it on an NFS share), so I don't really see your objection to it. Would you care to explain it a bit further?
Does the KDE project have a contributed software section?
kdeextragear or kdenonbeta are the CVS modules for non-official kde stuff (meaning stuff not really part of kde but still stored in the kde cvs repository)
Anders, On Monday 09 August 2004 13:58, Anders Johansson wrote:
On Monday 09 August 2004 22:36, Randall R Schulz wrote:
I use mbox format, not maildir, which strikes me as an insanely bloated way to store email
This statement requires elucidation
mbox and maildir store the exact same information, except that mbox *adds* an additional "From" header to signify envelope sender. So mbox actually stores more data. Assuming you are using a sane file system - like reiserfs for example - that is capable of filling up disk completely and not leave little empty "tails" for half filled blocks, á la ext2/3, it will consume *less* disk space. So how is it bloated?
I may be behind the times as regards Linux / Unix file systems, but I am under the impression that mass storage is allocated with granularities not less than a sector size and almost always a multiple thereof. Under such allocation schemes and a typical allocation unit size of 4096 (eight 512-byte sectors), a message whose entire contents (body, headers and mail client overhead / added headers) were, say, 2000 bytes, then 2096 bytes would be wasted if this message occupied a file of its own. Are you telling me that there are file systems under Linux that eliminate internal fragmentation? If so, all I can say is that I'm impressed. But I also have to wonder what happens under such a scheme when a file is to grow and it ends mid-sector (or mid-allocation unit) and another file shares that sector or allocation unit. It seems that a hell of a lot of shuffling would ensue. Either that or disk allocation occurs at sub-sector granularity. I'm using XFS for my native file systems (and I mount some "legacy" Windows partitions that are NTFS format).
It is also faster and safer (although I seem to recall some problems with storing it on an NFS share), so I don't really see your objection to it. Would you care to explain it a bit further?
It's only as I say above, extremely wasteful of space when disk allocation works in the way I understand it to.
Does the KDE project have a contributed software section?
kdeextragear or kdenonbeta are the CVS modules for non-official kde stuff (meaning stuff not really part of kde but still stored in the kde cvs repository)
Thanks. Randall Schulz
On Monday 09 August 2004 23:16, Randall R Schulz wrote:
Anders,
On Monday 09 August 2004 13:58, Anders Johansson wrote:
On Monday 09 August 2004 22:36, Randall R Schulz wrote:
I use mbox format, not maildir, which strikes me as an insanely bloated way to store email
This statement requires elucidation
mbox and maildir store the exact same information, except that mbox *adds* an additional "From" header to signify envelope sender. So mbox actually stores more data. Assuming you are using a sane file system - like reiserfs for example - that is capable of filling up disk completely and not leave little empty "tails" for half filled blocks, á la ext2/3, it will consume *less* disk space. So how is it bloated?
I may be behind the times as regards Linux / Unix file systems, but I am under the impression that mass storage is allocated with granularities not less than a sector size and almost always a multiple thereof.
Under such allocation schemes and a typical allocation unit size of 4096 (eight 512-byte sectors), a message whose entire contents (body, headers and mail client overhead / added headers) were, say, 2000 bytes, then 2096 bytes would be wasted if this message occupied a file of its own.
Are you telling me that there are file systems under Linux that eliminate internal fragmentation?
Yep
If so, all I can say is that I'm impressed. But I also have to wonder what happens under such a scheme when a file is to grow and it ends mid-sector (or mid-allocation unit) and another file shares that sector or allocation unit. It seems that a hell of a lot of shuffling would ensue.
Why? it would require at most blocksize-1 bytes be moved some place else and then the file can grow as under any other file system, and that is assuming that the tail of another file has had time to move in there. Have a look at the whitepapers at www.namesys.com for the technical specifics (or the reiserfs source, if you are that way inclined :)
Shuffling does not ensue as you say. The filesystem (most modern ones at least) are perfectly capable of dealing with non-contiguous files. Dynamic filesystems such as Reiser, VxFS (probably the free version too), and I do believe ext3 are perfectly capable of defragmenting themselves as files grow. You are also perfectly allowed to adjust the block size of your filesystem if you are very paranoid about running out of inodes due to many small files. Misty
I may be behind the times as regards Linux / Unix file systems, but I am under the impression that mass storage is allocated with granularities not less than a sector size and almost always a multiple thereof.
Under such allocation schemes and a typical allocation unit size of 4096 (eight 512-byte sectors), a message whose entire contents (body, headers and mail client overhead / added headers) were, say, 2000 bytes, then 2096 bytes would be wasted if this message occupied a file of its own.
Why? it would require at most blocksize-1 bytes be moved some place else and then the file can grow as under any other file system, and that is assuming that the tail of another file has had time to move in there. Have a look at the whitepapers at www.namesys.com for the technical specifics (or the reiserfs source, if you are that way inclined :)
On Tuesday 10 August 2004 16:35, Misty Stanley-Jones wrote:
Shuffling does not ensue as you say. The filesystem (most modern ones at least) are perfectly capable of dealing with non-contiguous files. Dynamic filesystems such as Reiser, VxFS (probably the free version too), and I do believe ext3 are perfectly capable of defragmenting themselves as files grow. You are also perfectly allowed to adjust the block size of your filesystem if you are very paranoid about running out of inodes due to many small files.
While this is true, it wasn't the question at issue. If you have a number of blocks B[1...1000], and file 1 uses blocks B[1...5] and file 2 uses blocks B[6...10] then to grow file 1 you can just start using block B[11] with - as you point out - no need to shuffle anything. As most (if not all) file systems, including FAT, is basically a linked list of file system blocks, all systems should be able to handle this. The problem here arises because reiserfs stores tails of multiple files in a single block. It may be my ignorance, but I thought reiserfs was alone in doing this. In this scheme B[19] could have the last 500 bytes of file 1 *and* the last 1003 bytes of file 2, instead of storing those two "tails" in separate blocks, wasting blocksize-500+blocksize-1003 bytes. Now, I haven't looked at the implementations of this in reiser , but if you grow file 1, while not absolutely necessary, it seems on first inspection like a good idea to move those 500 bytes to a fresh block and start from there. My point about the shuffling then was that all you would ever have to move around are blocksize-1 I agree that the fragmentation that is caused by growing files is handled so well by the file systems that it hardly ever becomes significant for performance, except in the most extreme cases
Misty, On Tuesday 10 August 2004 07:35, Misty Stanley-Jones wrote:
Shuffling does not ensue as you say. The filesystem (most modern ones at least) are perfectly capable of dealing with non-contiguous files. Dynamic filesystems such as Reiser, VxFS (probably the free version too), and I do believe ext3 are perfectly capable of defragmenting themselves as files grow. You are also perfectly allowed to adjust the block size of your filesystem if you are very paranoid about running out of inodes due to many small files.
But if you don't shuffle to minimize fragmentation as files that share sectors or allocation units grow, then access times would suffer over time, no? I can certainly see the possibility of addressing all the issues of of sub-sector allocation, but I was just surprised that it had been done. As I said, at one time I knew the Unix kernel quite well, but that was a long time ago. Clearly lots of progress has been made. Kudos to the folks that did all this and made it reliable! I have been noticing that my latest Linux install, for which I chose XFS, seems to be very easy on the disk drives, in that they make much less seek noise than I'm used to under heavy loads.
Misty
Randall Schulz
On Tuesday 10 August 2004 10:38, Randall R Schulz wrote:
I have been noticing that my latest Linux install, for which I chose XFS, seems to be very easy on the disk drives, in that they make much less seek noise than I'm used to under heavy loads.
Maybe it is my quirky sense of humor, but wouldn't it be funny if it's just hard drive technology becoming quieter? :)
Randall Schulz
On Tuesday 10 August 2004 12:46, Misty Stanley-Jones wrote:
On Tuesday 10 August 2004 10:38, Randall R Schulz wrote:
I have been noticing that my latest Linux install, for which I chose XFS, seems to be very easy on the disk drives, in that they make much less seek noise than I'm used to under heavy loads.
Maybe it is my quirky sense of humor, but wouldn't it be funny if it's just hard drive technology becoming quieter? :)
Randall Schulz
I think he's right. I see fairly frequent access to the drive (using Reiser FS) but I never hear it at all. And I don't even have the cover on the computer! --doug
On Tuesday 10 August 2004 21:43, Doug McGarrett wrote:
On Tuesday 10 August 2004 12:46, Misty Stanley-Jones wrote:
On Tuesday 10 August 2004 10:38, Randall R Schulz wrote:
I have been noticing that my latest Linux install, for which I chose XFS, seems to be very easy on the disk drives, in that they make much less seek noise than I'm used to under heavy loads.
Maybe it is my quirky sense of humor, but wouldn't it be funny if it's just hard drive technology becoming quieter? :)
Randall Schulz
I think he's right. I see fairly frequent access to the drive (using Reiser FS) but I never hear it at all. And I don't even have the cover on the computer! --doug
XFS is known for its aggressive memory caching. It makes for good performance since it flushes to disk relatively rarely, but on the other hand it means that in a crash you stand to lose more data compared to a file system that flushes more often. Every silver lining has a cloud
Anders, On Monday 09 August 2004 13:58, Anders Johansson wrote:
On Monday 09 August 2004 22:36, Randall R Schulz wrote:
I use mbox format, not maildir, which strikes me as an insanely bloated way to store email
This statement requires elucidation
mbox and maildir store the exact same information, except that mbox *adds* an additional "From" header to signify envelope sender. So mbox actually stores more data. Assuming you are using a sane file system - like reiserfs for example - that is capable of filling up disk completely and not leave little empty "tails" for half filled blocks, á la ext2/3, it will consume *less* disk space. So how is it bloated
...
I did a little experiment to gauge the internal fragmentation overhead of "mbox" vs. "maildir" storage in KMail when using an XFS file system. I had four small- to medium-sized mailboxes to import. I imported them into an mbox folder (the import function seems to ignore the global default folder type when creating a folder for import) and then copied the contents of those four folders to maildir folders. Then I used the "du" command to compute the amount of mass storage occupied by each of the eight mailboxes (four files for the mbox and four directories for the maildir). The difference is significant (output reordered to ease comparison): % du -s Mail/{,MD-}JayToRandy200? 756 Mail/JayToRandy2001 1052 Mail/MD-JayToRandy2001 7352 Mail/JayToRandy2002 9024 Mail/MD-JayToRandy2002 8076 Mail/JayToRandy2003 10832 Mail/MD-JayToRandy2003 7460 Mail/JayToRandy2004 8796 Mail/MD-JayToRandy2004 The folders' message counts are: ...2001: 141 messages ...2002: 841 messages ...2003: 1192 messages ...2004: 684 messages The percentage increase of maildir storage of mbox is: ...2001: %39 ...2002: %23 ...2003: %34 ...4004: %18 _That_'s why I prefer mbox to maildir! Randall Schulz
Randall wrote regarding 'Re: [SLE] KMail question' on Wed, Aug 11 at 20:14: [... comparing size of maildir to mbox...]
...2001: %39 ...2002: %23 ...2003: %34 ...4004: %18
_That_'s why I prefer mbox to maildir!
Testing on my 6062 message inbox (a good mix of short messages and some long ones with: dsauer@danny-pc:~> df /tmp Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda2 116154428 53784568 62369860 47% / dsauer@danny-pc:~> grep hda2 /etc/mtab /dev/hda2 / reiserfs rw 0 0 dsauer@danny-pc:~> file /tmp/mbox /tmp/maildir/ /tmp/mbox: ISO-8859 mail text, with very long lines /tmp/maildir/: directory dsauer@danny-pc:~> du -hs /tmp/mbox /tmp/maildir/ 114M /tmp/mbox 127M /tmp/maildir/ dsauer@danny-pc:~> rm -r /tmp/maildir && df /tmp Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda2 116154428 53663144 62491284 47% / dsauer@danny-pc:~> echo '53784568-53663144' | bc 121424 dsauer@danny-pc:~> echo '(53784568-53663144)/1024' | bc 118 dsauer@danny-pc:~> rm /tmp/mbox && df /tmp Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda2 116154428 53546432 62607996 47% / dsauer@danny-pc:~> echo '(53663144-53546432)/1024' | bc 113 So, using Reiser, the maildir actualy only took up ~5MB more space, or about 4%. It's worth noting that, while there was 127MB of files, it was only taking up 118MB of space on a reiser filesystem. It's also worth noting that it took a little over 4 times as long to create the mbox file than it did the maildir files, even though the maildir was created first so those files were more likely to be alread cached. On a mailing list folder (no binary attachments) with 65591 messages, du reports 363MB, but it's actually only taking up 350MB of disk space (as compared to 280MB for the mbox). That's 25% less space, but it's just 25MB for well over 65 *thousand* messages. It's hard to find a drive that's less than 10GB now. Say that costs $100. That's $10/GB. Less than 1 cent per megabyte. The performance hit and file corruption risk on an mbox is not worth the 25 cents worth of disk space saved, IMHO. With an mbox, you've gotta do a lot of seeking to find the message you're interested in (or pre-scan the file generating file offsets at the time of opening). With maildir, you list a directory and go to the file corresponding to your message. 6000 messages in a folder? It takes a long time to read that 114MB text file, but not so long to read the contents of 2 directories. :) Scenario: A file is open and being written to when the computer loses power due to the UPS malfunctioning. That file is only half written, and the journal didn't manage to catch it. The file's corrupted. Results: mbox - all mail is in one file, a bunch is lost due to the failure. maildir - one message per file, part of one message is lost due to the failure. Disks are cheap, esp for a 5MB difference. Data recovery time is not. _That_'s (a few reasons) why I prefer maildir. :) --Danny
Danny, On Thursday 12 August 2004 14:21, Danny Sauer wrote:
Randall wrote regarding 'Re: [SLE] KMail question' on Wed, Aug 11 at 20:14:
[... comparing size of maildir to mbox...]
...
So, using Reiser, the maildir actualy only took up ~5MB more space, or about 4%. It's worth noting that, while there was 127MB of files, it was only taking up 118MB of space on a reiser filesystem. It's also worth noting that it took a little over 4 times as long to create the mbox file than it did the maildir files, even though the maildir was created first so those files were more likely to be alread cached.
Well, I'm not using ReiserFS, for one thing. I'm using XFS.
On a mailing list folder (no binary attachments) with 65591 messages, du reports 363MB, but it's actually only taking up 350MB of disk space (as compared to 280MB for the mbox). That's 25% less space, but it's just 25MB for well over 65 *thousand* messages. It's hard to find a drive that's less than 10GB now. Say that costs $100. That's $10/GB. Less than 1 cent per megabyte. The performance hit and file corruption risk on an mbox is not worth the 25 cents worth of disk space saved, IMHO.
But I have the drives I have in the cabinet and power supply I have. To me, waste as waste. Also, since my work (software development) demands high performance computing hardware, I only buy 10,000 RPM, Ultra 160 (or maybe now SATA) drives. No quite as cheap and usually of more modest capcity. Right now I have two 37 GB drives in my system.
With an mbox, you've gotta do a lot of seeking to find the message you're interested in (or pre-scan the file generating file offsets at the time of opening). With maildir, you list a directory and go to the file corresponding to your message. 6000 messages in a folder? It takes a long time to read that 114MB text file, but not so long to read the contents of 2 directories. :)
Look in your ~/Mail directory. The mailbox files are indexed. There's no need to hunt around for individual messages. The overhead of accessing those indexes and then seeking to the mbox file offset required is certainly less than the overhead of accessing and maintaining file system directories.
Scenario: A file is open and being written to when the computer loses power due to the UPS malfunctioning. That file is only half written, and the journal didn't manage to catch it. The file's corrupted.
Results:
mbox - all mail is in one file, a bunch is lost due to the failure. maildir - one message per file, part of one message is lost due to the failure.
UPS malfunctioning? Perhaps. New mail is always appended to the end of the file. No power failure is going to disrupt a single sector write (modern "Winchester" drives never experience uncontrolled shut-down--they must retract the heads to their landing zone and have internal power reserves sufficient to do so regardless of how or when they lose power). So either the new message us added, it's partially added or it's not added at all (or the file system indexing doesn't reflect the data being written, which is effectively the same as the new message not being added to the mbox file). No existing mail is likely to be disturbed by a crash or power failure. In all my many years, what few disk problems I've encountered have all been absolute catastrophes (meaning there's no recourse be to go to your most recent backup).
Disks are cheap, esp for a 5MB difference. Data recovery time is not.
Waste is waste. If a crash happens, data recovery might be required. If it is, fewer files mean easier and faster recovery.
_That_'s (a few reasons) why I prefer maildir. :)
--Danny
Randall Schulz
Randall wrote regarding 'Re: [SLE] KMail question' on Thu, Aug 12 at 16:46:
Danny,
On Thursday 12 August 2004 14:21, Danny Sauer wrote:
Randall wrote regarding 'Re: [SLE] KMail question' on Wed, Aug 11 at 20:14:
[... comparing size of maildir to mbox...]
...
So, using Reiser, the maildir actualy only took up ~5MB more space, or about 4%. It's worth noting that, while there was 127MB of files, it was only taking up 118MB of space on a reiser filesystem. It's also worth noting that it took a little over 4 times as long to create the mbox file than it did the maildir files, even though the maildir was created first so those files were more likely to be alread cached.
Well, I'm not using ReiserFS, for one thing. I'm using XFS.
XFS is good for big files, and mbox is a big file. Reiser is good for lots of small files. Maildir uses small files. If using Maildir, it oughtta be on a ReiserFS.
On a mailing list folder (no binary attachments) with 65591 messages, du reports 363MB, but it's actually only taking up 350MB of disk space (as compared to 280MB for the mbox). That's 25% less space, but it's just 25MB for well over 65 *thousand* messages. It's hard to find a drive that's less than 10GB now. Say that costs $100. That's $10/GB. Less than 1 cent per megabyte. The performance hit and file corruption risk on an mbox is not worth the 25 cents worth of disk space saved, IMHO.
But I have the drives I have in the cabinet and power supply I have. To me, waste as waste. Also, since my work (software development) demands high performance computing hardware, I only buy 10,000 RPM, Ultra 160 (or maybe now SATA) drives. No quite as cheap and usually of more modest capcity. Right now I have two 37 GB drives in my system.
Perhaps you should consider storing those mbox files gzipped, too. They're text files, and compress very well. Waste is waste, you know? :)
With an mbox, you've gotta do a lot of seeking to find the message you're interested in (or pre-scan the file generating file offsets at the time of opening). With maildir, you list a directory and go to the file corresponding to your message. 6000 messages in a folder? It takes a long time to read that 114MB text file, but not so long to read the contents of 2 directories. :)
Look in your ~/Mail directory. The mailbox files are indexed. There's no need to hunt around for individual messages. The overhead of accessing those indexes and then seeking to the mbox file offset required is certainly less than the overhead of accessing and maintaining file system directories.
I don't have a ~/Mail directory. However, I do have a system that switched from mbox to Maildir that now performs much better in all measurable ways. Whiel it's true that an mbox file is faster to seek within when there are only a small number of messages, it scales badly. Maildir scales much more nicely. Clearly mbox serves your needs, but others searching the archive may benefit from knowing the limitations of both schemes. That, BTW, is my reason for posting.
Scenario: A file is open and being written to when the computer loses power due to the UPS malfunctioning. That file is only half written, and the journal didn't manage to catch it. The file's corrupted.
Results:
mbox - all mail is in one file, a bunch is lost due to the failure. maildir - one message per file, part of one message is lost due to the failure.
UPS malfunctioning? Perhaps.
I was gonna just say "power failure" but realized that most have UPSs by now. :)
New mail is always appended to the end of the file. No power failure is going to disrupt a single sector write (modern "Winchester" drives never experience uncontrolled shut-down--they must retract the heads to their landing zone and have internal power reserves sufficient to do so regardless of how or when they lose power). So either the new message us added, it's partially added or it's not added at all (or the file system indexing doesn't reflect the data being written, which is effectively the same as the new message not being added to the mbox file). No existing mail is likely to be disturbed by a crash or power failure.
What happens when you delete a message? That's right, the portion of the file after that deleted message has to be rewritten. When you pop your mail, and the pop server's deleting the first few messages, what happens when the SMTP server on the same machine starts delivering a message? I hope the locking mechanism works perfectly. With Maildir, you can do whatever you want with messages while new messages are being delivered. This may not matter to you personally, but a few years back I had an IMAP server crash while accessing my mbox. It didn't clear the filesystem lock it had on the mbox file. So, new messages started piling up in the SMTP queue. I didn't realize this, though, because my IMAP client didn't notify me. If I'd been using maildir, that wouldn't have happened. I've also had things corrupt the first line in an mbox file. Then, the accessing program can't get to any of the messages in there until I go in and fix the first line by hand. Again, this isn't an issue with Maildir. My priority is data reliability under modification. The space lost is nearly nothing. If setting up a new system where lots of messages will be stored, Maildir is really the only sensible choice. There are a few isolated situations where a simplistic low capacity system will work better, or where someone is stuck with a legacy setup. I'm sure mbox is fine, there. So far, I know exactly 1 person for whom mbox works better, though. :) --Danny, noting that NNTP storage works like maildir in most cases, and that such a decision was made for a reason ;)
Danny, On Friday 13 August 2004 08:56, Danny Sauer wrote:
Randall wrote regarding 'Re: [SLE] KMail question' on Thu, Aug 12 at 16:46:
...
Perhaps you should consider storing those mbox files gzipped, too. They're text files, and compress very well. Waste is waste, you know? :)
I don't see this option in KMail's options dialogs. How does one do this? Under Windows, I always applied system-level compression to the files that stored my mailbox files (Eudora always uses mbox format, by the way). Actually, I used compression for my archived mail, but kept active mailboxes (those still receiving new mail) uncompressed.
With an mbox, you've gotta do a lot of seeking to find the message you're interested in (or pre-scan the file generating file offsets at the time of opening). With maildir, you list a directory and go to the file corresponding to your message. 6000 messages in a folder? It takes a long time to read that 114MB text file, but not so long to read the contents of 2 directories. :)
Look in your ~/Mail directory. The mailbox files are indexed. There's no need to hunt around for individual messages. The overhead of accessing those indexes and then seeking to the mbox file offset required is certainly less than the overhead of accessing and maintaining file system directories.
I don't have a ~/Mail directory. However, I do have a system that switched from mbox to Maildir that now performs much better in all measurable ways. Whiel it's true that an mbox file is faster to seek within when there are only a small number of messages, it scales badly. Maildir scales much more nicely. Clearly mbox serves your needs, but others searching the archive may benefit from knowing the limitations of both schemes. That, BTW, is my reason for posting.
Overhead in space and time for multiple files will always be greater than for the same or very similar amount of data stored in a single file. Keep in mind that the indexing files are a practical necessity even for maildir storage, since without it the displaying the summaries in the mailbox pane of the mail client would require access to all the files each time those summaries was created (each time you switched mail folders in KMail).
Scenario: A file is open and being written to when the computer loses power due to the UPS malfunctioning. That file is only half written, and the journal didn't manage to catch it. The file's corrupted.
Results:
mbox - all mail is in one file, a bunch is lost due to the failure. maildir - one message per file, part of one message is lost due to the failure.
UPS malfunctioning? Perhaps.
I was gonna just say "power failure" but realized that most have UPSs by now. :)
New mail is always appended to the end of the file. No power failure is going to disrupt a single sector write (modern "Winchester" drives never experience uncontrolled shut-down--they must retract the heads to their landing zone and have internal power reserves sufficient to do so regardless of how or when they lose power). So either the new message us added, it's partially added or it's not added at all (or the file system indexing doesn't reflect the data being written, which is effectively the same as the new message not being added to the mbox file). No existing mail is likely to be disturbed by a crash or power failure.
What happens when you delete a message? That's right, the portion of the file after that deleted message has to be rewritten. When you pop your mail, and the pop server's deleting the first few messages, what happens when the SMTP server on the same machine starts delivering a message? I hope the locking mechanism works perfectly. With Maildir, you can do whatever you want with messages while new messages are being delivered. This may not matter to you personally, but a few years back I had an IMAP server crash while accessing my mbox. It didn't clear the filesystem lock it had on the mbox file. So, new messages started piling up in the SMTP queue. I didn't realize this, though, because my IMAP client didn't notify me. If I'd been using maildir, that wouldn't have happened. I've also had things corrupt the first line in an mbox file. Then, the accessing program can't get to any of the messages in there until I go in and fix the first line by hand. Again, this isn't an issue with Maildir.
You seem to be confusing local mail storage by the client (KMail) and mail storage in mail drop files or on servers. The latter is not what we're discussing. SMTP, POP, IMAP, mail drop files etc. are all irrelevant here. Locking mechanisms either work or they don't. I, too, "hope" the Linux kernel works. And again, I'm a single user on a personal computer. No one else is going to be manipulating my mailbox files, concurrently or otherwise. I'm certainly not going to accidentally edit an mailbox file while I'm using KMail. The deleted message are simply abandoned. The index no longer refers to them. When the mailbox file is compacted (I compact when exiting), that abandoned space is recovered. I also "hope" that KMail's file compaction is implemented intelligently, i.e., so as to minimize the likelihood of data loss in the unlikely event that some sort of hardware or software failure occurs during compcation.
My priority is data reliability under modification. The space lost is nearly nothing. If setting up a new system where lots of messages will be stored, Maildir is really the only sensible choice. There are a few isolated situations where a simplistic low capacity system will work better, or where someone is stuck with a legacy setup. I'm sure mbox is fine, there. So far, I know exactly 1 person for whom mbox works better, though. :)
--Danny, noting that NNTP storage works like maildir in most cases, and that such a decision was made for a reason ;)
NNTP server implementation issues are also irrelevant to the matter of storing email client-managed mail stores. We are talking about using KMail, right? And please stop winking at me, OK? Randall Schulz
Randall wrote regarding 'Re: [SLE] KMail question' on Fri, Aug 13 at 11:23:
Danny,
On Friday 13 August 2004 08:56, Danny Sauer wrote:
Randall wrote regarding 'Re: [SLE] KMail question' on Thu, Aug 12 at 16:46:
...
Perhaps you should consider storing those mbox files gzipped, too. They're text files, and compress very well. Waste is waste, you know? :)
I don't see this option in KMail's options dialogs. How does one do this? Under Windows, I always applied system-level compression to the files that stored my mailbox files (Eudora always uses mbox format, by the way). Actually, I used compression for my archived mail, but kept active mailboxes (those still receiving new mail) uncompressed.
I had something like squashfs in mind, but it appears to be read-only. I was sure there was a dynamic compression option at the filesystem level in one of the filesystems, but it eludes me for now. Maybe one of those that support dynamic encryption? Maybe I was thinking of the planned ext4... Eh, that'd require a seperate partition anyway. Is read-write NTFS support stable yet? :)
New mail is always appended to the end of the file. No power failure is going to disrupt a single sector write (modern "Winchester" drives [...]
What happens when you delete a message? That's right, the portion of the file after that deleted message has to be rewritten. When you pop your mail, and the pop server's deleting the first few messages, what [...]
You seem to be confusing local mail storage by the client (KMail) and mail storage in mail drop files or on servers. The latter is not what we're discussing. SMTP, POP, IMAP, mail drop files etc. are all irrelevant here.
I was discussing Maildir for general mail storage, typically on a server. I'd just as soon have my client store messages in RAM, personally, since my workstations typically have enough RAM to do so. Perhaps this is why we're not seeing eye-to-eye - different topics of dission. :)
The deleted message are simply abandoned. The index no longer refers to them. When the mailbox file is compacted (I compact when exiting), that abandoned space is recovered. I also "hope" that KMail's file compaction is implemented intelligently, i.e., so as to minimize the likelihood of data loss in the unlikely event that some sort of hardware or software failure occurs during compcation.
Again, on a mail client, an mbox file would probably be fine, since there can be a resident index, only one program accessing it, etc. I don't use kmail, I use mutt on the same file that messages get delivered to and the same spool that a couple of other programs check periodically via IMAP. So, from my client view, I prefer maildir. If I was using Kmail - which won't happen due to the major problems it's caused me - mbox would be just fine if that's what it wanted to do. Anyway, I figured that we'd drifted away from just kmail and back to maildir v/s mbox in general...
And please stop winking at me, OK?
Preference noted. :p --Danny, using emoticons to compensate for lack of inflection 'n body language
On Thu, Aug 12, 2004 at 04:21:54PM -0500 or thereabouts, Danny Sauer wrote:
Randall wrote regarding 'Re: [SLE] KMail question' on Wed, Aug 11 at 20:14: <snip>
_That_'s why I prefer mbox to maildir!
mbox - all mail is in one file, a bunch is lost due to the failure. maildir - one message per file, part of one message is lost due to the failure.
One other factor, Maildir does not use any locking as mbox does One of the benefits of the maildir format is that, even though it doesn't use locking to prevent simultaneous updates from different delivery agents, it's reliable. This means maildir mailboxes can safely reside on NFS-mounted filesystems. you can't use mbox on NFS and expect mail not to be lost or have broken spools. -- Gary
Gary, On Thursday 12 August 2004 14:44, Gary wrote:
On Thu, Aug 12, 2004 at 04:21:54PM -0500 Danny Sauer wrote:
Randall wrote regarding 'Re: [SLE] KMail question' on Wed, Aug 11 at 20:14:
<snip>
_That_'s why I prefer mbox to maildir!
mbox - all mail is in one file, a bunch is lost due to the failure. maildir - one message per file, part of one message is lost due to the failure.
One other factor, Maildir does not use any locking as mbox does
Make no mistake about it. Concurrency control has to happen, whether it is via an explicit use of a file locking system call or protocol or whether it is hidden in the operating system's file system code.
One of the benefits of the maildir format is that, even though it doesn't use locking to prevent simultaneous updates from different delivery agents, it's reliable. This means maildir mailboxes can safely reside on NFS-mounted filesystems.
Virtually all of my mail is accessed via POP3 servers (I pick up a trickle via my login's /var/spool/mail drop file, where root's mail is redirected). There's only one me, here, and I can only use one mailer at a time, so there's no chance of lock conflicts on the mail storage used by KMail. Lock overhead is not an issue.
you can't use mbox on NFS and expect mail not to be lost or have broken spools.
NFS is irrelevant to me.
-- Gary
Randall Schulz
On Thu, Aug 12, 2004 at 02:56:17PM -0700 or thereabouts, Randall R Schulz wrote:
Gary,
On Thursday 12 August 2004 14:44, Gary wrote:
On Thu, Aug 12, 2004 at 04:21:54PM -0500 Danny Sauer wrote:
Randall wrote regarding 'Re: [SLE] KMail question' on Wed, Aug 11 at 20:14: <snip>
_That_'s why I prefer mbox to maildir!
mbox - all mail is in one file, a bunch is lost due to the failure. maildir - one message per file, part of one message is lost due to the failure.
One other factor, Maildir does not use any locking as mbox does
Make no mistake about it. Concurrency control has to happen, whether it is via an explicit use of a file locking system call or protocol or whether it is hidden in the operating system's file system code.
of course, in this case, my SMTP servers takes care of that.
One of the benefits of the maildir format is that, even though it doesn't use locking to prevent simultaneous updates from different delivery agents, it's reliable. This means maildir mailboxes can safely reside on NFS-mounted filesystems.
Virtually all of my mail is accessed via POP3 servers (I pick up a trickle via my login's /var/spool/mail drop file, where root's mail is redirected). There's only one me, here, and I can only use one mailer at a time, so there's no chance of lock conflicts on the mail storage used by KMail.
Your above case, and needs, are thoroughly explained, and in this situation, I agree with you... Mail via POP3, etc.. My main thrust was/is using Maildir on the server level, where incoming 20-30 or more email / second at peak times, is best handled via Maildir without any locking spools.
Lock overhead is not an issue.
in your situation, I agree.
you can't use mbox on NFS and expect mail not to be lost or have broken spools.
NFS is irrelevant to me.
ummm... I usually live by it <g> Best regards, -- Gary
On Thursday 12 August 2004 23:21, Danny Sauer wrote:
Randall wrote regarding 'Re: [SLE] KMail question' on Wed, Aug 11 at 20:14:
[... comparing size of maildir to mbox...]
...2001: %39 ...2002: %23 ...2003: %34 ...4004: %18
_That_'s why I prefer mbox to maildir!
Testing on my 6062 message inbox (a good mix of short messages and some long ones with:
dsauer@danny-pc:~> df /tmp Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda2 116154428 53784568 62369860 47% / dsauer@danny-pc:~> grep hda2 /etc/mtab /dev/hda2 / reiserfs rw 0 0 dsauer@danny-pc:~> file /tmp/mbox /tmp/maildir/ /tmp/mbox: ISO-8859 mail text, with very long lines /tmp/maildir/: directory dsauer@danny-pc:~> du -hs /tmp/mbox /tmp/maildir/ 114M /tmp/mbox 127M /tmp/maildir/ dsauer@danny-pc:~> rm -r /tmp/maildir && df /tmp Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda2 116154428 53663144 62491284 47% / dsauer@danny-pc:~> echo '53784568-53663144' | bc 121424 dsauer@danny-pc:~> echo '(53784568-53663144)/1024' | bc 118 dsauer@danny-pc:~> rm /tmp/mbox && df /tmp Filesystem 1K-blocks Used Available Use% Mounted on /dev/hda2 116154428 53546432 62607996 47% / dsauer@danny-pc:~> echo '(53663144-53546432)/1024' | bc 113
So, using Reiser, the maildir actualy only took up ~5MB more space, or about 4%.
Did the maildir include index files, or was it just the mails?
On Thursday 12 August 2004 03:13, Randall R Schulz wrote:
I did a little experiment to gauge the internal fragmentation overhead of "mbox" vs. "maildir" storage in KMail when using an XFS file system.
I was talking about reiserfs. No solution is right for everyone, and several factors usually needs to be considered. xfs is known to be good for few large files, while reiserfs excels at systems with many small files, so a benchmark of "mbox vs maildir" will have to consider more than just "mbox vs. maildir" [1]. I have no religious issues on this topic, I just took issue with your comment about its being "insanely bloated", which clearly gives a whole new meaning to the word "insane". Space tradeoffs for speed aren't exactly unheard of, so "insane == large" isn't an obvious conclusion. Anders [1] http://www.courier-mta.org/mbox-vs-maildir/
Anders, On Thursday 12 August 2004 15:19, Anders Johansson wrote:
On Thursday 12 August 2004 03:13, Randall R Schulz wrote:
I did a little experiment to gauge the internal fragmentation overhead of "mbox" vs. "maildir" storage in KMail when using an XFS file system.
I was talking about reiserfs. No solution is right for everyone, and several factors usually needs to be considered. xfs is known to be good for few large files, while reiserfs excels at systems with many small files, so a benchmark of "mbox vs maildir" will have to consider more than just "mbox vs. maildir" [1].
The material I read suggested that XFS is meant to accommodate very large files, very large disks, very many disks as well as very many (not-so-big) files. I did mention from the very beginning that I was using XFS. Unfortunately, I used only the brief descriptions in the installation section of the SuSE Administration Guide in choosing which file system to use. That was undoubtedly ill-advised.
I have no religious issues on this topic, I just took issue with your comment about its being "insanely bloated", which clearly gives a whole new meaning to the word "insane". Space tradeoffs for speed aren't exactly unheard of, so "insane == large" isn't an obvious conclusion.
It must be the climate of hyperbole here in the States...
Anders
Randall Schulz
Doug wrote regarding '[SLE] KMail question' on Mon, Aug 09 at 15:13:
I like KMail, and have had no problems with it. (I'm not doing complicated filters, etc.) And thank heavens it _doesn't_ look like a Microsoft routine! But I can't find a way to implement a feature that Eudora has: a way to automatically copy an attachment on an incoming message to a folder. It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
Does anybody know of a way to do that? TIA!
Ok, let's see if we can actually solve your probem, then, now that we have a good list of tools to do so. :) First question which determines how to solve the problem: How do you get your mail? Is it from a local maildir/mbox file, or do you use pop/imap? Also, do you only receive mail your mail on one machine in one place, or do you require access from multiple machines? --Danny
On Tuesday 10 August 2004 12:20, Danny Sauer wrote:
Doug wrote regarding '[SLE] KMail question' on Mon, Aug 09 at 15:13:
I like KMail, and have had no problems with it. (I'm not doing /snip/ It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
Does anybody know of a way to do that? TIA!
Ok, let's see if we can actually solve your probem, then, now that we have a good list of tools to do so. :)
First question which determines how to solve the problem:
How do you get your mail? Is it from a local maildir/mbox file, or do you use pop/imap? Also, do you only receive mail your mail on one machine in one place, or do you require access from multiple machines?
--Danny Under Kmail>settings>configure>Network>Receiving: Name Type Folder dmcgarrett pop inbox
Under Kmail>settings>configure>Network>Sending: name Type Domain dmcgarrett@optonline.net smtp optonline.net Somewhere else in the configure file, it has "default: maildir" or something to that effect. The first thing the incoming signal from the cable sees is a Lynksys router, which feeds this machine as well as a Windows-only machine, either of which can read and send mail with the other machine turned off, if desired. I'd just as soon keep things that way, in case one fails. The Win machine is very old and very slow, and while I do have an XP drive on here, it has been more than a month since I used that system.
Doug wrote regarding 'Re: [SLE] KMail question' on Tue, Aug 10 at 14:38:
On Tuesday 10 August 2004 12:20, Danny Sauer wrote:
Doug wrote regarding '[SLE] KMail question' on Mon, Aug 09 at 15:13:
I like KMail, and have had no problems with it. (I'm not doing /snip/ It would be even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder. i.e., if the incoming message was a .doc file, it would automatically copy it to the /home/doug/Documents folder.
Does anybody know of a way to do that? TIA!
Ok, let's see if we can actually solve your probem, then, now that we have a good list of tools to do so. :)
First question which determines how to solve the problem:
How do you get your mail? Is it from a local maildir/mbox file, or do you use pop/imap? Also, do you only receive mail your mail on one machine in one place, or do you require access from multiple machines?
--Danny Under Kmail>settings>configure>Network>Receiving: Name Type Folder dmcgarrett pop inbox
Under Kmail>settings>configure>Network>Sending: name Type Domain dmcgarrett@optonline.net smtp optonline.net
Somewhere else in the configure file, it has "default: maildir" or something to that effect.
[...] Alright. You're popping your mail. The solution I was going to suggest involved setting up fetchmail to pop your message, delivering them locally though procmail, which could then yank the attachments out and put them into another place. However, I then read the kmail docs. :) They allow the use of a filter action called "pipe through" wherin the whole message is sent through a program, and whatever the program prints out replaces the message - if anything's printed. That'll do. So, now all you need is a program that can pull attachments off of messages, save them, and return the message sans attachments. Sounds like a job for perl, since I didn't find anything pre-written. :) ...time passes... I hope this doesn't violate some unspoken list rule, but I wrote something to do what you want. I know you don't do Perl, but if you can manage to install a couple of CPAN modules and can figure out how to change a couple of vars, this oughtta do what you want if you save it to a text file, make it executable, and call it from the pipe filter. You'll have to install Email::MIME and Email::MIME::Attachment::Stripper from CPAN. Ask seperatly if you don't know how to do that. Perl's definatey worth learning, though. It does lots of cool stuff pretty easily. ...start_copy_n_paste... #!/usr/bin/perl -w use Email::MIME; use Email::MIME::Attachment::Stripper; # set your mapping of content-types to destinations here. # mime_type => path/to/folder my %destinations = ( 'image/jpeg' => '/tmp/attachments/jpegs', 'image/gif' => '/tmp/attachments/gifs', 'application/pdf' => '/tmp/attachments/pdfs', ); my $default_destination = '/tmp/attachments'; # read/parse the message my $mail = join('', <>); my $mail_parsed = Email::MIME->new($mail); # pull off attachments and save them appropriately my $stripper = Email::MIME::Attachment::Stripper->new($mail_parsed); my Email::MIME $mail_stripped = $stripper->message; my @attachments = $stripper->attachments; foreach $attachment (@attachments){ my $destination = $default_destination; if(defined($destination = $destinations{$attachment->{'content_type'}})){ $destination = $destinations{$attachment->{'content_type'}} } # compensate for missing filenames (should rarely happen) unless($attachment->{'filename'}){ $attachment->{'filename'} = 'attachment_'.localtime(); } my $outfile = $destination.'/'.$attachment->{'filename'}; open(HANDLE, '>', $outfile) or die "problem with '$outfile': $!"; print HANDLE $attachment->{'payload'}; close HANDLE; } # spit stripped email back out to kmail print $mail_stripped->as_string(); ...end_copy_n_paste... Hopefully that'll work for you. It worked in my limited testing, but if it breaks your mailbox, you get to keep the pieces. The program should be more robust, but this'll work for the common cases. Make sure that the directories exist before running - you could lose mail if you specify a destination folder that doesn't exist (that one of those "robust" things that oughtta be addressed). --Danny, releasing this with no license ;)
On Tuesday 10 August 2004 22:26, Danny Sauer wrote:
Doug wrote regarding 'Re: [SLE] KMail question' on Tue, Aug 10 at 14:38:
On Tuesday 10 August 2004 12:20, Danny Sauer wrote:
Doug wrote regarding '[SLE] KMail question' on Mon, Aug 09 at 15:13:
I like KMail, and have had no problems with it. (I'm not doing
/snip/
It would be
even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder.
Danny, only minor crit is that he did say copy. Perhaps a cmdline option to copy/strip? Kudos for actually putting something out there though. -- Steve Boddy
Stephen wrote regarding 'Re: [SLE] KMail question' on Tue, Aug 10 at 16:51:
On Tuesday 10 August 2004 22:26, Danny Sauer wrote:
Doug wrote regarding 'Re: [SLE] KMail question' on Tue, Aug 10 at 14:38:
On Tuesday 10 August 2004 12:20, Danny Sauer wrote:
Doug wrote regarding '[SLE] KMail question' on Mon, Aug 09 at 15:13:
I like KMail, and have had no problems with it. (I'm not doing
/snip/
It would be
even cooler if I could specify the filename extension of the incoming attachment and only copy that file to a specified folder.
Danny, only minor crit is that he did say copy. Perhaps a cmdline option to copy/strip?
Kudos for actually putting something out there though.
Whoops - I thought he said strip, not copy. :) replace print $mail_stripped->as_string(); with print $mail; and it'll print the original message unharmed. Kmail will also leave the message unchanged if nothing's printed, so it should work just as well (it'd probably be a little quicker) to just comment out that line entirely. --Danny
participants (11)
-
Anders Johansson
-
Bruce Marshall
-
Danny Sauer
-
dh
-
Doug McGarrett
-
Gary
-
Misty Stanley-Jones
-
Patrick Shanahan
-
Randall R Schulz
-
Sid Boyce
-
Stephen Boddy