Regis Matejcik said the following on 02/12/2009 02:39 PM:

Thanks for the reply Carlos,

I tried opening it in several text editors. They all fail before opening complaining that the file is too large.

Try a 'stream' (pipe&fitler) method such as split, sed. awk or perl to break the large file up into smaller ones. Make sure you have enough di space for this. Or make it easy - google for 'formail'. -- Teachers open the door. You enter by yourself. Chinese Proverb -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Thursday, 2009-02-12 at 18:50 +0100, Per Jessen wrote:

formail is your friend. You could use formai to split the mailbox file into individual mailfiles for instance.

Ah, yes... perhaps: +skip Skip the first skip messages while splitting. -total Output at most total messages while splitting. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkmUf3IACgkQtTMYHG2NR9XkSgCfT7UDh1o0b8UC9NalqcVpSVsP nTIAn2o4j44YWq/RdtL4JDrSJCeF7kvg =pGUW -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Fri, 13 Feb 2009 15:20:53 Arun Khan wrote:

On Thursday 12 Feb 2009, Regis Matejcik wrote:

...

I'm using OpenSuse 11.0 and Evolution 2.22.1.1

The problem is, even though I try to create sub directories, there seems to be a limit on the file size that evolution can handle. Last week Evolution wouldn't crashed and wouldn't re-start. I went into the .mail / inbox file and moved the 2gig+ inbox file to another directory. Evolution then started, recreated the inbox file and functions as normal.

After you have split the file, switch to maildir format instead of mbox (if Evolution has the option). This will help with the large file issue. Please do get into the habit of organizing your messages into separate folders but that is up to you :)

-- Arun Khan

I concur. Evolution will read maildir format (as will most mail clients). There are scripts on the net to convert mbox files to maildir directories without losing any mail (but back up the mbox file first, just in case). The one I use was called mb2md and was a perl script that I found via google. The current page for it is http://batleth.sapienti-sat.org/projects/mb2md/ HTH. Rodney. -- =================================================== Rodney Baker VK5ZTV rodney.baker@iinet.net.au ===================================================

Carlos E. R. said the following on 02/13/2009 06:17 AM:

On Friday, 2009-02-13 at 17:56 +1030, Rodney Baker wrote:

...

I concur. Evolution will read maildir format (as will most mail clients).

Not "most", just "some". Some that don't: Thunderbird, Alpine.

This is why I use 'fetchmail' and have a "mail host" serve up the email to Thunderbird via IMAP. It has a number of other advantages over allowing storage format choice, such as allowing access via any machine on my LAN, which has proven very useful in some circumstances. My 'workstation' is a high-end laptop and the "mail host" can collect mail from all my external mailboxes that I've accumulated over the years whether I'm present or not. Thunderbird doesn't have to be active for mail to be collected. Running 'fetchmail' on the "mail host" means that I can also have sophisticated 'procmail' and 'SpamAssassin' filters without impacting the performance of my workstation. If I'm at a remote site with my laptop I can still access my mail via a SSH tunnel. Mail gets backed up along with everything else on that server. Its not an elaborate set-up, but its very robust and flexible. And it lets me use Thunderbird with the plug-ins I like. Some people choose to off-load their mail processing onto a service provider, be it their ISP or Gmail/Hotmail. In many ways this isn't so different, but I've brought it all under my control If you really want to know a downside of Thunderbird: It is really a Gnome application, uses gnome libraries, which offends my KDE-purist soul. YMMV of course. -- The author who benefits you most is not the one who tells you something you did not know before, but the one who gives expression to the truth that has been dumbly struggling in you for utterance." -- Oswald Chambers -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Carlos E. R. wrote:

On Friday, 2009-02-13 at 07:34 -0500, Anton Aylward wrote:

...

Carlos E. R. said the following on 02/13/2009 06:17 AM:

...

...

I concur. Evolution will read maildir format (as will most mail clients).

Not "most", just "some". Some that don't: Thunderbird, Alpine.

...

This is why I use 'fetchmail' and have a "mail host" serve up the email to Thunderbird via IMAP.

Yes, so do I, fetchmail, postfix, procmail. But that doesn't mean I have to use maildir. I will not while it doesn't become a standard for all programs. I have read some very strong things agains maildir from some developpers...

Care to tell what kind of problems in what context were caused by maildir? Most of the time maildir is the best choice for imap servers, I don't really check the direct compatibility of enduser clients with maildir. Simply point the client to the imap server and everything should run fine. Mbox on the other hand is known to possibly cause lots of problems for imap servers, especially on a busy server with big mbox files. -- Sandy List replies only please! Please address PMs to: news-reply2 (@) japantest (.) homelinux (.) com -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

* Carlos E. R. [02-13-09 15:07]:

I am, of course, interested in the client side. What an imap server uses internally, does not matter. And the problem is that not all mail client use the same format of maildir folders, if they use any at all. The two clients I use most, ie, Alpine and Thunderbird, do not support maildir.

*Unfortunately*, the same is true of mbox, particularly it's handling of the From_/From(white-space) header. -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

* Carlos E. R. [02-13-09 18:52]:

On Friday, 2009-02-13 at 18:13 -0500, Patrick Shanahan wrote:

...

*Unfortunately*, the same is true of mbox, particularly it's handling of the From_/From(white-space) header.

Mmmm? What's that?

So far, I've never had problems feeding the mbox of one client to another.

:^), I haven't either.....but, A recent discussion on the mutt-users list: Date: Tue, 20 Jan 2009 15:51:31 -0600 From: Kyle Wheeler To: mutt-users@mutt.org Subject: Re: definition of signature separator ..... If you're interested in some quick reading about the hilarity of "the mbox format" (such as it is), check this out: http://homepages.tesco.net./~J.deBoynePollard/FGA/mail-mbox-formats.html In any case: DON'T USE MBOX! It's a lousy format for general-purpose email. The right mbox flavor can be good for read-only archives, but that's about it. ..... I commented that I had *never* experienced any problems with mbox that I knew about and after some further discussion some definitive points were presented that afaiac did not change my mind, I continue to use mbox. I can forward the entire conversation or provide it for http access if you prefer (off-list), as long as the backups have not become outdated and are deleted :^) -- Patrick Shanahan Plainfield, Indiana, USA HOG # US1244711 http://wahoo.no-ip.org Photo Album: http://wahoo.no-ip.org/gallery2 Registered Linux User #207535 @ http://counter.li.org -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

On Friday 13 February 2009 06:57:06 pm Carlos E. R. wrote:

I've read recently very strong reasons against maildir. I understand their proponents think it is superior to mbox, and those of mbox think it is superior. Kind of vi vs emacs. Thus I choose mbox.

My feelings as "a programmer that was" favour mbox: a file per message at a size about 2K, with file counts in the thousands or millions seems to me a waste of inodes. Disk space is wasted (unfilled clusters). Search has to be slower. On some filesystems it will simply not work.

I use mix of both because KMail allows to have different types for folder and subfolders. I didn't make use of this property, it is actually mess. It happened that one day that I was tired of slow opening of opensuse@opensuse.org, at the time maildir, just as any other, that I replaced it with mbox file. It took some time to convert, but when it was done opening was much faster. As experiment succeeded some bigger folders with mbox files were transformed as well, and the rest is left as is. Also I changed default for new folders to mbox. The point of mbox being faster is that you don't run whole a bunch of routines that will open file, read a 1-2 KB, and then repeat that file after file. Modern hard disks transfer speed is something like 50 MB/s, so for the biggest mbox that has old mails from opensuse list and is 500 MB, computer needs 10s. time cat main > /dev/null real 0m10.335s user 0m0.044s sys 0m1.884s If I repeat the same command: time cat main > /dev/null real 0m1.414s user 0m0.012s sys 0m1.360s KMail always needs 35s to switch to that folder, which means that it doesn't use copy already present in memory. I haven't checked kde4-kmail. -- Regards, Rajko -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

Thanks for all the replies. Not wanting to enter the Mbox vs. Maildir debate - But I thought I'd follow up with how I ended up solving this problem, using a really clever utility called "ArchMbox", available at SourceForge. It's dead simple to install and use. With a simple command line statement I was able to archive all mail older than a given date interval, making the problem file small enough to handle in evolution. thanks again, Regis On Sat, 2009-02-14 at 14:21 +0100, Sandy Drobic wrote:

Rajko M. wrote:

...

Also I changed default for new folders to mbox.

The point of mbox being faster is that you don't run whole a bunch of routines that will open file, read a 1-2 KB, and then repeat that file after file. Modern hard disks transfer speed is something like 50 MB/s, so for the biggest mbox that has old mails from opensuse list and is 500 MB, computer needs 10s.

time cat main > /dev/null

real 0m10.335s user 0m0.044s sys 0m1.884s

If I repeat the same command: time cat main > /dev/null

real 0m1.414s user 0m0.012s sys 0m1.360s

KMail always needs 35s to switch to that folder, which means that it doesn't use copy already present in memory. I haven't checked kde4-kmail.

Whoah, that's pretty miserable! Thunderbird uses indices that speed up switching folders a lot. Though I am using imap, so I can't compare it to local storage.

On my setup I have about 5.8 GB in my personal mailbox (via imap on server). In my case it is a Cyrus imap, that I plan to switch to dovecot in the future.

Changing folders takes at most a few seconds, even if the folder contains more than 40000 mails.

Though I can understand that for a personal mailbox on local storage mbox is indeed a better solution than maildir. The I/O to access the mailboxes is much less than for maildir.

-- Sandy

List replies only please! Please address PMs to: news-reply2 (@) japantest (.) homelinux (.) com

-- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday, 2009-02-14 at 14:13 +0100, Sandy Drobic wrote: (I forgot to reply to this one, I had fever)

Carlos E. R. wrote:

...

On Friday, 2009-02-13 at 20:09 +0100, Sandy Drobic wrote:

...

...

Yes, so do I, fetchmail, postfix, procmail. But that doesn't mean I have to use maildir. I will not while it doesn't become a standard for all programs. I have read some very strong things agains maildir from some developpers...

...

Care to tell what kind of problems in what context were caused by maildir?

Just ask dev Mark Crispin what he thinks about maildir... in the context of MUAs, specially, not servers. I have links to the alpine-alpha mail list, where he has been quite explicit, but they require password.

For example: inode waste, in the count of millions. Slow text search, unless you use some database indexing, like beagle. It does not really support concurrency, like one client reading one email and another different client erasing that same email. That there is no standard, and each client or server does its own interpretation. Courier, for instance, uses something like maildir, but not actually maildir.

Granted, it does use more space to store mails in separate files than in one big inbox, this will also slow down search operations since it takes a lot more I/O to scan lots of small files compared to one big file.

Though I challenge you to give me one real life example where one user/program is reading a mail and another one is deleting the same mail without it being a misconfiguration. (^-^)

It wasn't my claim, but his :-) However, I do just that, though I have to be careful not to open the same folder on two programs. I read most of my email with Alpine, but I have to read, lets say business mails, that come in html, with Thunderbird, so I can end with both clients opened - although I usually close one. Another example: I fetch mail from remote servers, like gmail, via imap (fetchmail). This happens in the background. At the same time, I may open the browser to get into gmail and clear the spam. While I'm there, fetchmail may trigger and do a download - and it works fine. However, gmail does not actually delete email.

Especially the issue of concurrency is exactly why maildir is superior to mbox. Try to have ten applications submit mails to the mailbox at the same time. With mbox you will have to serialize it and lock/unlock the mbox files. It has a severe performance drop and is known to cause lots of problems when separate applications are supposed to access mails from the mbox file.

That's understandable. The solution would be an "engine" like a database that would receive and process read and write process to the mail storage. I suppose that if you have an imap local delivery it does something of the kind. The wu-imap server I think uses mbox for storage, I think; I wonder how it behaves under a big load.

You will find a lot of woe reports in the postfix-users mailing list, where users reported performance problems, when the files got big and lots of mails hat to be delivered to the mbox files. The result was delayed delivery due to frequent timeouts on delivery when the new mail could not be added to the mbox file in time.

Supposedly procmail would move the mail to the user local folders before that.

Another problem that often occured was mbox files on nfs storage. The locking was often problematic.

No wonder, nfs and locking is problematic. In fact, IMO, the entire locking issue in Linux is problematic, because there have been several different locking methods: voluntary locking flag files (which name has to be specified), kernel mandatory lock, whatever. In the procmail man page there are references about how to achieve that locking - if the locking mechanism were well defined since ages, this would be clear cut and a non-issue. I remember reading in the msdos programmer's manual the locking methods available. You could lock an entire file, a region, for read, for write, in several combinations. When I cam to Linux and read that procmail page, I was... quite surprised.

The issue with an imap server not using maildir you mentioned is probably Cyrus imap, not Courier. Courier does use maildir while Cyrus uses its own proprietary storage format, even if the mails are stored in single files.

At this point I haven't even started on the problem of acls on shared mailboxes and how they are supposed to work on a single big file.

Buff!

...

I am, of course, interested in the client side. What an imap server uses internally, does not matter. And the problem is that not all mail client use the same format of maildir folders, if they use any at all. The two clients I use most, ie, Alpine and Thunderbird, do not support maildir.

True, though I consider is little loss. It is easier and more in the spirit of linux to use programs that do their job well and not to have a multipurpose do-it-all program with many features and an equal collection of bugs and incompatibilities.

So I someone uses email intensively it does make sense to let an imap server handle mailstorage and simply use a client that works via imap. That way you can have your choice of clients without having to worry about mailstorage format or migrating all your stored mails to another format just to test a new client.

Mmm... Yes, I know of that method, and I would possible use it of "others", not for me. I'm happy with my setup ;-)

It's fine to use mbox for smaller and few mailboxes but it will simply not scale.

I don't need it to scale :-)

By the way, does anyone know if the problematic setup with the 2 GB mbox files was a 64bit system? I wonder if it was a 32bit limitation of the program and if it could be avoided on a 64 bit system.

Good question. It maybe that the functions to access a text file position is a longint, which is 2GiB in 32 bits. I know that is the situation in dos/win (for fat files at least); I do not know about Linux. But if the same function calls use 64 bit sized variables, automatically the 2GiB limit would be broken. Did I mention the "mix" format for mail storage here? It's neither one file per message nor one for all. Googling for it I found some references that wu-imap might be using it. And also I found confirmation about the 2GiB limit: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=478545 ] The report is correct. The c-client library makes no attempt to use the ] 64bit system calls; and thus flat files are limited to 2GB. ] ] The recommended solution for mailboxes with aggregate size greater than ] 2GB is to use mix format instead of a flat file format. Even if ] c-client were updated to use the 64bit system calls, flat files ] (especially in traditional UNIX format) do not perform well at multi-GB ] sizes. ] ] If there is some compelling reason offered, I will consider 64bit ] support. However, as of right now, I am highly skeptical that the ] benefit would be worth the effort, especially since there is a superior ] solution for large mailboxes available now ("superior" since it will ] always work better than a large flat file). - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkmcnqcACgkQtTMYHG2NR9X/xgCdFPS14b6LSiLAE34HTUkZ6pao THsAn0Q8aH+8vD+Sfr9u6fQPzlXpYUq8 =71rm -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org