[opensuse] Strange KMail Behavior on 11.1
Hi, When I created my new openSUSE 11.1 installation, I did everything from scratch including a brand new home directory with one exception. I transferred all my KMail files (both ~/Mail—yes the old location for mail files) and the various settings files. Everything went surprisingly well with this but since the very first run of KMail with those transplanted files, I've noticed one odd symptom: Every time I quit KMail (not just close its window(s), but actually quit the application), there's a huge burst of I/O, as evidenced by a buzz from the many rapid seeks of the drive holding my home directory. This lasts about a second or so (it's a 10,000 RPM Ultra-160 SCSI drive, so that represents a lot of I/O). Can anybody shed any light on what is being done and how, hopefully, I can stop that from happening? It never did that under 10.0. Thanks. Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday 13 January 2009 15:24:26 Randall R Schulz wrote:
Everything went surprisingly well with this but since the very first run of KMail with those transplanted files, I've noticed one odd symptom: Every time I quit KMail (not just close its window(s), but actually quit the application), there's a huge burst of I/O, as evidenced by a buzz from the many rapid seeks of the drive holding my home directory. This lasts about a second or so (it's a 10,000 RPM Ultra-160 SCSI drive, so that represents a lot of I/O).
Can anybody shed any light on what is being done and how, hopefully, I can stop that from happening? It never did that under 10.0.
If you were on Windows, I'd suggest running ProcMon from SysInternals, and find out that way what's going on. Does anyone here know if there's an equivalent monitor for Linux? Seems like there oughta be. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Tuesday, 2009-01-13 at 15:24 -0800, Randall R Schulz wrote: ...
Can anybody shed any light on what is being done and how, hopefully, I can stop that from happening? It never did that under 10.0.
Wild guess: compacting folders. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAkltKHkACgkQtTMYHG2NR9XBwwCfbF5J8ezeDJwcw/9oKTGHIb+V +9AAniaVEJJMD60rcTxzVbruybv0H+cn =QXoM -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday January 13 2009, Carlos E. R. wrote:
On Tuesday, 2009-01-13 at 15:24 -0800, Randall R Schulz wrote:
...
Can anybody shed any light on what is being done and how, hopefully, I can stop that from happening? It never did that under 10.0.
Wild guess: compacting folders.
I guess that's plausible—I do use mbox mailboxes. I wonder how to confirm it? If I select multiple folders, the "Compact Folder" command in both the Folder menu and the folder list context menu are dimmed. Also, I wonder how to prevent it? I see no options that seem related. Another thought that occurs to me is that it could be related to message "expiry." I wonder where that might be configured? I see reference to it in the KMail Settings under Misc -> Folders as "Exclude important messages from expiry." I see no other options related to expiry.
-- Cheers, Carlos E. R.
Randal Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wednesday 14 January 2009 02:02:56 Randall R Schulz wrote:
On Tuesday January 13 2009, Carlos E. R. wrote:
On Tuesday, 2009-01-13 at 15:24 -0800, Randall R Schulz wrote:
...
Can anybody shed any light on what is being done and how, hopefully, I can stop that from happening? It never did that under 10.0.
Wild guess: compacting folders.
I guess that's plausible—I do use mbox mailboxes. I wonder how to confirm it? If I select multiple folders, the "Compact Folder" command in both the Folder menu and the folder list context menu are dimmed. Also, I wonder how to prevent it? I see no options that seem related.
Another thought that occurs to me is that it could be related to message "expiry." I wonder where that might be configured? I see reference to it in the KMail Settings under Misc -> Folders as "Exclude important messages from expiry." I see no other options related to expiry.
Expiry settings are per-folder - right click a folder in the list and select Expire... It's off by default, obviously, and when it is on it happens in the background while KMail is running. KMail uses mmap'ed folder index files to get its speed. I guess the write happens when these are closed. It could also be Beagle noticing that all the kmail folders changed timestamp and scanning them. Will -- Will Stephenson Desktop Engineer KDE Team -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Wednesday January 14 2009, Will Stephenson wrote:
On Wednesday 14 January 2009 02:02:56 Randall R Schulz wrote:
On Tuesday January 13 2009, Carlos E. R. wrote:
On Tuesday, 2009-01-13 at 15:24 -0800, Randall R Schulz wrote:
...
Can anybody shed any light on what is being done and how, hopefully, I can stop that from happening? It never did that under 10.0.
Wild guess: compacting folders.
I guess that's plausible—I do use mbox mailboxes. ...
Another thought that occurs to me is that it could be related to message "expiry." ...
Expiry settings are per-folder - right click a folder in the list and select Expire... It's off by default, obviously, and when it is on it happens in the background while KMail is running.
Yes, I found them eventually. I've never used it and found none that were enabled.
KMail uses mmap'ed folder index files to get its speed.
The amount of a I/O that occurs is not dependent on the number of folders that were altered while running. This would seem to infer that the compacting hypothesis is not correct either, unless there's a bug of some sort.
I guess the write happens when these are closed.
Possible, though it seems unlikely. For one thing, I've had KMail crash, and if it was holding all the writes until shut-down, much mail would be lost, and that has never happened.
It could also be Beagle noticing that all the kmail folders changed timestamp and scanning them.
Beagle is not installed.
Will
-- Will Stephenson Desktop Engineer KDE Team
Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Tuesday January 13 2009, Randall R Schulz wrote:
Hi,
When I created my new openSUSE 11.1 installation, I did everything from scratch including a brand new home directory with one exception. I transferred all my KMail files (both ~/Mail—yes the old location for mail files) and the various settings files.
Everything went surprisingly well with this but since the very first run of KMail with those transplanted files, I've noticed one odd symptom: Every time I quit KMail (not just close its window(s), but actually quit the application), there's a huge burst of I/O, as evidenced by a buzz from the many rapid seeks of the drive holding my home directory. This lasts about a second or so (it's a 10,000 RPM Ultra-160 SCSI drive, so that represents a lot of I/O).
Can anybody shed any light on what is being done and how, hopefully, I can stop that from happening? It never did that under 10.0.
I've discovered what this huge burst of I/O is when quitting KMail. By using the "-mmin" option of find, it's clear that every ".index.ids" file in the mail storage folder (~/Mail, old-style) is being written when KMail quits. I also notice that while KMail is running, they're being updated selectively (apparently when a mailbox associated with the .index.ids file gets new mail) but nonetheless, when KMail quits, they're all written again, whether or not any new mail was deposited into them by KMail while it was running. Furthermore, it appears that merely opening / viewing a mailbox in KMail will cause the corresponding .index.ids file to be touched. I don't know what these .index.ids files are used for, but for some reason, KMail is writing to them all unconditionally when it shuts down. I wonder if simply removing these .index.ids files might cause KMail to regenerate them and in doing so eliminate this behavior? Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Saturday 2009 January 17 10:46:07 Randall R Schulz wrote:
I wonder if simply removing these .index.ids files might cause KMail to regenerate them and in doing so eliminate this behavior?
Unlikely, but rather than deleting them, you might try right-clicking a folder and investigating the "Troubleshoot IMAP Cache..." option -- Boyd Stephen Smith Jr. ,= ,-_-. =. bss@iguanasuicide.net ((_/)o o(\_)) ICQ: 514984 YM/AIM: DaTwinkDaddy `-'(. .)`-' http://iguanasuicide.net/ \_/
On Saturday January 17 2009, Boyd Stephen Smith Jr. wrote:
On Saturday 2009 January 17 10:46:07 Randall R Schulz wrote:
I wonder if simply removing these .index.ids files might cause KMail to regenerate them and in doing so eliminate this behavior?
Unlikely, but rather than deleting them, you might try right-clicking a folder and investigating the "Troubleshoot IMAP Cache..." option
No IMAP in evidence. Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Saturday January 17 2009, Randall R Schulz wrote:
...
I wonder if simply removing these .index.ids files might cause KMail to regenerate them and in doing so eliminate this behavior?
Well, I tried this. After removing them and restarting, there was, unsurprisingly, a lot of I/O at start-up while the .index.ids files were recreate. Sadly, the original symptom remains. What's more, the second start-up after removing them causes KMail to appear to go into an infinite loop. It turns out that KMail is again processing every mailbox file (mbox format). For me, this took a little over 5 minutes of 100% CPU. (I thought it was broken, but by repeatedly running lsof for the KMail process and filtering out those files in my ~/Mail directory, I could see it was slowly progressing from one mailbox to another.) In the end, I'm back where I started, a huge burst of I/O and the modification times of every single .index.ids file being updated when KMail quits. This is really quite annoying. Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sunday, 2009-01-18 at 06:23 -0800, Randall R Schulz wrote: ...
In the end, I'm back where I started, a huge burst of I/O and the modification times of every single .index.ids file being updated when KMail quits. This is really quite annoying.
Probably by design :-P - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAklzu4kACgkQtTMYHG2NR9VhRgCeKhIM75b//FQ1UztwWg5wbtts C/EAn0lXsloWrXSyLHRghSRudVverEWU =5tGy -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sunday January 18 2009, Carlos E. R. wrote:
On Sunday, 2009-01-18 at 06:23 -0800, Randall R Schulz wrote:
...
In the end, I'm back where I started, a huge burst of I/O and the modification times of every single .index.ids file being updated when KMail quits. This is really quite annoying.
I also verified that what's being written does not alter the contents, at least insofar as the checksums of each of the 258 .index.ids files are the same before and after shutting down KMail, while every one of them has its modification time updated.
Probably by design :-P
Well, it never happened before I started using 11.1, and the mailboxes and all configuration are the same as I was using on 10.0.
-- Cheers, Carlos E. R.
Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sunday, 2009-01-18 at 15:40 -0800, Randall R Schulz wrote:
In the end, I'm back where I started, a huge burst of I/O and the modification times of every single .index.ids file being updated when KMail quits. This is really quite annoying.
I also verified that what's being written does not alter the contents, at least insofar as the checksums of each of the 258 .index.ids files are the same before and after shutting down KMail, while every one of them has its modification time updated.
Touching a file is very fast, shouldn't be noticed; however, 258 files...? Maybe they are copied to another file? Check the inodes before and after. Backup archive? Doesn't kde4 have a new search engine? I forgot the name, not beagle.
Probably by design :-P
Well, it never happened before I started using 11.1, and the mailboxes and all configuration are the same as I was using on 10.0.
Dunno, maybe new feature. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAklzxE4ACgkQtTMYHG2NR9VNYQCeLPM/odmvfEZIgZDTMnKzVhtD uoUAoI/0rHzHu5f1MEImYmKgE7OUlKAq =Yt7e -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sunday January 18 2009, Carlos E. R. wrote:
On Sunday, 2009-01-18 at 15:40 -0800, Randall R Schulz wrote:
In the end, I'm back where I started, a huge burst of I/O and the modification times of every single .index.ids file being updated when KMail quits. This is really quite annoying.
I also verified that what's being written does not alter the contents, at least insofar as the checksums of each of the 258 .index.ids files are the same before and after shutting down KMail, while every one of them has its modification time updated.
Touching a file is very fast, shouldn't be noticed; however, 258 files...? Maybe they are copied to another file? Check the inodes before and after. Backup archive?
I'm not sure, but I don't think just touching those files would cause the physical I/O activity I'm seeing (well, hearing). I think it's actually writing the file with the same data it already contains.
Doesn't kde4 have a new search engine? I forgot the name, not beagle.
I'm using KDE 3.5.10 "release 21.9".
Probably by design :-P
Well, it never happened before I started using 11.1, and the mailboxes and all configuration are the same as I was using on 10.0.
Dunno, maybe new feature.
The "do lots of stupid, redundant, noisy, obnoxious I/O when quitting KMail" feature??
-- Cheers, Carlos E. R.
RRS -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Sunday, 2009-01-18 at 16:33 -0800, Randall R Schulz wrote:
Touching a file is very fast, shouldn't be noticed; however, 258 files...? Maybe they are copied to another file? Check the inodes before and after. Backup archive?
I'm not sure, but I don't think just touching those files would cause the physical I/O activity I'm seeing (well, hearing). I think it's actually writing the file with the same data it already contains.
I think you can make certain comparing the inode numbers before and after.
Dunno, maybe new feature.
The "do lots of stupid, redundant, noisy, obnoxious I/O when quitting KMail" feature??
Dunno, I was half joking. Maybe there is some obscure hidden feature that requires that activity or that has that side effect. - -- Cheers, Carlos E. R. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (GNU/Linux) iEYEARECAAYFAklz6GQACgkQtTMYHG2NR9U5EwCeIbSj77wwMKxGmzVEyPxCBxSb xwYAn2jQr6mq8BaxjWW76QwMP+BEKcgS =nhqN -----END PGP SIGNATURE----- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sunday 18 January 2009 06:33:53 pm Randall R Schulz wrote:
On Sunday January 18 2009, Carlos E. R. wrote:
On Sunday, 2009-01-18 at 15:40 -0800, Randall R Schulz wrote:
In the end, I'm back where I started, a huge burst of I/O and the modification times of every single .index.ids file being updated when KMail quits. This is really quite annoying.
I also verified that what's being written does not alter the contents, at least insofar as the checksums of each of the 258 .index.ids files are the same before and after shutting down KMail, while every one of them has its modification time updated.
Touching a file is very fast, shouldn't be noticed; however, 258 files...? Maybe they are copied to another file? Check the inodes before and after. Backup archive?
I'm not sure, but I don't think just touching those files would cause the physical I/O activity I'm seeing (well, hearing). I think it's actually writing the file with the same data it already contains.
Doesn't kde4 have a new search engine? I forgot the name, not beagle.
I'm using KDE 3.5.10 "release 21.9".
Probably by design :-P
Well, it never happened before I started using 11.1, and the mailboxes and all configuration are the same as I was using on 10.0.
Dunno, maybe new feature.
The "do lots of stupid, redundant, noisy, obnoxious I/O when quitting KMail" feature??
There is something that you use and I don't. I have no much hard disk activity on exit. It could be some program that is used by KMail and it is shutting down when KMail exits. KWallet, spamassasin, something else. Here: KMail is standalone, not PIM. I don't use encryption. Folders are mostly mailboxes, with few toplevel maildirs. The accounts are POP3. No IMAP. I just checked with the index of seldom used folders. Time was updated when I clicked on a folder to read messages, and again when I went away. The archive folder with 100,000 messages is some 500MB, and first read burst is 1 second, which with disk speed 50 MB/s means it reads 50 MB at once, and the rest takes another >30 s, with 2 short bursts of activity. Looking at file sizes: 513,530,948 main 47,154,361 .main.index 400161 .main.index.ids 5001649 .main.index.sorted it seems that it doesn't look in 'main', but only indexes. Compacting above folder takes 30 seconds of HD activity, with few shorter bursts until cursor goes back from busy to arrow. Total time is about 1'15". -- Regards, Rajko -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
On Sunday January 18 2009, Rajko M. wrote:
On Sunday 18 January 2009 06:33:53 pm Randall R Schulz wrote:
...
The "do lots of stupid, redundant, noisy, obnoxious I/O when quitting KMail" feature??
There is something that you use and I don't.
I take it you mean that as a conjecture, not a statement of fact.
I have no much hard disk activity on exit.
And I'll bet it's not something the developers have seen, or they'd have fixed it!
It could be some program that is used by KMail and it is shutting down when KMail exits. KWallet, spamassasin, something else.
I am now (11.1) using KWallet but was not before (10.0). No spamassasin or other external programs I'm aware of associated with KMail.
Here: KMail is standalone, not PIM. I don't use encryption. Folders are mostly mailboxes, with few toplevel maildirs. The accounts are POP3. No IMAP.
Likewise, I don't use any other KDE PIM modules / applications or encryption. Likewise, apart from the stock "inbox," "outbox," etc., all my mail "folders" are mbox format. I usually don't let any of them get bigger than about 5000 messages, which is partly why there are so many (I archive mailing lists indefinitely and when the primary list folder gets too big, I move older messages into a sub-folder with a date stamp in its name).
I just checked with the index of seldom used folders. Time was updated when I clicked on a folder to read messages, and again when I went away.
The archive folder with 100,000 messages is some 500MB, and first read burst is 1 second, which with disk speed 50 MB/s means it reads 50 MB at once, and the rest takes another >30 s, with 2 short bursts of activity. Looking at file sizes:
513,530,948 main 47,154,361 .main.index 400161 .main.index.ids 5001649 .main.index.sorted
it seems that it doesn't look in 'main', but only indexes.
I believe it has to go to the actual mbox file to get message content to display and will update it only when mail is added or removed. "Sorting" just builds a new index, it does not reorder the messages in the mbox file.
Compacting above folder takes 30 seconds of HD activity, with few shorter bursts until cursor goes back from busy to arrow. Total time is about 1'15".
-- Regards, Rajko
Randall Schulz -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org
participants (6)
-
Boyd Stephen Smith Jr. -
Carlos E. R. -
Jerry Houston -
Rajko M. -
Randall R Schulz -
Will Stephenson