[opensuse-factory] kmail hangs with 100% CPU

newer
[opensuse-factory] Questionable...

Achim Gratz

10 Sep 2017 10 Sep '17

17:03

Since the KDEPIM update in August akonadi would at random times start using up CPU and had some errors in the mysql error log about deadlocks. So I moved the ~/.local/share/akonadi directory aside and let it create a new one. That worked and it also seemed to have indexed some stuff that for whatever reason it hadn't before. However, kmail now completely freaks out when encountering large folders (they're local maildirs) and just hangs with 100% CPU and no further reaction. It doesn't appear to try and communicate with akonadi or anything else, it just hangs. Sometimes it tells it has either threaded or processed some large percentage of the entries (however the total number doesn't match what's in the folder), but there is absolutely no reaction to whatever attempts at leaving the folder or anything else. The largest folder that still works has 4091 entries, the next size up that fails is 5506 mails (about 200 of that are in a sub-folder, though). So I've moved the old directory back and I can use these folders again. The wierd background activity from mysqld (CPU + I/O, both can be stopped by doing an "akonadictl restart", but will return after some activity in kmail) is back as are these errors in mysql.err: 2017-09-10 18:52:16 7f5328a0fb00 InnoDB: Error: Fetch of persistent statistics requested for table "akonadi"."tagtypetable" but the required system tables mysql.innodb_table_stats and mysql.innodb_index_stats are not present or have unexpected structure. Using transient stats instead. 2017-09-10 18:53:32 7f530abff700 InnoDB: Error: Column last_update in table "mysql"."innodb_table_stats" is INT UNSIGNED NOT NULL but should be BINARY(4) NOT NULL (type mismatch). Does anybody have an idea what is going on and how to fix it properly? Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Show replies by date

Martin Herkt

11 Sep 11 Sep

12:45

On 2017 M09 10, Sun 19:03:18 CEST Achim Gratz wrote:

...

Does anybody have an idea what is going on and how to fix it properly?

No idea what’s going on. Been having other trouble with the MySQL backend myself, more than half a year ago. It wouldn’t index properly and sometimes fail to move items to different directories (INBOX to Trash in particular). Restarting Akonadi or rebuilding the local database would sometimes mitigate the issue, but it wasn’t a long-term solution. I’ve since switched to the PostgreSQL backend, which seems to work perfectly with no issues (plus it’s faster).

Achim Gratz

17:52

New subject: [opensuse-factory] Re: kmail hangs with 100% CPU

Martin Herkt writes:

...

I’ve since switched to the PostgreSQL backend, which seems to work perfectly with no issues (plus it’s faster).

So how do I do that? Is there some tutorial or step-by-step someplace? Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Martin Herkt

21:44

New subject: [opensuse-factory] Re: kmail hangs with 100% CPU

On 2017 M09 11, Mon 19:52:50 CEST Achim Gratz wrote:

...

Martin Herkt writes:

...
I’ve since switched to the PostgreSQL backend, which seems to work perfectly with no issues (plus it’s faster).

So how do I do that? Is there some tutorial or step-by-step someplace?

There used to be a configuration thing for Akonadi in systemsettings which allowed you to change the storage backend, but that seems to be gone now :( Basically make sure libQt5Sql5-postgresql and postgresql-server are installed, and change Driver=QMYSQL to Driver=QPSQL in ~/.config/akonadi/akonadiserverrc. Then restart Akonadi (akonadictl restart). Should work. There’s an SQLite backend for Akonadi as well (requires a separate package), but I don’t know how mature it is. Last time I tried that, it was quite slow and unreliable, but that was years ago and things might have changed.

Luca Beltrame

12 Sep 12 Sep

05:25

New subject: [opensuse-factory] Re: kmail hangs with 100% CPU

Il giorno Mon, 11 Sep 2017 23:44:51 +0200 Martin Herkt ha scritto:

...

Then restart Akonadi (akonadictl restart). Should work.

Note that *no* migration of the data is performed. This is why the option was removed from akonadiconsole, to prevent people from accidentally destroying their data. -- Luca Beltrame - KDE Forums team GPG key ID: A29D259B

Wolfgang Bauer

08:05

New subject: [opensuse-factory] Re: kmail hangs with 100% CPU

Am Dienstag, 12. September 2017, 07:25:05 CEST schrieb Luca Beltrame:

...

Il giorno Mon, 11 Sep 2017 23:44:51 +0200

Martin Herkt ha scritto:

...
Then restart Akonadi (akonadictl restart). Should work.

Note that *no* migration of the data is performed. This is why the option was removed from akonadiconsole, to prevent people from accidentally destroying their data.

It's no different to wiping out the database though (i.e. deleting/moving away ~/.local/share/akonadi/), as the OP already did... ;-) I don't see the described problem here btw on Leap 42.3 and kdepim 17.08.1 (17.08.0 worked fine too), with the mysql backend. But there is an upstream bug report that sounds similar: https://bugs.kde.org/show_bug.cgi?id=384359 Kind Regards, Wolfgang -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Achim Gratz

18:22

New subject: [opensuse-factory] Re: kmail hangs with 100% CPU

Wolfgang Bauer writes:

...

It's no different to wiping out the database though (i.e. deleting/moving away ~/.local/share/akonadi/), as the OP already did... ;-)

Well, I moved the old one back for now. Anyway, I'm curious to know what data I'd supposedly lose by doing that.

...

I don't see the described problem here btw on Leap 42.3 and kdepim 17.08.1 (17.08.0 worked fine too), with the mysql backend. But there is an upstream bug report that sounds similar: https://bugs.kde.org/show_bug.cgi?id=384359

I don't think this describes the same situation. In that bugreport, mysqld and akonadi eat CPU and do disk I/O. In my case it's kmail alone (and no I/O I can see). Is there a way to figure out what kmail tries to do? Based on the suspicious relation to the folder size there might be something (relatively) easy to fix going on. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Q+, Q and microQ: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Luca Beltrame

19:01

New subject: [opensuse-factory] Re: kmail hangs with 100% CPU

Il giorno Tue, 12 Sep 2017 20:22:56 +0200 Achim Gratz ha scritto:

...

alone (and no I/O I can see). Is there a way to figure out what kmail tries to do? Based on the suspicious relation to the folder size there might be something (relatively) easy to fix going on.

If you can install the relevant debug symbols, attach gdb and provide a backtrace of kmail in that state it would be very useful in figuring out what's going on. -- Luca Beltrame - KDE Forums team GPG key ID: A29D259B

Achim Gratz

13 Sep 13 Sep

17:15

New subject: [opensuse-factory] Re: kmail hangs with 100% CPU

Luca Beltrame writes:

...

If you can install the relevant debug symbols, attach gdb and provide a backtrace of kmail in that state it would be very useful in figuring out what's going on.

I was hoping for something less intrusive, like snooping the dbus messages. Not sure if I find the time to install all that stuff and then run things in the debugger. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Waldorf MIDI Implementation & additional documentation: http://Synth.Stromeko.net/Downloads.html#WaldorfDocs -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Wolfgang Bauer

10:24

New subject: [opensuse-factory] Re: kmail hangs with 100% CPU

Am Dienstag, 12. September 2017, 20:22:56 CEST schrieb Achim Gratz:

...

Well, I moved the old one back for now. Anyway, I'm curious to know what data I'd supposedly lose by doing that.

I answered a similar question here recently: https://lists.opensuse.org/opensuse-factory/2017-08/msg00623.html There may be other things stored in the database though, in general I would expect this to be the case for all "special" things that are not supported by the "conventional" storage backends (maildir, IMAP, ...), e.g. certain mail flags that are not supported by your IMAP server, custom tags, things like that. If that could be a problem for you depends on your usage I suppose.

...

I don't think this describes the same situation. In that bugreport, mysqld and akonadi eat CPU and do disk I/O. In my case it's kmail alone (and no I/O I can see).

Well, you did mention high CPU usage and disk I/O by Akonadi and mysql too in your original mail... But ok, I read it completely again, and those problems apparently seem to be fixed by removing the database. If its actually kmail that uses up the CPU now, I highly doubt that switching the Akonadi database backend would have any effect though. Kmail only talks to Akonadi, not the database backend. As a side-note: The mysql errors mentioned do sound to me like the database got corrupted somehow... (the mysql internal part, not Akonadi's tables)

...

Is there a way to figure out what kmail tries to do?

Getting a backtrace with gdb, as Luca wrote. According to your problem description, it might be somehow related to message threading/grouping though, so try to turn that off maybe and see if it helps. I.e. try to switch to one of the "flat" options in View->Message List->Aggregation. Another thing that would probably be worth a try is to rename ~/.local/share/kmail2/ and/or the settings file (~/.config/kmail2rc). I think I heard about problems with corrupted autosaved mails in the past, and I think saved searches can (or could in the past) also cause such problems, but I'm not sure at the moment where those are saved as I never used them myself (maybe ~/.local/share/kmail2/ as well). Although, IIUIC the problem only occurs with certain mail folders, so that probably won't help then. Kind Regards, Wolfgang -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Achim Gratz

17:28

New subject: [opensuse-factory] Re: kmail hangs with 100% CPU

Wolfgang Bauer writes:

...

Am Dienstag, 12. September 2017, 20:22:56 CEST schrieb Achim Gratz:

...
Well, I moved the old one back for now. Anyway, I'm curious to know what data I'd supposedly lose by doing that.

I answered a similar question here recently: https://lists.opensuse.org/opensuse-factory/2017-08/msg00623.html

Hmm. I guess I won't have a problem then. As I said, all mail is already local and any flags that may have been stored somewhere were lost anyway during the botched kmail2 transition since I had to manually import all mails at that time.

...

Well, you did mention high CPU usage and disk I/O by Akonadi and mysql too in your original mail...

Yes, that's back with the old (somewhat damaged) database back in use at the moment. It mostly happens when I send a new mail and a restart of akonadi gets rid of it, so I can cope for now.

...

But ok, I read it completely again, and those problems apparently seem to be fixed by removing the database.

Yes. Just that I get another problem I don't have a workaround for.

...

If its actually kmail that uses up the CPU now, I highly doubt that switching the Akonadi database backend would have any effect though.

Well, I don't really know what happens and it really seems to be triggered by moving above the 4096 barrier, so maybe if the database backends serve up the data in smaller or larger chunks this might make a difference. There is no discernible delay when opening a slightly smaller folder, so I doubt some superlinear algorithm triggers this.

...

Kmail only talks to Akonadi, not the database backend.

I _think_ that the problem is precisely that Akonadi does deliver all the data. Currently these large folders have a notceable delay when entering them before any entries are shown, but eventually they will be (and kmail remains responsive in the meantime, so I can switch to a different folder for instance).

...

According to your problem description, it might be somehow related to message threading/grouping though, so try to turn that off maybe and see if it helps. I.e. try to switch to one of the "flat" options in View->Message List->Aggregation.

Hmm. Could be although I got at least once a status message from kmial that it had threaded something close to the number of messages in the folder before it stopped responding.

...

Another thing that would probably be worth a try is to rename ~/.local/share/kmail2/ and/or the settings file (~/.config/kmail2rc).

I did that once while trying to fixup things while trsnsitioning from kmail2 and I don't want to do it again if I can avoid it.

...

I think I heard about problems with corrupted autosaved mails in the past, and I think saved searches can (or could in the past) also cause such problems, but I'm not sure at the moment where those are saved as I never used them myself (maybe ~/.local/share/kmail2/ as well).

I don't think that's a problem or at least I wouldn't know where these would come from. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Achim Gratz

12 Sep 12 Sep

18:26

New subject: [opensuse-factory] Re: kmail hangs with 100% CPU

Martin Herkt writes:

...

On 2017 M09 11, Mon 19:52:50 CEST Achim Gratz wrote:

...
Martin Herkt writes:

...
I’ve since switched to the PostgreSQL backend, which seems to work perfectly with no issues (plus it’s faster).

So how do I do that? Is there some tutorial or step-by-step someplace?

There used to be a configuration thing for Akonadi in systemsettings which allowed you to change the storage backend, but that seems to be gone now :(

Basically make sure libQt5Sql5-postgresql and postgresql-server are installed, and change Driver=QMYSQL to Driver=QPSQL in ~/.config/akonadi/akonadiserverrc.

Then restart Akonadi (akonadictl restart). Should work.

Oh OK. Based on some descriptions from Debian I was under the impression I'd need some akonadi-*-postgresql package for this to work.

...

There’s an SQLite backend for Akonadi as well (requires a separate package), but I don’t know how mature it is. Last time I tried that, it was quite slow and unreliable, but that was years ago and things might have changed.

I don't want to read too much into those forum discussions, but if what I read is close enough to the truth the expected performance would not be great. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Terratec KOMPLEXER: http://Synth.Stromeko.net/Downloads.html#KomplexerWaves -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

Knurpht - Gertjan Lettink

14 Sep 14 Sep

14:20

Op maandag 11 september 2017 14:45:37 CEST schreef Martin Herkt:

...

On 2017 M09 10, Sun 19:03:18 CEST Achim Gratz wrote:

...
Does anybody have an idea what is going on and how to fix it properly?

No idea what’s going on. Been having other trouble with the MySQL backend myself, more than half a year ago. It wouldn’t index properly and sometimes fail to move items to different directories (INBOX to Trash in particular). Restarting Akonadi or rebuilding the local database would sometimes mitigate the issue, but it wasn’t a long-term solution.

I’ve since switched to the PostgreSQL backend, which seems to work perfectly with no issues (plus it’s faster).

Thanks for this. Got 9 accounts, ~ 120,000 emails ( kept everything from 2001 on, except spam ) and faster? Oh yeah, much faster. I never had any real trouble with akonadi/mysql until today. Akonadi indexer is still busy, but the overall behaviour of kmail re. opening huge folders is much snappier. -- Gertjan Lettink, a.k.a. Knurpht openSUSE Board Member openSUSE Forums Team -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org

2422

Age (days ago)

2426

Last active (days ago)

List overview

Download

12 comments

5 participants

participants (5)

Achim Gratz
Knurpht - Gertjan Lettink
Luca Beltrame
Martin Herkt
Wolfgang Bauer