Am Freitag, 25. November 2011, 01:56:57 schrieb Roger Luedecke:

On Friday, November 25, 2011 10:41:51 AM Adrian Schröter wrote:

...

Hi,

we observe that it seems not to be possible to disable the nepomuk feeders in akonadi. Since this seems to be the root that kmail2 is not usable at all by default for plenty of people, I would like to propose to remove the Autostart of these feeders in following files:

/usr/share/akonadi/agents/nepomukcalendarfeeder.desktop /usr/share/akonadi/agents/nepomukcontactfeeder.desktop /usr/share/akonadi/agents/nepomukemailfeeder.desktop

That would change the default (not tested), it would still be possible to add them in the GUI tools and people can suddenly read mails again ;)

any opinions on this ? adrian

PS: I have not seen any working kmail2 installation yet where these are enabled. Really everybody around me has switched to Thunderbird and I think the problem is actually not akonadi/kmail, but nepomuk which basically creates a very high load which basically stops the system. Killing all these feeders is solving the problem (if you don't want to wait for the crash to be able to move your mouse again), but they come back on next restart, even when you remove them in akonadiconsole.

I have been using Kontact/Kmail since day one with no strange fuddling. Can I help you find out what you need? My system works damn near to perfect with KMail. Better than thunder*&^% and dEvolution.

please find out why the nepomuk feeders create such high load when you have mail boxes > 10000 mails (in my personal case >100000 actually). Maybe the reason is also the combination with disconnected imap, which everybody uses here. -- Adrian Schroeter SUSE Linux Products GmbH email: adrian@suse.de -- To unsubscribe, e-mail: opensuse-kde+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kde+owner@opensuse.org

On Friday 25 November 2011 11:00:12 Adrian Schröter wrote:

Am Freitag, 25. November 2011, 10:59:38 schrieb Adrian Schröter:

...

Am Freitag, 25. November 2011, 01:56:57 schrieb Roger Luedecke:

...

On Friday, November 25, 2011 10:41:51 AM Adrian Schröter wrote:

...

Hi,

we observe that it seems not to be possible to disable the nepomuk feeders in akonadi. Since this seems to be the root that kmail2 is not usable at all by default for plenty of people, I would like to propose to remove the Autostart of these feeders in following files:

/usr/share/akonadi/agents/nepomukcalendarfeeder.desktop /usr/share/akonadi/agents/nepomukcontactfeeder.desktop /usr/share/akonadi/agents/nepomukemailfeeder.desktop

That would change the default (not tested), it would still be possible to add them in the GUI tools and people can suddenly read mails again ;)

any opinions on this ? adrian

PS: I have not seen any working kmail2 installation yet where these are enabled. Really everybody around me has switched to Thunderbird

Well, they're losers for not reporting their issues to us! And I blames management for seating the boosters in a disconnected part of the building from the rest of R&D. The people at SUSE who have come to me went away happy (me love users long time).

...

...

...

and I think the problem is actually not akonadi/kmail, but nepomuk which basically creates a very high load which basically stops the system. Killing all these feeders is solving the problem (if you don't want to wait for the crash to be able to move your mouse again), but they come back on next restart, even when you remove them in akonadiconsole.

Known issue, but I think shooting the feeders in the head in the packages as you suggest is overkill. For one, the only feeder that generally has significant amounts of data to index is the email feeder. Killing the contacts feeder would break email address completion. Furthermore, I'm working on 2 changes to make indexing large quantities of mail voluntary, but possible: 1) Remove the "index" flag from all folders by default. Perahps only index inbox and sent-mail by default 2) Add a dialog that allows selecting the folders to be indexed by multiple selection from a folder tree, instead of only hiding the per folder indexing checkbox in Folder Properties->Maintenance->Enable full-text indexing. I should get that done next week.

...

...

I have been using Kontact/Kmail since day one with no strange fuddling. Can I help you find out what you need? My system works damn near to perfect with KMail. Better than thunder*&^% and dEvolution.

please find out why the nepomuk feeders create such high load when you have mail boxes > 10000 mails (in my personal case >100000 actually).

Suboptimal implementation: Firstly, Akonadi and KMail2 communicate quite efficiently via an IMAP-like streaming protocol. But the feeders watch all the PIM folders for changes, then pass the data to be indexed to nepomuk over dbus, which then has to communicate with the virtuoso server via a local socket. (processes) [transports] (akonadiserver) -[akonadi-imap]->(feeder)-[dbus]-

(nepomukstorage)-[localsocket]->(virtuoso)

which is wasteful. Secondly, all mail is indexed by default. Thirdly the feeder implementation in 4.7 is trivial, and naively sucks in everything in a given collection to be indexed, with the result that akonadi_nepomuk_email_feeder's memory usage balloons to include the entirety of your 100000 mail box, then chews through each mail to index its headers and full text. 4.8's implementation should process items to be indexed in chunks. Fourthly There are low hanging fruit optimisations still to be done throughout the indexing end of the pipeline, as Ismael's change today from a commit yesterday shows (https://bugs.kde.org/show_bug.cgi?id=286516), although looking at the change, I don't understand how it will improve mail indexing. See we used to have a fulltime PIM developer to take care of making beatiful designs ugly enough to work in the real SUSE world, but something happened to him...

...

Maybe the reason is also the combination with disconnected imap, which everybody uses here.

That does guarantee that the mail bodies are available to be indexed, increasing the amount of work.

But apart from this, why shouldn't it be possible to permanently not to use these feeders ?

There's exactly this checkbox in KDE 4.8 now (and the per-type feeders have been merged into one process). -- Will Stephenson, KDE Developer, openSUSE Boosters Team SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany -- To unsubscribe, e-mail: opensuse-kde+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kde+owner@opensuse.org

On Friday 25 Nov 2011 13:01:33 Will Stephenson wrote:

Secondly, all mail is indexed by default.

I didnt really mean to type such a long email but once I started it just kept going, apologies.... I've noticed the high load that Adrian talks about, but it's only when I *initially* add IMAP accounts with thousands of emails, few 30k folders in there, only one with as much as 80k and none above 100k. But I have a lot of these folders, so some accounts easily have over 200k emails in total. So I assumed it was just indexing as much as it could after adding these accounts and I just let it churn away. I've no idea how long it took to index everything because I left it overnight and went to bed; but it was easily pegging a full core and possibly more while it was doing it's thing. So I can imagine this might be pretty annoying on a netbook or laptop. However there's a bright side! I've honestly not experienced this high load since I let everything get indexed and I'm actually happier with Kmail, Akonadi and Strigi all running and *not* being noticable in any way now. Nepomuk actually being useful and fast to search email contents and find them using krunner makes me happy as a pig in *$#! So I'm convinced after the initial indexing, Kmail and Nepomuk is actually in a pretty good place but this initial load is certainly an annoyance, and since there are no warnings about what is going on to create the load it can look pretty bad to the user. Perhaps there should be a user notification/warning that indexing may take some time? There's one other issue that I see occasionally, I'm trying to find a reproduce case for it, I think I'm almost there; and I think it may exagerate the problems people are seeing when the try to setup kmail2 for the first time. The issue I'm seeing is that if I perform a "batch" operation in kmail, say deletion of 5k emails, then any other action to view emails, mark as read etc are blocked for a while. A brief poke at ~/.xsession-errors shows it filling up with the following type of lines, at about 2 per second. So if this corresponds to each deleted email, and at a rate of approximately 2 per second, then my mail viewing is blocked for some time. 14212 +FLAGS (\Deleted) 14213 +FLAGS (\Deleted) 14214 +FLAGS (\Deleted) 14215 +FLAGS (\Deleted) 14216 +FLAGS (\Deleted) 14217 +FLAGS (\Deleted) [/usr/bin/nepomukservicestub] virtual void Soprano::Server::LocalServer::incomingConnection(quintptr) [/usr/bin/nepomukservicestub] void Soprano::Server::ServerCorePrivate::addConnection(Soprano::Server::ServerConnection*) New connection. New count: 15 [/usr/bin/nepomukservicestub] Soprano::ODBC::Connection::Connection() Soprano::Server::ServerConnection(0x17ba730) void Nepomuk::Query::QueryServiceClient::close() 14218 +FLAGS (\Deleted) 14219 +FLAGS (\Deleted) 14220 +FLAGS (\Deleted) 14221 +FLAGS (\Deleted) 14222 +FLAGS (\Deleted) 14223 +FLAGS (\Deleted) 14224 +FLAGS (\Deleted) 14225 +FLAGS (\Deleted) 14226 +FLAGS (\Deleted) etc etc etc... So it really wouldn't surprise me if nempomuk is also blocking mail viewing while the initial indexing is taking place, which pretty much looks to the user that everything is frozen and broken, which it actually is in a sad way even though things are churning over in the background. So yeah nepomuk feeders for email should be off by default until the initial indexing is less painfull and resource hungry, I'm pretty much convinced peoples kmail issues would almost all dissapear. Personally I'm leaving nepomuk turned *on* because really, truthfully and honestly, it's actually working pretty nice for me with some pretty large IMAP folders. I can't say the same for IMAP in the previous Kmail ;) Cheers the noo, Graham -- To unsubscribe, e-mail: opensuse-kde+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kde+owner@opensuse.org

On Friday, November 25, 2011 01:01:33 PM Will Stephenson wrote:

On Friday 25 November 2011 11:00:12 Adrian Schröter wrote:

...

Am Freitag, 25. November 2011, 10:59:38 schrieb Adrian Schröter:

...

Am Freitag, 25. November 2011, 01:56:57 schrieb Roger Luedecke:

...

On Friday, November 25, 2011 10:41:51 AM Adrian Schröter wrote:

...

Hi,

we observe that it seems not to be possible to disable the nepomuk feeders in akonadi. Since this seems to be the root that kmail2 is not usable at all by default for plenty of people, I would like to propose to remove the Autostart of these feeders in following files:

/usr/share/akonadi/agents/nepomukcalendarfeeder.desktop /usr/share/akonadi/agents/nepomukcontactfeeder.desktop /usr/share/akonadi/agents/nepomukemailfeeder.desktop

That would change the default (not tested), it would still be possible to add them in the GUI tools and people can suddenly read mails again ;)

any opinions on this ? adrian

PS: I have not seen any working kmail2 installation yet where these are enabled. Really everybody around me has switched to Thunderbird

Well, they're losers for not reporting their issues to us! And I blames management for seating the boosters in a disconnected part of the building from the rest of R&D. The people at SUSE who have come to me went away happy (me love users long time).

...

...

...

...

and I think the problem is actually not akonadi/kmail, but nepomuk which basically creates a very high load which basically stops the system. Killing all these feeders is solving the problem (if you don't want to wait for the crash to be able to move your mouse again), but they come back on next restart, even when you remove them in akonadiconsole.

Known issue, but I think shooting the feeders in the head in the packages as you suggest is overkill. For one, the only feeder that generally has significant amounts of data to index is the email feeder. Killing the contacts feeder would break email address completion.

Furthermore, I'm working on 2 changes to make indexing large quantities of mail voluntary, but possible:

1) Remove the "index" flag from all folders by default. Perahps only index inbox and sent-mail by default

2) Add a dialog that allows selecting the folders to be indexed by multiple selection from a folder tree, instead of only hiding the per folder indexing checkbox in Folder Properties->Maintenance->Enable full-text indexing.

I should get that done next week.

...

...

...

I have been using Kontact/Kmail since day one with no strange fuddling. Can I help you find out what you need? My system works damn near to perfect with KMail. Better than thunder*&^% and dEvolution.

please find out why the nepomuk feeders create such high load when you have mail boxes > 10000 mails (in my personal case >100000 actually).

Suboptimal implementation:

Firstly, Akonadi and KMail2 communicate quite efficiently via an IMAP-like streaming protocol. But the feeders watch all the PIM folders for changes, then pass the data to be indexed to nepomuk over dbus, which then has to communicate with the virtuoso server via a local socket.

(processes) [transports]

(akonadiserver) -[akonadi-imap]->(feeder)-[dbus]-

...

(nepomukstorage)-[localsocket]->(virtuoso)

which is wasteful.

Secondly, all mail is indexed by default.

Thirdly the feeder implementation in 4.7 is trivial, and naively sucks in everything in a given collection to be indexed, with the result that akonadi_nepomuk_email_feeder's memory usage balloons to include the entirety of your 100000 mail box, then chews through each mail to index its headers and full text. 4.8's implementation should process items to be indexed in chunks.

Fourthly There are low hanging fruit optimisations still to be done throughout the indexing end of the pipeline, as Ismael's change today from a commit yesterday shows (https://bugs.kde.org/show_bug.cgi?id=286516), although looking at the change, I don't understand how it will improve mail indexing.

See we used to have a fulltime PIM developer to take care of making beatiful designs ugly enough to work in the real SUSE world, but something happened to him...

...

...

Maybe the reason is also the combination with disconnected imap, which everybody uses here.

That does guarantee that the mail bodies are available to be indexed, increasing the amount of work.

...

But apart from this, why shouldn't it be possible to permanently not to use these feeders ?

There's exactly this checkbox in KDE 4.8 now (and the per-type feeders have been merged into one process).

-- Will Stephenson, KDE Developer, openSUSE Boosters Team SUSE LINUX GmbH, GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer, HRB 21284 (AG Nürnberg) Maxfeldstraße 5 90409 Nürnberg Germany That all sounds very good. Would there be perhaps a way to have it not DL and index messages that are marked as read on the server? -- Roger Luedecke openSUSE Ambassador Ind. Repairs and Consulting **Looking for a C++ etc. mentor*** -- To unsubscribe, e-mail: opensuse-kde+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-kde+owner@opensuse.org