[opensuse] experience with dovecot search systems?
I note that dovecot has 3 different full text search methods one can use: dovecot23-fts-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-lucene-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-solr-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-squat-2.3.4.1-1.1.x86_64.rpm Has anyone ever used any of these? I've no clue how one differs or is better than another. Thanks, -l -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 13/04/2019 22.25, L A Walsh wrote:
I note that dovecot has 3 different full text search methods one can use:
dovecot23-fts-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-lucene-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-solr-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-squat-2.3.4.1-1.1.x86_64.rpm
Has anyone ever used any of these? I've no clue how one differs or is better than another.
I tried lucene on 2016 (I know because the index file is still there). I was not impressed, Thunderbird would not do good use of it. Then it got removed on some update, and I keep thinking that I should activate again some search method, but like you I don't know which one to apply. -- Cheers / Saludos, Carlos E. R. (from 15.0 x86_64 at Telcontar)
L A Walsh
I note that dovecot has 3 different full text search methods one can use:
dovecot23-fts-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-lucene-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-solr-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-squat-2.3.4.1-1.1.x86_64.rpm
Has anyone ever used any of these? I've no clue how one differs or is better than another.
I am still using Lucene, but apparently (according to the Dovecot docs) the recommended one to use these days is Solr. From what I have read, Solr is actually a server (written in Java) which used Lucene as a backend. Dovecot Lucene itself uses a library, but apparently it is buggy and haven't been updated in a while, so it is not recommended anymore along with squat. Charles -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 13.04.19 22:25, L A Walsh wrote:
I note that dovecot has 3 different full text search methods one can use:
dovecot23-fts-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-lucene-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-solr-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-squat-2.3.4.1-1.1.x86_64.rpm
Has anyone ever used any of these? I've no clue how one differs or is better than another.
Thanks, -l
Your message remembered me to try solr search in dovecot. Installation works as described here: https://wiki.dovecot.org/Plugins/FTS/Solr But you should: - Install at lease Java 8 (java-1_8_0-openjdk) - restart solr after replacing schema.xml and solrconfig.xml - Do not optimize Indexes Form here: https://wiki.apache.org/solr/SolrPerformanceFactors "You may want to optimize an index in certain situations -- ie: if you build your index once, and then never modify it. If you have a rapidly changing index, rather than optimizing, you likely simply want to use a lower merge factor. Optimizing is very expensive, and if the index is constantly changing, the slight performance boost will not last long. The tradeoff is not often worth it for a non static index. " So: don't do that. Conclusion: Search is faster. Searching the first time in a mail folder takes some time: the search index is built at first access, but then search is definitively faster.
On 21/04/2019 10.51, Florian Gleixner wrote:
On 13.04.19 22:25, L A Walsh wrote:
I note that dovecot has 3 different full text search methods one can use:
dovecot23-fts-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-lucene-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-solr-2.3.4.1-1.1.x86_64.rpm dovecot23-fts-squat-2.3.4.1-1.1.x86_64.rpm
Has anyone ever used any of these? I've no clue how one differs or is better than another.
Your message remembered me to try solr search in dovecot. Installation works as described here: https://wiki.dovecot.org/Plugins/FTS/Solr But you should: - Install at lease Java 8 (java-1_8_0-openjdk)
If it uses Java it will probably be memory expensive? -- Cheers / Saludos, Carlos E. R. (from openSUSE, Leap 15.1 x86_64 (ssd-test)) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
On 21.04.19 12:24, Carlos E. R. wrote:
If it uses Java it will probably be memory expensive?
No and yes. Java and memory is like projects and money. If you put more memory/money in Java/projects you will likely succeed, but you may fail too. If you look closer, sometimes another strategy will be better. If you look at solr, then you will see, that solr uses 2 memory resources: java heap memory and unix filesystem cache. Java heap memory is the internal java memory for variables and so on. You can set a minimum and a maximum with -Xms and -Xmx. Both are set by default to 512M for solr, and this is enough for most dovecot installations. If you get out of memory errors in the solr log, then set -Xmx higher. If the heap usage is always over 512M, then set -Xms higher. Easy. Solr/lucene uses memory mapping for the search index. So it relies on the unix filesystem cache to keep hot data in memory. So solr is fast, if all of your frequently queried solr index data stay in the filesystem cache. Putting all your memory in the Java heap via -Xmx/-Xms slows down solr, because there is no memory left for the filesystem cache and index data have to be read from disk over and over again.
On 21/04/2019 19.26, Florian Gleixner wrote:
On 21.04.19 12:24, Carlos E. R. wrote:
If it uses Java it will probably be memory expensive?
No and yes. Java and memory is like projects and money. If you put more memory/money in Java/projects you will likely succeed, but you may fail too. If you look closer, sometimes another strategy will be better.
If you look at solr, then you will see, that solr uses 2 memory resources: java heap memory and unix filesystem cache.
Java heap memory is the internal java memory for variables and so on. You can set a minimum and a maximum with -Xms and -Xmx. Both are set by default to 512M for solr, and this is enough for most dovecot installations. If you get out of memory errors in the solr log, then set -Xmx higher. If the heap usage is always over 512M, then set -Xms higher. Easy.
Solr/lucene uses memory mapping for the search index. So it relies on the unix filesystem cache to keep hot data in memory. So solr is fast, if all of your frequently queried solr index data stay in the filesystem cache. Putting all your memory in the Java heap via -Xmx/-Xms slows down solr, because there is no memory left for the filesystem cache and index data have to be read from disk over and over again.
Considering that my system is tight on memory, I can not use this, unless memory is released immediately after the index is created. -- Cheers / Saludos, Carlos E. R. (from openSUSE, Leap 15.1 x86_64 (ssd-test)) -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse+owner@opensuse.org
participants (4)
-
Carlos E. R. -
Charles Philip Chan -
Florian Gleixner -
L A Walsh