tracker no longer a CPU consumer ? (was: [opensuse-factory] "Automatic" desktop selection)
Hi,
On Thu, Apr 20, 2017 at 4:47 PM, Frederic Crozat
1) High CPU consuming applications as for example the gnome-tracker
First, it is called tracker (it is a fdo project, not a GNOME one) and it has not been a CPU consumer for years now (it runs only on idle time).
I have a counter-point for that. tracker has been chewing my CPU over and over again. As we speak, /usr/lib/tracker-miner-fs has 100% CPU usage, and has been that way for some time. $ tracker status Currently indexed: 461158 files, 165743 folders Remaining space on database partition: 88.6 GB (16.51%) Data is still being indexed: Estimated less than one second left $ tracker daemon status Store: 20 Apr 2017, 17:34:56: ✓ Store - Idle Miners: 20 Apr 2017, 17:34:56: ✓ Userguides - Idle 20 Apr 2017, 17:34:56: 1% File System - Crawling recursively directory 'file:///home/robert/Documents' 20 Apr 2017, 17:34:56: ✓ Extractor - Idle 20 Apr 2017, 17:34:56: ✓ Applications - Idle Needless to say, the 'less than one second left' and '1%' progress claims never change. I've tried removing the tracker database, resetting preferences, but I'm still left with a runaway tracker indexer unless I manually stop it. What am I doing wrong? Thanks, Robert -- http://robert.muntea.nu/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 20 Apr 2017 17:37:38 +0300, Robert Munteanu
Hi,
On Thu, Apr 20, 2017 at 4:47 PM, Frederic Crozat
wrote: 1) High CPU consuming applications as for example the gnome-tracker
First, it is called tracker (it is a fdo project, not a GNOME one) and it has not been a CPU consumer for years now (it runs only on idle time).
I have a counter-point for that. tracker has been chewing my CPU over and over again. As we speak, /usr/lib/tracker-miner-fs has 100% CPU usage, and has been that way for some time.
$ tracker status Currently indexed: 461158 files, 165743 folders Remaining space on database partition: 88.6 GB (16.51%) Data is still being indexed: Estimated less than one second left
$ tracker daemon status Store: 20 Apr 2017, 17:34:56: ✓ Store - Idle
Miners: 20 Apr 2017, 17:34:56: ✓ Userguides - Idle 20 Apr 2017, 17:34:56: 1% File System - Crawling recursively directory 'file:///home/robert/Documents' 20 Apr 2017, 17:34:56: ✓ Extractor - Idle 20 Apr 2017, 17:34:56: ✓ Applications - Idle
Needless to say, the 'less than one second left' and '1%' progress claims never change. I've tried removing the tracker database, resetting preferences, but I'm still left with a runaway tracker indexer unless I manually stop it.
I don't see that load. Both systems running Plasma5 Linux 4.10.9-1-default [openSUSE Tumbleweed 20170418] HP EliteBook 8560p/1618 Core(TM) i7-2620M CPU @ 2.70GHz/3200(4) x86_64 7931 Mb $ tracker status Currently indexed: 107 files, 38 folders Remaining space on database partition: 118.9 GB (24.27%) All data miners are idle, indexing complete $ tracker daemon status Store: 20 Apr 2017, 16:43:41: ✓ Store - Idle Miners: 20 Apr 2017, 16:43:41: ✓ File System - Idle 20 Apr 2017, 16:43:41: ✓ Userguides - Idle 20 Apr 2017, 16:43:41: ✓ Extractor - Idle 20 Apr 2017, 16:43:41: ✓ Applications - Idle Linux 4.1.39-53-default [openSUSE Leap 42.1 (x86_64)] HP Z420/1589 Xeon(R) CPU E5-1650 0 @ 3.20GHz/1521(12) x86_64 15974 Mb $ tracker status Currently indexed: 1204 files, 161 folders Remaining space on database partition: 672.6 GB (70.72%) All data miners are idle, indexing complete $ tracker daemon status Store: 20 Apr 2017, 16:44:26: ✓ Store - Idle Miners: 20 Apr 2017, 16:44:26: ✓ Userguides - Idle 20 Apr 2017, 16:44:26: ✓ File System - Idle 20 Apr 2017, 16:44:26: ✓ Extractor - Idle 20 Apr 2017, 16:44:26: ✓ Applications - Idle
What am I doing wrong?
Robert
-- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.25 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
Hi,
On Thu, Apr 20, 2017 at 5:45 PM, H.Merijn Brand
$ tracker status Currently indexed: 1204 files, 161 folders
Currently indexed: 461158 files, 165743 folders There's a slight difference between what is indexed on your machine and what is indexed on mine :-) And my claim was not that it's a CPU hog for everyone. I merely countered the claim that is no longer a CPU hog - which implied that it's fast for everyone. Robert -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2017-04-20 16:50, Robert Munteanu wrote:
Hi,
On Thu, Apr 20, 2017 at 5:45 PM, H.Merijn Brand
wrote: $ tracker status Currently indexed: 1204 files, 161 folders
Currently indexed: 461158 files, 165743 folders
There's a slight difference between what is indexed on your machine and what is indexed on mine :-)
And my claim was not that it's a CPU hog for everyone. I merely countered the claim that is no longer a CPU hog - which implied that it's fast for everyone.
To monitor 165743 directories with inotify, it would need that many file descriptors. Normal sessions however only have space for 1024, so I would expect that it then uses some form of repeated polling. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 20 April 2017, Jan Engelhardt wrote:
On Thursday 2017-04-20 16:50, Robert Munteanu wrote:
Hi,
On Thu, Apr 20, 2017 at 5:45 PM, H.Merijn Brand
wrote: $ tracker status Currently indexed: 1204 files, 161 folders
Currently indexed: 461158 files, 165743 folders
There's a slight difference between what is indexed on your machine and what is indexed on mine :-)
And my claim was not that it's a CPU hog for everyone. I merely countered the claim that is no longer a CPU hog - which implied that it's fast for everyone.
To monitor 165743 directories with inotify, it would need that many file descriptors.
Normal sessions however only have space for 1024,
I have this default $ sysctl fs.inotify.max_user_watches fs.inotify.max_user_watches = 65536 cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 2017-04-20 18:48, Ruediger Meier wrote:
Currently indexed: 461158 files, 165743 folders To monitor 165743 directories with inotify, it would need that many file descriptors. Normal sessions however only have space for 1024,
I have this default $ sysctl fs.inotify.max_user_watches fs.inotify.max_user_watches = 65536
But you would also need to bump `ulimit -n` (that is where the 1024 come from currently). -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thursday 20 April 2017, Jan Engelhardt wrote:
On Thursday 2017-04-20 18:48, Ruediger Meier wrote:
Currently indexed: 461158 files, 165743 folders
To monitor 165743 directories with inotify, it would need that many file descriptors. Normal sessions however only have space for 1024,
I have this default $ sysctl fs.inotify.max_user_watches fs.inotify.max_user_watches = 65536
But you would also need to bump `ulimit -n` (that is where the 1024 come from currently).
Isn't `ulimit -n` to limit "open files"? Here they probably want to inotify_add_watch(2) rather than open(2) all files at the same time. The inotify(7) manpage explains which limits are relevant. cu, Rudi -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 20 Apr 2017 17:50:04 +0300, Robert Munteanu
Hi,
On Thu, Apr 20, 2017 at 5:45 PM, H.Merijn Brand
wrote: $ tracker status Currently indexed: 1204 files, 161 folders
Currently indexed: 461158 files, 165743 folders
There's a slight difference between what is indexed on your machine and what is indexed on mine :-)
And my claim was not that it's a CPU hog for everyone. I merely countered the claim that is no longer a CPU hog - which implied that it's fast for everyone.
Robert
As it never struck me as a resource hog, I never disabled it. I did disable some other stuff that I didn't use that was a resource hog, like akonadi TW $ locate -r . | wc -l 1064447 42.1 $ locate -r . | wc -l 1840435 42.2 $ locate -r . | wc -l 1895987 As I am unaware of using tracker for some day-to-day work, I never changed any settings -- H.Merijn Brand http://tux.nl Perl Monger http://amsterdam.pm.org/ using perl5.00307 .. 5.25 porting perl5 on HP-UX, AIX, and openSUSE http://mirrors.develooper.com/hpux/ http://www.test-smoke.org/ http://qa.perl.org http://www.goldmark.org/jeff/stupid-disclaimers/
Le jeudi 20 avril 2017 à 17:50 +0300, Robert Munteanu a écrit :
Hi,
On Thu, Apr 20, 2017 at 5:45 PM, H.Merijn Brand
wrote: $ tracker status Currently indexed: 1204 files, 161 folders
Currently indexed: 461158 files, 165743 folders
There's a slight difference between what is indexed on your machine and what is indexed on mine :-)
The defaults in tracker are to index: - $HOME not recursively - XDG_DEFAULT_DIRS (Downloads,Images,Videos,Music,Documents) recursively and that's it.. It might be interesting to look which directories are indexed and how on your system. -- Frederic Crozat Enterprise Desktop Release Manager SUSE -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Apr 20, 2017 at 7:20 PM, Frederic Crozat
It might be interesting to look which directories are indexed and how on your system.
In addition to that I have $HOME/Dropbox indexed recursively. I have set tracker to index file contents ( not sure is that is the default setting ). Robert -- http://robert.muntea.nu/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/20/2017 11:09 AM, Robert Munteanu wrote:
On Thu, Apr 20, 2017 at 7:20 PM, Frederic Crozat
wrote: It might be interesting to look which directories are indexed and how on your system.
In addition to that I have $HOME/Dropbox indexed recursively. I have set tracker to index file contents ( not sure is that is the default setting ).
Robert
Tracker does full text indexing by default, but the GNOME Shell search doesn't actually *use* the full text index for searching: https://bugzilla.gnome.org/show_bug.cgi?id=775961 Nate -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Apr 20, 2017 at 8:13 PM, Nate Graham
On 04/20/2017 11:09 AM, Robert Munteanu wrote:
On Thu, Apr 20, 2017 at 7:20 PM, Frederic Crozat
wrote: It might be interesting to look which directories are indexed and how on your system.
In addition to that I have $HOME/Dropbox indexed recursively. I have set tracker to index file contents ( not sure is that is the default setting ).
Robert
Tracker does full text indexing by default, but the GNOME Shell search doesn't actually *use* the full text index for searching: https://bugzilla.gnome.org/show_bug.cgi?id=775961
Good to know, I will disable the fulltext engine then. Robert
Nate
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-- http://robert.muntea.nu/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/20/2017 12:18 PM, Robert Munteanu wrote:
On Thu, Apr 20, 2017 at 8:13 PM, Nate Graham
wrote: Tracker does full text indexing by default, but the GNOME Shell search doesn't actually *use* the full text index for searching: https://bugzilla.gnome.org/show_bug.cgi?id=775961 Good to know, I will disable the fulltext engine then.
If you like full-text searching, then instead of the GNOME Shell search, you can use the tracker-needle GUI program, Synapse, Cerebro or KDE's KRunner, all of which behave properly and return full-text search results. Nate -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2017-04-20 20:36, Nate Graham wrote:
On 04/20/2017 12:18 PM, Robert Munteanu wrote:
On Thu, Apr 20, 2017 at 8:13 PM, Nate Graham
wrote: Tracker does full text indexing by default, but the GNOME Shell search doesn't actually *use* the full text index for searching: https://bugzilla.gnome.org/show_bug.cgi?id=775961 Good to know, I will disable the fulltext engine then.
If you like full-text searching, then instead of the GNOME Shell search, you can use the tracker-needle GUI program, Synapse, Cerebro or KDE's KRunner, all of which behave properly and return full-text search results.
I'm on XFCE. Just installed Synapse to try. I get a button on the panel in the proper place. Nice. But I search for "bank" or "openSUSE", and it finds or displays a single document. Not nice. I search in YaST for "Cerebro", finds nothing. Webpin also finds nothing. -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
On 04/20/2017 02:26 PM, Carlos E. R. wrote:
I'm on XFCE. Just installed Synapse to try. I get a button on the panel in the proper place. Nice.
But I search for "bank" or "openSUSE", and it finds or displays a single document. Not nice.
I believe Synapse uses the Zeitgeist indexing engine, so it may not have finished indexing yet. Or there might be a bug that needs investigating.
I search in YaST for "Cerebro", finds nothing. Webpin also finds nothing.
It's not available in the openSUSE repos (boo). But here's the code and the marketing page: https://github.com/KELiON/cerebro/ https://cerebroapp.com/ Nate -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 2017-04-21 15:25, Nate Graham wrote:
On 04/20/2017 02:26 PM, Carlos E. R. wrote:
I'm on XFCE. Just installed Synapse to try. I get a button on the panel in the proper place. Nice.
But I search for "bank" or "openSUSE", and it finds or displays a single document. Not nice.
I believe Synapse uses the Zeitgeist indexing engine, so it may not have finished indexing yet. Or there might be a bug that needs investigating.
It did not request installing anything else, and zeitgeist*rpm has been installed for months. If I type my home city name then it finds dozens of documents. Then only one. Ah, I see. You have to press the down arrow key to see the list of files really found. But when I go to another window, the synapse window disappears: I have to type again the search text to see the list again. It is not highly usable. But I can not find the default system search in the menu, either :-(, except by typing search on synapse which then finds for me the "desktop search" tool, which finds also files for my city name. It is not broken here. :-?
I search in YaST for "Cerebro", finds nothing. Webpin also finds nothing.
It's not available in the openSUSE repos (boo). But here's the code and the marketing page:
I'll try have a look later, I have to go out now. Thanks :-) -- Cheers / Saludos, Carlos E. R. (from 42.2 x86_64 "Malachite" at Telcontar)
Il 20/04/2017 16:45, H.Merijn Brand ha scritto:
$ tracker status Currently indexed: 461158 files, 165743 folders ... $ tracker status Currently indexed: 107 files, 38 folders
Numbers explain all, or almost all.. Daniele. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/20/2017 10:08 AM, Daniele wrote:
Il 20/04/2017 16:45, H.Merijn Brand ha scritto:
$ tracker status Currently indexed: 461158 files, 165743 folders ... $ tracker status Currently indexed: 107 files, 38 folders
Numbers explain all, or almost all..
Daniele.
Tracker is known to bog down with large indexes; a tracker developer explained this to me himself when I submitted a patch to make it index the files in git repos by default: https://bugzilla.gnome.org/show_bug.cgi?id=775949 It's a known issue, not a controversy. Nate -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
Il 20/04/2017 18:11, Nate Graham ha scritto:
On 04/20/2017 10:08 AM, Daniele wrote:
Il 20/04/2017 16:45, H.Merijn Brand ha scritto:
$ tracker status Currently indexed: 461158 files, 165743 folders ... $ tracker status Currently indexed: 107 files, 38 folders
Numbers explain all, or almost all..
Daniele.
Tracker is known to bog down with large indexes; a tracker developer explained this to me himself when I submitted a patch to make it index the files in git repos by default: https://bugzilla.gnome.org/show_bug.cgi?id=775949
It's a known issue, not a controversy.
Nate
But bug or not, you can't compare 461158 to 107, IMHO.. Daniele. -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/20/2017 10:16 AM, Daniele wrote:
Il 20/04/2017 18:11, Nate Graham ha scritto:
On 04/20/2017 10:08 AM, Daniele wrote:
Il 20/04/2017 16:45, H.Merijn Brand ha scritto:
$ tracker status Currently indexed: 461158 files, 165743 folders ... $ tracker status Currently indexed: 107 files, 38 folders
Numbers explain all, or almost all..
Daniele.
Tracker is known to bog down with large indexes; a tracker developer explained this to me himself when I submitted a patch to make it index the files in git repos by default: https://bugzilla.gnome.org/show_bug.cgi?id=775949
It's a known issue, not a controversy.
Nate
But bug or not, you can't compare 461158 to 107, IMHO.. Daniele.
Right. The point is, under heavy load, Tracker is known by its developers to stop performing well. Any file indexer should be able to index 107 files without breaking a sweat. That's easy. The real test is indexing a large file set that's representative of how much stuff most people actually put in their computers. FWIW I'm using Plasma with the competing Baloo framework, and it performs flawlessly with my data set: $ balooctl status Baloo File Indexer is running Indexer state: Idle Indexed 23999 / 23999 files By contrast, Tracker started to have trouble with this data set, and would frequently max out a CPU core and also simply miss inxeding many files. It was a frustrating experience. Nate -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Apr 20, 2017 at 7:21 PM, Nate Graham
Right. The point is, under heavy load, Tracker is known by its developers to stop performing well.
And that is a very valuable point for me. I will probably need to restrict the way my Documents folder is structured to avoid tracker getting bogged down. -- http://robert.muntea.nu/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On 04/20/2017 11:01 AM, Robert Munteanu wrote:
On Thu, Apr 20, 2017 at 7:21 PM, Nate Graham
wrote: Right. The point is, under heavy load, Tracker is known by its developers to stop performing well.
And that is a very valuable point for me. I will probably need to restrict the way my Documents folder is structured to avoid tracker getting bogged down.
Or you could use something superior to Tracker... ;) Nate -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, Apr 20, 2017 at 8:03 PM, Nate Graham
On 04/20/2017 11:01 AM, Robert Munteanu wrote:
On Thu, Apr 20, 2017 at 7:21 PM, Nate Graham
wrote: Right. The point is, under heavy load, Tracker is known by its developers to stop performing well.
And that is a very valuable point for me. I will probably need to restrict the way my Documents folder is structured to avoid tracker getting bogged down.
Or you could use something superior to Tracker... ;)
I am not opposed to that, as long as it has gnome shell integration. I got used to stop thinking about what I need to search and started to just open gnome shell and type. Robert
Nate
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
-- http://robert.muntea.nu/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
On Thu, 2017-04-20 at 20:01 +0300, Robert Munteanu wrote:
On Thu, Apr 20, 2017 at 7:21 PM, Nate Graham
wrote: Right. The point is, under heavy load, Tracker is known by its developers to stop performing well.
And that is a very valuable point for me. I will probably need to restrict the way my Documents folder is structured to avoid tracker getting bogged down.
(I know, I'm late to the game, but still) Just to add here: you can add a .trackerignore file to any directory you do not want to be indexed - so you don't have to change the entire file system layout. I for example ignore all my source checkouts - as much as it could be interesting to do full text search in there, most of the time it's totally useless to do full text search over ALL source repositories I have. Just some thoughts and a way to make your tracker possibly behave nicer. Cheers, Dominique
On Thu, May 25, 2017 at 10:00 AM, Dominique Leuenberger / DimStar
Just to add here: you can add a .trackerignore file to any directory you do not want to be indexed - so you don't have to change the entire file system layout.
I for example ignore all my source checkouts - as much as it could be interesting to do full text search in there, most of the time it's totally useless to do full text search over ALL source repositories I have.
That's a good point, and it's much simpler to automate when using puppet to provision a workstation compared to manually excluding directories using tracker's GUI. Thanks! Robert -- http://robert.muntea.nu/ -- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org
participants (10)
-
Carlos E. R.
-
Carlos E. R.
-
Daniele
-
Dominique Leuenberger / DimStar
-
Frederic Crozat
-
H.Merijn Brand
-
Jan Engelhardt
-
Nate Graham
-
Robert Munteanu
-
Ruediger Meier