Hello guys,
as of now we have mlocate [1] package in factory do you think it would be wise to remove the findutils-locate as whole to only keep this implementation which is faster and more reliable around?
When I checked other distros like fedora [2] or gentoo they both patch it out.
Cheers
Tom
[1] https://build.opensuse.org/package/view_file?expand=1&file=mlocate.spec&... [2] http://pkgs.fedoraproject.org/cgit/findutils.git/tree/findutils-4.4.0-no-loc...
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2013-06-12 15:39, Tom￡? Chv￡tal wrote:
Hello guys,
as of now we have mlocate [1] package in factory do you think it would be wise to remove the findutils-locate as whole to only keep this implementation which is faster and more reliable around?
I wanted to try this, but I see only the 32bit rpm for 12.3 at "server:search". Any chance someone building the 54 bit version for 12.3?
- -- Cheers / Saludos,
Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
On Wednesday 2013-06-12 15:39, Tomáš Chvátal wrote:
Hello guys,
as of now we have mlocate [1] package in factory do you think it would be wise to remove the findutils-locate as whole to only keep this implementation which is faster and more reliable around?
There is no tempting reason (license issues; fails to compile for 60+ days and package abandoned) to delete the package at this time.
That of course does not impede _changing_ whatever preference of other packages or yast installation patterns, etc.
Dne St 12. června 2013 16:10:57, Jan Engelhardt napsal(a):
On Wednesday 2013-06-12 15:39, Tomáš Chvátal wrote:
Hello guys,
as of now we have mlocate [1] package in factory do you think it would be wise to remove the findutils-locate as whole to only keep this implementation which is faster and more reliable around?
There is no tempting reason (license issues; fails to compile for 60+ days and package abandoned) to delete the package at this time.
Good point, it indeed is not dead nor failing to build, but technically if other distros disable it we might be the only ones having the codepath and thus seeing and dealing with the bugs.
That of course does not impede _changing_ whatever preference of other packages or yast installation patterns, etc.
This should be enough, will look on the patterns, is there something else defining the to-install lists apart from it?
Also I would like to notify the current users of the findutils-locate that they can use the actually better implementation is that somehow possible?
Cheers
Tom
On Wednesday 2013-06-12 16:19, Tomáš Chvátal wrote:
That of course does not impede _changing_ whatever preference of other packages or yast installation patterns, etc.
This should be enough, will look on the patterns, is there something else defining the to-install lists apart from it?
Also I would like to notify the current users of the findutils-locate that they can use the actually better implementation is that somehow possible?
- news.opensuse.org - release-notes-openSUSE package
but there are not so many communication channels. Those who need, want or prefer mlocate will install it and relay it by word of mouth.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2013-06-12 18:48, Jan Engelhardt wrote:
On Wednesday 2013-06-12 16:19, Tomáš Chvátal wrote:
Also I would like to notify the current users of the findutils-locate that they can use the actually better implementation is that somehow possible?
- news.opensuse.org - release-notes-openSUSE package
but there are not so many communication channels. Those who need, want or prefer mlocate will install it and relay it by word of mouth.
Yes, but please, make it available for people not using factory.
- -- Cheers / Saludos,
Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On 2013-06-12 18:48, Jan Engelhardt wrote:
On Wednesday 2013-06-12 16:19, Tomáš Chvátal wrote:
but there are not so many communication channels. Those who need, want or prefer mlocate will install it and relay it by word of mouth.
It has interesting features:
A new locate implementation. The m character stands for merging, because updatedb reuses the existing database to avoid re-reading most of the file system.
User must be member of locate group in order to use this package.
An improvement would be that locate finds only those files for which the user running it has permissions.
Ie, if I'm plain user john:users, I should have permissions to use locate on my "/home/john" directory, but not those of /root, and maybe yes, maybe not, those of "/home/ian", depending on group permissions.
- -- Cheers / Saludos,
Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Carlos E. R. wrote:
Ie, if I'm plain user john:users, I should have permissions to use locate on my "/home/john" directory, but not those of /root, and maybe yes, maybe not, those of "/home/ian", depending on group permissions.
--- I would say it should let you see the same files you can see with 'ls'.
If I am not root, and ls /root, I can see nothing.
OTOH, Just because I can't write to files in /usr/bin or whatever, doesn't mean I wouldn't want them to show up in a locate. Even if the files are unreadable -- if they are in a directory that is readable, then their names should appear on locate.
BTW I don't know if I remember seeing mlocate, but the permissions thing sounds familiar -- isn't it fairly buggy -- i.e it doesn't show you files you could otherwise do a find on?
On 2013-06-14 11:33, Linda Walsh wrote:
Carlos E. R. wrote:
Ie, if I'm plain user john:users, I should have permissions to use locate on my "/home/john" directory, but not those of /root, and maybe yes, maybe not, those of "/home/ian", depending on group permissions.
I would say it should let you see the same files you can seewith 'ls'.
That's my meaning.
On 2013-06-14 12:08, Carlos E. R. wrote:
On 2013-06-14 11:33, Linda Walsh wrote:
Carlos E. R. wrote:
Ie, if I'm plain user john:users, I should have permissions to use locate on my "/home/john" directory, but not those of /root, and maybe yes, maybe not, those of "/home/ian", depending on group permissions.
I would say it should let you see the same files you can seewith 'ls'.
That's my meaning.
Surprise :-)
locate /root/Documents is empty as user, full as root.
So the feature is there already! :-)
Carlos E. R. wrote:
On 2013-06-12 18:48, Jan Engelhardt wrote:
On Wednesday 2013-06-12 16:19, Tomáš Chvátal wrote:
but there are not so many communication channels. Those who need, want or prefer mlocate will install it and relay it by word of mouth.
It has interesting features: [...] User must be member of locate group in order to use this package.
What's the background of that requirement?
cu Ludwig
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tuesday, 2013-06-18 at 08:50 +0200, Ludwig Nussel wrote:
Carlos E. R. wrote:
On 2013-06-12 18:48, Jan Engelhardt wrote:
On Wednesday 2013-06-12 16:19, Tomáš Chvátal wrote:
but there are not so many communication channels. Those who need, want or prefer mlocate will install it and relay it by word of mouth.
It has interesting features: [...] User must be member of locate group in order to use this package.
What's the background of that requirement?
I don't know how the designers decided on that, but I find useful that the search can run as root, locate everything on the system, yet not everybody can run the search, only those that are authorized.
Even more, a user can only locate those files that he could reach with a normal find whith his permissions.
Look:
Telcontar:~ # locate testing_manpage.log /home/cer2/Documents/testing_manpage.log Telcontar:~ #
cer@Telcontar:~> locate testing_manpage.log cer@Telcontar:~>
My user can not locate a file that belongs to another user, but root can.
- -- Cheers, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Carlos E. R. wrote:
Telcontar:~ # locate testing_manpage.log /home/cer2/Documents/testing_manpage.log Telcontar:~ #
cer@Telcontar:~> locate testing_manpage.log cer@Telcontar:~> My user can not locate a file that belongs to another user, but root can.
---- Are you using mlocate?
So is it that you cannot read testing_manpage.log, or is it that you can't read the directory where it is at?
I.e. could you do an 'ls' or a 'find' and find that file? (but not be able to read it), or is it not findable?
How's the speed compared to locate (not that locate is fast, mind you, but slower would be bad). I'd thought about multithreading locate to bring up the speed a bit.
After downloading a new set of rpms, I usually expire the older versions...found that can be sped up by using multiple clients...
remove-oldver-rpms-in-dir.pl
Read 22276 rpm names. Use 9 procs w/2477 items/process #pkgs=21330, #deletes=946, total=22276 Recycling 946 duplicates...Done Cumulative This Phase ID 0.000s 0.000s Init 0.000s 0.000s start_program 0.062s 0.062s starting_children 0.070s 0.008s end_starting_children 51.704s 51.634s endRdFrmChldrn_n_start_re_sort 56.315s 4.611s afterFinalSort 56.665s 0.350s afterRecycle
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tuesday, 2013-06-18 at 16:17 -0700, Linda Walsh wrote:
Carlos E. R. wrote:
Telcontar:~ # locate testing_manpage.log /home/cer2/Documents/testing_manpage.log Telcontar:~ #
cer@Telcontar:~> locate testing_manpage.log cer@Telcontar:~>
My user can not locate a file that belongs to another user, but root can.
Are you using mlocate?
Yes, of course. mlocate-0.26-14.1.x86_64
So is it that you cannot read testing_manpage.log, or is it that you can't read the directory where it is at?
I can not browse the directory.
cer@Telcontar:~> l /home/cer2/Documents/testing_manpage.log ls: cannot access /home/cer2/Documents/testing_manpage.log: Permission denied cer@Telcontar:~>
Telcontar:/home/cer # l /home/cer2/Documents/testing_manpage.log - -rw-r--r-- 1 cer2 users 14410 Feb 9 2008 /home/cer2/Documents/testing_manpage.log Telcontar:/home/cer #
I.e. could you do an 'ls' or a 'find' and find that file? (but not be able to read it), or is it not findable?
Nope, I can not use a find. mlocate is behaving like find would behave.
How's the speed compared to locate (not that locate is fast, mind you, but slower would be bad). I'd thought about multithreading locate
to
bring up the speed a bit.
Locate is instant. I can not compare both mlocate and locate because both have the same binary name, can not install both simultanously.
cer@Telcontar:~> time locate testing_manpage.log
real 0m0.792s user 0m0.776s sys 0m0.006s cer@Telcontar:~>
Multithreading is useless if you have to seek a 30MB data file for a name. And the seek is not cached:
cer@Telcontar:~> time locate testing_manpage.log
real 0m0.769s user 0m0.762s sys 0m0.006s cer@Telcontar:~> time locate testing_manpage.log
real 0m0.769s user 0m0.759s sys 0m0.009s cer@Telcontar:~>
- -- Cheers, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Carlos E. R. wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Tuesday, 2013-06-18 at 16:17 -0700, Linda Walsh wrote:
Carlos E. R. wrote:
Telcontar:~ # locate testing_manpage.log /home/cer2/Documents/testing_manpage.log Telcontar:~ #
cer@Telcontar:~> locate testing_manpage.log cer@Telcontar:~>
My user can not locate a file that belongs to another user, but root can.
Are you using mlocate?Yes, of course. mlocate-0.26-14.1.x86_64
So is it that you cannot read testing_manpage.log, or is it that youcan't read the directory where it is at?
I can not browse the directory.
cer@Telcontar:~> l /home/cer2/Documents/testing_manpage.log ls: cannot access /home/cer2/Documents/testing_manpage.log: Permission denied cer@Telcontar:~>
Telcontar:/home/cer # l /home/cer2/Documents/testing_manpage.log
- -rw-r--r-- 1 cer2 users 14410 Feb 9 2008
/home/cer2/Documents/testing_manpage.log Telcontar:/home/cer #
I.e. could you do an 'ls' or a 'find' and find that file? (but notbe able to read it), or is it not findable?
Nope, I can not use a find. mlocate is behaving like find would behave.
How's the speed compared to locate (not that locate is fast, mindyou, but slower would be bad). I'd thought about multithreading locate
to
bring up the speed a bit.
Locate is instant. I can not compare both mlocate and locate because both have the same binary name, can not install both simultanously.
cer@Telcontar:~> time locate testing_manpage.log
real 0m0.792s user 0m0.776s sys 0m0.006s cer@Telcontar:~>
Multithreading is useless if you have to seek a 30MB data file for a name. And the seek is not cached:
cer@Telcontar:~> time locate testing_manpage.log
real 0m0.769s user 0m0.762s sys 0m0.006s cer@Telcontar:~> time locate testing_manpage.log
real 0m0.769s user 0m0.759s sys 0m0.009s cer@Telcontar:~> ----------------
You see a 30MB file, I see
30 units of a 1MB stripe size. Spindle-numbers are to disks as cores are to cpu's.
If you have a 1MB stripe size, 5 spindles, now you can divide that up into 5 threads of 6MB searching each. You won't get a 5x speed up, but you'll likely get at least 2-3x. Besides, I doubt disk I/O is the limiting factor. Looking at your timings -- it's all spent in user space and is fairly constant for 3 runs -- indicating your datafile is likely already cached But at .7s, it's likely not worth it. Vs.:
time locate testing_manpage.log
11.10sec 11.00usr 0.06sys (99.72% cpu) Ishtar:law> time locate testing_manpage.log 11.00sec 10.97usr 0.03sys (100.00% cpu)
locate /|wc -l
8574395
How many files are in your locate DB? That might give a better indication of speed...
If you have 5-10M files, and are taking .7s to search through them all... that's pretty impressive, vs.
vs. 500-800K files? Might be similar perf...?
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wednesday, 2013-06-19 at 04:57 -0700, Linda Walsh wrote:
Carlos E. R. wrote:
You see a 30MB file, I see
30 units of a 1MB stripe size. Spindle-numbers are to disks as cores are to cpu's.
Yes, but still the kernel provides one read operation to the process accessing a single file. Reads are sequential, even if the kernel reads the 30 blocks at once, simultaneously. To read and scan several blocks from a single process, you actually have to do parallell processing at the application level, to split the job on several cores. This is not automatic, I know no language that does that.
If you have a 1MB stripe size, 5 spindles, now you can divide that up into 5 threads of 6MB searching each. You won't get a 5x speed up, but you'll likely get at least 2-3x. Besides, I doubt disk I/O is the limiting factor. Looking at your timings -- it's all spent in user space and is fairly constant for 3 runs -- indicating your datafile is likely already cached But at .7s, it's likely not worth it. Vs.:
Yes, it is probably cached.
time locate testing_manpage.log
11.10sec 11.00usr 0.06sys (99.72% cpu) Ishtar:law> time locate testing_manpage.log 11.00sec 10.97usr 0.03sys (100.00% cpu)
locate /|wc -l
8574395
How many files are in your locate DB?
No idea how to find out. It is not a text file, line count doesn't tell it. The old locate database maybe.
Telcontar:~ # l /var/lib/locatedb /var/lib/mlocate/mlocate.db - -rw-r--r-- 1 root root 27285793 Jun 12 00:16 /var/lib/locatedb - -rw-r----- 1 root locate 29122515 Jun 19 22:24 /var/lib/mlocate/mlocate.db Telcontar:~ # file /var/lib/locatedb /var/lib/mlocate/mlocate.db /var/lib/locatedb: GNU findutils locate database data, format 02 (frcode) /var/lib/mlocate/mlocate.db: data Telcontar:~ #
Telcontar:~ # locate * | wc -l 1293562 Telcontar:~ # cd
- -- Cheers, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Tomáš Chvátal wrote:
Dne Út 18. června 2013 08:50:42, Ludwig Nussel napsal(a):
What's the background of that requirement?
cu Ludwig
Because the security didn't review the sgid bit on the mlocate for 2 years. The alternative is this.
---- Um... security did this?
Before, with the sgid bit set, no one could read the values in the mlocatedb. Now anyone who is in the group locate (which is anyone who wants to use locate), can read that file.
The purpose of mlocate was to provide privacy that other users wouldn't be able to use to list the filenames.
It seems like removing the sgid bit removes that feature -- which is the main reason for using mlocate.
If mlocate is shipping without the sgid bit, then I don't see any reason why it should replace findutils-locate. What does it offer? On the downside -- everyone has to be in group locate now to retain functionality.
Linda Walsh wrote:
Tomáš Chvátal wrote:
Dne Út 18. června 2013 08:50:42, Ludwig Nussel napsal(a):
What's the background of that requirement?
cu Ludwig
Because the security didn't review the sgid bit on the mlocate for 2 years. The alternative is this.
Um... security did this? Before, with the sgid bit set, no one could read thevalues in the mlocatedb. Now anyone who is in the group locate (which is anyone who wants to use locate), can read that file.
---- Given that security decided to ship mlocate without the security bit,
wouldn't it be more practical to ensure that it's database is generated with the "--require-visibility no" option and have locate set to normal 755 permissions and the db file set to 644.
That would give the benefit of compatibility with locate and no false impression of security. It seems to give a speed up in small cases, but a slowdown on larger ones (locate /|wc -l) took about 150% longer (2.5x), but a single file was about 50% faster (.5x).
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Wednesday, 2013-06-19 at 06:47 -0700, Linda Walsh wrote:
Linda Walsh wrote:
Because the security didn't review the sgid bit on the mlocate for 2 years. The alternative is this.
Um... security did this? Before, with the sgid bit set, no one could read thevalues in the mlocatedb. Now anyone who is in the group locate (which is anyone who wants to use locate), can read that file.
Anybody in the group can read the file, but they can only locate the files they have permissions to find. Thus, a guest can not find my personal files.
This is good.
- -- Cheers, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
Tomáš Chvátal wrote:
Dne Út 18. června 2013 08:50:42, Ludwig Nussel napsal(a):
What's the background of that requirement?
Because the security didn't review the sgid bit on the mlocate for 2 years. The alternative is this.
See the only mlocate bug in bugzie about it.
That is not an explanation. I had to read the source to understand what mlocate uses the setgid bit for. It's an interesting approach but bears the risk of information leaks or worse (set[ug]id is always fishy). Bonus points for not being installed by default aside safe defaults for such a tool would be to run the indexer unprivileged to be absolutely sure the DB only ever contains files that are world readable anyways.
cu Ludwig
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
On Thursday, 2013-06-20 at 14:25 +0200, Ludwig Nussel wrote:
That is not an explanation. I had to read the source to understand what mlocate uses the setgid bit for. It's an interesting approach but bears the risk of information leaks or worse (set[ug]id is always fishy). Bonus points for not being installed by default aside safe defaults for such a tool would be to run the indexer unprivileged to be absolutely sure the DB only ever contains files that are world readable anyways.
I actually prefer the way it is now.
The database has knowledge of every file, from all users.
However, when I ask locate for a file, it will only give me the files for which I have permissions.
This way, locate finds all my home files, something that is highly interesting for me, and will not find those files belonging to a different user, for which I have no permission.
Root finds all.
It's perfect! :-)
- -- Cheers, Carlos E. R. (from 12.3 x86_64 "Dartmouth" at Telcontar)
* Dne Středa 12. červen 2013, 16:19:30 [CEST] Tomáš Chvátal napsal:
Dne St 12. června 2013 16:10:57, Jan Engelhardt napsal(a):
On Wednesday 2013-06-12 15:39, Tomáš Chvátal wrote:
Hello guys,
as of now we have mlocate [1] package in factory do you think it would be wise to remove the findutils-locate as whole to only keep this implementation which is faster and more reliable around?
There is no tempting reason (license issues; fails to compile for 60+ days and package abandoned) to delete the package at this time.
Good point, it indeed is not dead nor failing to build, but technically if other distros disable it we might be the only ones having the codepath and thus seeing and dealing with the bugs.
That of course does not impede _changing_ whatever preference of other packages or yast installation patterns, etc.
This should be enough, will look on the patterns, is there something else defining the to-install lists apart from it?
Also I would like to notify the current users of the findutils-locate that they can use the actually better implementation is that somehow possible?
If you want to notify all users with findutils-locate installed, you can write an update message for findutils-locate (via /var/adm/update-messages) Once the update of the package is happening, the message will be displayed to the users. But I guess it would be a pretty ugly way to advertise a package alternative.
Tomáš Chvátal wrote:
Hello guys,
as of now we have mlocate [1] package in factory do you think it would be wise to remove the findutils-locate as whole to only keep this implementation which is faster and more reliable around?
When I checked other distros like fedora [2] or gentoo they both patch it out.
Considering its not even an option for suse, it seems a bit sudden to ask for it replacing findutils-locate.
Is it possible to have it released first so we could try it?
naw... I'm sure it's script and feature compatible, so why would anyone notice?
Besides, wasn't findutils going to be absorbed by systemd?
-
Carlos E. R. -
Carlos E. R. -
Jan Engelhardt -
Linda Walsh -
Ludwig Nussel -
Tomáš Chvátal -
Vitezslav Cizek