https://bugzilla.novell.com/show_bug.cgi?id=435160 Michal Seben changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mseben@novell.com AssignedTo|bnc-team-screening@forge.provo.novell.com |werner@novell.com -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c2 Pavol Rusnak changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |prusnak@novell.com Status|NEW |NEEDINFO Info Provider| |sdelcroix@novell.com --- Comment #2 from Pavol Rusnak 2008-10-14 06:55:40 MDT --- Indeed, it is scout, particularly python-satsolver module, which is out of my scope. But Klaus Kaempf made some significant improvements during last weeks. Stephane: Please add repository http://download.opensuse.org/repositories/home:/prusnak:/scout/openSUSE_Fact... and issue "zypper install scout command-not-found python-satsolver" This should install newer versions of packages (they will be available in beta 3). According to my measurements these are approx. 4x times faster than the ones you are testing. Could you please test and report back? Werner: no need for skipping support. In cause of more serious trouble we might exclude command-not-found package from distribution so no handler will get called by default, but users wanting this feature could install it later. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c4 Pavol Rusnak changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kkaempf@novell.com, dmacvicar@novell.com --- Comment #4 from Pavol Rusnak 2008-10-16 03:26:04 MDT --- 50ms is IMO not possible. Adding Duncan and Klaus as they are responsible for satsolver/python-satsolver and search spends nearly all of time in them (according to profiler). Maybe they could provide more info if this time is possible. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c6 --- Comment #6 from Pavol Rusnak 2008-10-16 03:41:18 MDT --- Python is not used for lookups. Scout just calls repo.search( self.pathre % term, satsolver.SEARCH_REGEX | s atsolver.SEARCH_FILES, None, 'solvable:filelist' ) from python-satsolver, which is wrapped C function from satsolver library. Most (95-99%) of the time is spent in this C code. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User kkaempf@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c8 Klaus Kämpf changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mls@novell.com, matz@novell.com --- Comment #8 from Klaus Kämpf 2008-10-16 03:45:16 MDT --- The most time consuming function is evaluation of the regular expression. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c10 --- Comment #10 from Michael Schröder 2008-10-16 04:30:47 MDT --- Just wait till the library fetches the complete file list from the repositories some day in the future, then this is going to be much slower... If really the regexp implementation is to blame, don't specify a match string but do custom matching instead. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c12 --- Comment #12 from Michael Schröder 2008-10-16 05:09:48 MDT --- First you should find out if it's really the regexp lib that's to blame. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c14 --- Comment #14 from Pavol Rusnak 2008-10-16 08:18:10 MDT --- I suggest everyone to try this feature before commenting. We are looking for commands and their fullpath. We are also able to find commands in user-added repositories. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c16 --- Comment #16 from Dr. Werner Fink 2008-10-16 08:23:52 MDT --- Normally the name space of the command in this path are simply ASCII. What difference is seen in execution time if locale is set hard to POSIX within scout? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mvyskocil@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c18 --- Comment #18 from Michal Vyskocil 2008-10-16 08:29:29 MDT --- Werner: the # -*- coding: utf-8 -*- in __init__.py switchs[1] all of string literals in Python to unicode. [1] http://www.python.org/dev/peps/pep-0263/ -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c20 --- Comment #20 from Pavol Rusnak 2008-10-16 08:32:58 MDT --- Michal: I really do not see the difference by using your patch. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c22 --- Comment #22 from Michael Schröder 2008-10-16 08:55:48 MDT --- Regarding #19: come on, you can do better. As the basename is known, you can do this: repo.search( basename, satsolver.SEARCH_STRING, None, 'solvable:filelist' ) Note the missing SEARCH_FILES, which means that we're just matching basenames. If there's a hit, the iterator value will be the complete filename and you can do your regexp match. This should also speed up things a lot, as the costly regexp match is only done when the basename matches. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c24 --- Comment #24 from Michael Schröder 2008-10-16 10:12:33 MDT --- Argh, we have to make a tiny change in the code for this. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c26 --- Comment #26 from Michael Schröder 2008-10-20 09:40:25 MDT --- Do it like proposed in comment #22. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c28 --- Comment #28 from Michael Schröder 2008-10-21 04:00:38 MDT --- That's surprising. Where's all the time spent? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c29 --- Comment #29 from Pavol Rusnak 2008-10-21 04:45:08 MDT --- Here is hotshot python profiler output: ncalls tottime percall cumtime percall filename:lineno(function) 6 0.552 0.092 0.552 0.092 satsolver.py:93(add_solv) 4 0.066 0.017 0.066 0.017 satsolver.py:972(step) 1 0.020 0.020 0.020 0.020 satsolver.py:946(__init__) 1 0.003 0.003 0.647 0.647 bin.py:82(main) 7 0.002 0.000 0.002 0.000 ConfigParser.py:434(_read) 12/3 0.001 0.000 0.002 0.001 sre_parse.py:385(_parse) 1 0.001 0.001 0.555 0.555 bin.py:25(__init__) 15/3 0.001 0.000 0.001 0.000 sre_compile.py:38(_compile) 116 0.000 0.000 0.001 0.000 sre_parse.py:207(get) 128 0.000 0.000 0.000 0.000 sre_parse.py:188(__next) 17/5 0.000 0.000 0.000 0.000 sre_parse.py:146(getwidth) 1 0.000 0.000 0.651 0.651 scout-cmd.py:10(runscout) 7 0.000 0.000 0.002 0.000 ConfigParser.py:266(read) 108 0.000 0.000 0.000 0.000 sre_parse.py:136(__getitem__) 1 0.000 0.000 0.089 0.089 bin.py:39(search) 109 0.000 0.000 0.000 0.000 sre_parse.py:144(append) 6 0.000 0.000 0.000 0.000 optparse.py:554(__init__) 64 0.000 0.000 0.000 0.000 ConfigParser.py:354(optionxform) 1 0.000 0.000 0.000 0.000 __init__.py:498(format) 4/3 0.000 0.000 0.002 0.001 sre_parse.py:307(_parse_sub) It seems that most of the time is spent by adding solvables. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c31 --- Comment #31 from Michael Schröder 2008-10-21 07:01:00 MDT --- That's probably adding a complete repository, not a single solvable. So it takes half a second to read all 6 repositories? That's pretty slow. Is 'zypper se bash' also that slow? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c33 Michael Schröder changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|mls@novell.com | --- Comment #33 from Michael Schröder 2008-10-21 07:27:17 MDT --- Why not? zypper does pretty much the same, except that it also does some gpg-key handling. We can probably make solv file reading a bit faster, but I don't think we can do more than double the speed. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c35 --- Comment #35 from Pavol Rusnak 2008-10-21 08:48:53 MDT --- SOLV files are read for each completion. Is daemon holding all solvdata in memory (I assume max 20 MBs for typical user) an option? It would also need to detect changes in solv files and reload the data. I could implement it, but I don't know if this approach is reasonable. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c37 --- Comment #37 from Michael Schröder 2008-10-21 09:36:54 MDT --- Do you have any "monster repositories" in your repo list, i.e. repos with more than 30000 solvables? How big are your solv files? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c39 --- Comment #39 from Michael Schröder 2008-10-21 11:09:21 MDT --- Ah, found the reason why it is so slow. I'm to blame for it, the new solv format parser has a bug that makes it move some data much to often. Fixed version is committed to svn... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c41 --- Comment #41 from Michael Schröder 2008-10-22 02:27:27 MDT --- For which solv file? Can you make one of them available to me, please? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User sdelcroix@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c43 --- Comment #43 from Stephane Delcroix 2008-10-23 03:24:33 MDT --- This regressed in beta3, I'm now down to 1.6sec. What about not activating this by default ? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User sdelcroix@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c45 --- Comment #45 from Stephane Delcroix 2008-10-23 04:37:12 MDT --- (In reply to comment #44 from Werner Fink)

Simply do unset command_not_found_handle and be fine, you may add this to your personal ~/.bashrc file

oh, am fine. I was speaking of the distribution default, not my egotist one. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User aj@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c47 Andreas Jaeger changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aj@novell.com --- Comment #47 from Andreas Jaeger 2008-10-23 05:02:24 MDT --- (In reply to comment #15 from Pavol Rusnak)

Regexp for filelists is

^/(s?bin|usr/(s?bin|games)|opt/(kde3|gnome)/s?bin)/%s$

which matches files in

{ /bin, /sbin, /usr/bin, /usr/sbin, /usr/games, /opt/kde3/bin, /opt/kde3/sbin, /opt/gnome/bin, /opt/gnome/sbin }

With openSUSE 11.1 there should be no /opt/gnome/ anymore and there's neither /opt/kde3/sbin So, let's simplify the regex to look only in: { /bin, /sbin, /usr/bin, /usr/sbin, /usr/games, /opt/kde3/bin} What about the following (untested): ^/(s?bin|usr/(s?bin|games)|opt/kde3/bin)/%s$ -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.

https://bugzilla.novell.com/show_bug.cgi?id=435160 User froh@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c49 Susanne Oberhauser changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |froh@novell.com --- Comment #49 from Susanne Oberhauser 2009-02-04 02:54:23 MST --- I've filed a feature request to discuss an alternative, faster approach: https://features.opensuse.org/305803 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.