[Bug 435160] New: bash takes too long on "command not found"
https://bugzilla.novell.com/show_bug.cgi?id=435160 Summary: bash takes too long on "command not found" Product: openSUSE 11.1 Version: Beta 2 Platform: Other OS/Version: Other Status: NEW Severity: Normal Priority: P5 - None Component: Other AssignedTo: bnc-team-screening@forge.provo.novell.com ReportedBy: sdelcroix@novell.com QAContact: qa@suse.de Found By: --- when mistyping a cmd name in bash (or calling an uninstalled app), it takes a few seconds before getting the "Command not found" message and the prompt back. that looked very strange at first sight, Ctrl-C during that showed that bash was running /usr/bin/command-not-found, which looks like being a slow python script. I figured you're trying to do something smart, telling the user how to install the missing application, but that does not even works...
patch -p1 < patch
bash: patch: command not found
I time'd that command... takes 2 full seconds :( -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 Michal Seben <mseben@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mseben@novell.com AssignedTo|bnc-team-screening@forge.provo.novell.com |werner@novell.com -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c1 Dr. Werner Fink <werner@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |werner@novell.com AssignedTo|werner@novell.com |prusnak@novell.com --- Comment #1 from Dr. Werner Fink <werner@novell.com> 2008-10-14 06:19:45 MDT --- IMHO this was introduced by the support for scout. Pavol? Such reports may cause me to skip this support. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c2 Pavol Rusnak <prusnak@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |prusnak@novell.com Status|NEW |NEEDINFO Info Provider| |sdelcroix@novell.com --- Comment #2 from Pavol Rusnak <prusnak@novell.com> 2008-10-14 06:55:40 MDT --- Indeed, it is scout, particularly python-satsolver module, which is out of my scope. But Klaus Kaempf made some significant improvements during last weeks. Stephane: Please add repository http://download.opensuse.org/repositories/home:/prusnak:/scout/openSUSE_Fact... and issue "zypper install scout command-not-found python-satsolver" This should install newer versions of packages (they will be available in beta 3). According to my measurements these are approx. 4x times faster than the ones you are testing. Could you please test and report back? Werner: no need for skipping support. In cause of more serious trouble we might exclude command-not-found package from distribution so no handler will get called by default, but users wanting this feature could install it later. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User sdelcroix@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c3 Stephane Delcroix <sdelcroix@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|sdelcroix@novell.com | --- Comment #3 from Stephane Delcroix <sdelcroix@novell.com> 2008-10-14 07:27:39 MDT --- it's indeed ~3 times faster, topping at 700ms... it's way better, but still far from being acceptable (the acceptation threshold should be at ~50ms) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c4 Pavol Rusnak <prusnak@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |kkaempf@novell.com, dmacvicar@novell.com --- Comment #4 from Pavol Rusnak <prusnak@novell.com> 2008-10-16 03:26:04 MDT --- 50ms is IMO not possible. Adding Duncan and Klaus as they are responsible for satsolver/python-satsolver and search spends nearly all of time in them (according to profiler). Maybe they could provide more info if this time is possible. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c5 --- Comment #5 from Dr. Werner Fink <werner@novell.com> 2008-10-16 03:33:05 MDT --- Hmmm .. instead of using python .. what's about a hash table used with C code. Compare with spell checkers like aspell or the original ispell for speed. With this the hash table is loadable once, e.g. by a user based deamon started by the bash only once on each system providing a socket for scout for the communication. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c6 --- Comment #6 from Pavol Rusnak <prusnak@novell.com> 2008-10-16 03:41:18 MDT --- Python is not used for lookups. Scout just calls repo.search( self.pathre % term, satsolver.SEARCH_REGEX | s atsolver.SEARCH_FILES, None, 'solvable:filelist' ) from python-satsolver, which is wrapped C function from satsolver library. Most (95-99%) of the time is spent in this C code. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User kkaempf@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c7 --- Comment #7 from Klaus Kämpf <kkaempf@novell.com> 2008-10-16 03:43:50 MDT --- The search time heavily depends on the number and the size of enabled repositories. Scout is a Python application using the satsolver-python bindings. The search is a direct call to satsolvers 'dataiterator' interface. All internal satsolver data is hashed. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User kkaempf@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c8 Klaus Kämpf <kkaempf@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |mls@novell.com, matz@novell.com --- Comment #8 from Klaus Kämpf <kkaempf@novell.com> 2008-10-16 03:45:16 MDT --- The most time consuming function is evaluation of the regular expression. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c9 --- Comment #9 from Dr. Werner Fink <werner@novell.com> 2008-10-16 03:59:48 MDT --- Hmm ... regular expressions and UTF-8 locale could cause slowdowns as then wide-character strings are used. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c10 --- Comment #10 from Michael Schröder <mls@novell.com> 2008-10-16 04:30:47 MDT --- Just wait till the library fetches the complete file list from the repositories some day in the future, then this is going to be much slower... If really the regexp implementation is to blame, don't specify a match string but do custom matching instead. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c11 --- Comment #11 from Pavol Rusnak <prusnak@novell.com> 2008-10-16 04:50:23 MDT --- We could still use statically build non-updating SQLite database like in pre-zypp times (10.3 and older). This approach could meet desired times in tens of ms. But we'll miss the files in packages from user-added repos (as .db files contain only distro packages), which is the biggest advantage of using .solv files. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c12 --- Comment #12 from Michael Schröder <mls@novell.com> 2008-10-16 05:09:48 MDT --- First you should find out if it's really the regexp lib that's to blame. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User matz@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c13 --- Comment #13 from Michael Matz <matz@novell.com> 2008-10-16 07:01:38 MDT --- Um, you search in the filelists? That will become extremely slow in the future. An SQLite DB is overkill I think. What's the nature of the lookup? How does the regexp look like? Aren't you just looking for commands, instead of arbitrary files? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c14 --- Comment #14 from Pavol Rusnak <prusnak@novell.com> 2008-10-16 08:18:10 MDT --- I suggest everyone to try this feature before commenting. We are looking for commands and their fullpath. We are also able to find commands in user-added repositories. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c15 --- Comment #15 from Pavol Rusnak <prusnak@novell.com> 2008-10-16 08:19:53 MDT --- Regexp for filelists is ^/(s?bin|usr/(s?bin|games)|opt/(kde3|gnome)/s?bin)/%s$ which matches files in { /bin, /sbin, /usr/bin, /usr/sbin, /usr/games, /opt/kde3/bin, /opt/kde3/sbin, /opt/gnome/bin, /opt/gnome/sbin } -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c16 --- Comment #16 from Dr. Werner Fink <werner@novell.com> 2008-10-16 08:23:52 MDT --- Normally the name space of the command in this path are simply ASCII. What difference is seen in execution time if locale is set hard to POSIX within scout? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mvyskocil@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c17 --- Comment #17 from Michal Vyskocil <mvyskocil@novell.com> 2008-10-16 08:25:55 MDT --- If the problem is in a parsing of UTF-8 regexp, this should be fixed in Python side. I expect, that almost binaries available in packages for openSUSE contains a ascii characters only. So you could try to convert an unicode string term to ascii. If the python-satsolver works with arguments properly (detect the unicode, vs normal strings), this should help. diff --git a/scout/bin.py b/scout/bin.py index 642c104..ded881f 100644 --- a/scout/bin.py +++ b/scout/bin.py @@ -17,7 +17,7 @@ class SolvParser(object): etcpath = '/etc/zypp/repos.d' solvfile = '/var/cache/zypp/solv/%s/solv' # path regular expression for { /bin, /sbin, /usr/bin, /usr/sbin, /usr/games, /opt/kde3/bin, /opt/kde3/sbin, /opt/gnome/bin, /opt/gnome/sbin } - pathre = '^/(s?bin|usr/(s?bin|games)|opt/(kde3|gnome)/s?bin)/%s$' + pathre = '^/(s?bin|usr/(s?bin|games)|opt/(kde3|gnome)/s?bin)/%s$'.encode("ascii") def __init__(self): self.pool = satsolver.Pool() @@ -35,6 +35,10 @@ class SolvParser(object): def search(self, term): pkgmatch = [] + try: + term = term.encode("ascii") + except UnicodeEncodeError: + pass for repo in self.pool.repos(): for d in repo.search( self.pathre % term, satsolver.SEARCH_REGEX | satsolver.SEARCH_FILES, None, 'solvable:filelist' ): path = d.value() -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mvyskocil@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c18 --- Comment #18 from Michal Vyskocil <mvyskocil@novell.com> 2008-10-16 08:29:29 MDT --- Werner: the # -*- coding: utf-8 -*- in __init__.py switchs[1] all of string literals in Python to unicode. [1] http://www.python.org/dev/peps/pep-0263/ -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User matz@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c19 --- Comment #19 from Michael Matz <matz@novell.com> 2008-10-16 08:31:54 MDT --- Thanks. The problem with using sat-solver for matching the filelist is that it is targeted at queries resulting from interactive use (user searching for something via a GUI). It wasn't designed for these kind of automatic queries, especially searching in filelists isn't particularly fast (it needs to construct each entry from a directory list and a basename just to run the regexp on it). We could optimize this a bit further as a special query type (where directory and basename are given separately) but I wouldn't hold my breath. For the time being I either would make this feature optional (and I think it's basically a nice feature), or implement a cache just for the programs that could be looked up faster. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c20 --- Comment #20 from Pavol Rusnak <prusnak@novell.com> 2008-10-16 08:32:58 MDT --- Michal: I really do not see the difference by using your patch. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c21 --- Comment #21 from Pavol Rusnak <prusnak@novell.com> 2008-10-16 08:36:48 MDT --- (In reply to comment #19 from Michael Matz)
basically a nice feature), or implement a cache just for the programs that could be looked up faster.
Users do not usually write the same missing command to console over and over. They install the package providing it as soon as they see the instructions. I don't know if caching is going to help there, because of the usage nature. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c22 --- Comment #22 from Michael Schröder <mls@novell.com> 2008-10-16 08:55:48 MDT --- Regarding #19: come on, you can do better. As the basename is known, you can do this: repo.search( basename, satsolver.SEARCH_STRING, None, 'solvable:filelist' ) Note the missing SEARCH_FILES, which means that we're just matching basenames. If there's a hit, the iterator value will be the complete filename and you can do your regexp match. This should also speed up things a lot, as the costly regexp match is only done when the basename matches. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User matz@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c23 --- Comment #23 from Michael Matz <matz@novell.com> 2008-10-16 09:17:21 MDT --- Oh right. Heh, funny, considering I implemented the iterator :-) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c24 --- Comment #24 from Michael Schröder <mls@novell.com> 2008-10-16 10:12:33 MDT --- Argh, we have to make a tiny change in the code for this. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c25 Pavol Rusnak <prusnak@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |ma@novell.com --- Comment #25 from Pavol Rusnak <prusnak@novell.com> 2008-10-20 09:35:22 MDT --- mls, ma: Klaus mentioned today's commits to satsolver SVN. I'm digging through logs, but could you give me any pointer how to utilize optimizations (if there were any?) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c26 --- Comment #26 from Michael Schröder <mls@novell.com> 2008-10-20 09:40:25 MDT --- Do it like proposed in comment #22. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c27 --- Comment #27 from Pavol Rusnak <prusnak@novell.com> 2008-10-21 03:57:22 MDT --- mls: I converted SOLV files to new format, built satsolver from SVN, did modifications mentioned in comment #22 and improvements are minimal :( (eg. 850ms vs. 750ms) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c28 --- Comment #28 from Michael Schröder <mls@novell.com> 2008-10-21 04:00:38 MDT --- That's surprising. Where's all the time spent? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 Klaus Kämpf <kkaempf@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |NEEDINFO Info Provider| |mls@novell.com Priority|P5 - None |P3 - Medium -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c29 --- Comment #29 from Pavol Rusnak <prusnak@novell.com> 2008-10-21 04:45:08 MDT --- Here is hotshot python profiler output: ncalls tottime percall cumtime percall filename:lineno(function) 6 0.552 0.092 0.552 0.092 satsolver.py:93(add_solv) 4 0.066 0.017 0.066 0.017 satsolver.py:972(step) 1 0.020 0.020 0.020 0.020 satsolver.py:946(__init__) 1 0.003 0.003 0.647 0.647 bin.py:82(main) 7 0.002 0.000 0.002 0.000 ConfigParser.py:434(_read) 12/3 0.001 0.000 0.002 0.001 sre_parse.py:385(_parse) 1 0.001 0.001 0.555 0.555 bin.py:25(__init__) 15/3 0.001 0.000 0.001 0.000 sre_compile.py:38(_compile) 116 0.000 0.000 0.001 0.000 sre_parse.py:207(get) 128 0.000 0.000 0.000 0.000 sre_parse.py:188(__next) 17/5 0.000 0.000 0.000 0.000 sre_parse.py:146(getwidth) 1 0.000 0.000 0.651 0.651 scout-cmd.py:10(runscout) 7 0.000 0.000 0.002 0.000 ConfigParser.py:266(read) 108 0.000 0.000 0.000 0.000 sre_parse.py:136(__getitem__) 1 0.000 0.000 0.089 0.089 bin.py:39(search) 109 0.000 0.000 0.000 0.000 sre_parse.py:144(append) 6 0.000 0.000 0.000 0.000 optparse.py:554(__init__) 64 0.000 0.000 0.000 0.000 ConfigParser.py:354(optionxform) 1 0.000 0.000 0.000 0.000 __init__.py:498(format) 4/3 0.000 0.000 0.002 0.001 sre_parse.py:307(_parse_sub) It seems that most of the time is spent by adding solvables. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User matz@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c30 --- Comment #30 from Michael Matz <matz@novell.com> 2008-10-21 06:58:58 MDT --- Are you by chance loading all the .solv files all the time again and again? That would be very slow of course. You have to keep them in memory of course, the format is designed to be loaded into memory and then worked on. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c31 --- Comment #31 from Michael Schröder <mls@novell.com> 2008-10-21 07:01:00 MDT --- That's probably adding a complete repository, not a single solvable. So it takes half a second to read all 6 repositories? That's pretty slow. Is 'zypper se bash' also that slow? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c32 --- Comment #32 from Pavol Rusnak <prusnak@novell.com> 2008-10-21 07:20:08 MDT --- (In reply to comment #30 from Michael Matz) No, you can see that add_solv is called only 6 times (6 repositories) (In reply to comment #31 from Michael Schröder) $ zypper se | wc -l 20534 $ time zypper se bash > /dev/null real 0m1.771s user 0m1.412s sys 0m0.296s I don't think it makes sense to compare this to adding solvables. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c33 Michael Schröder <mls@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEEDINFO |NEW Info Provider|mls@novell.com | --- Comment #33 from Michael Schröder <mls@novell.com> 2008-10-21 07:27:17 MDT --- Why not? zypper does pretty much the same, except that it also does some gpg-key handling. We can probably make solv file reading a bit faster, but I don't think we can do more than double the speed. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User matz@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c34 --- Comment #34 from Michael Matz <matz@novell.com> 2008-10-21 08:33:59 MDT --- re comment #32: well, then your measurement doesn't reveal much. This is about slowness of completion. If most of the time taken is for reading in the repositories _and_ this happens only once (for the first tried completion) then this time, though large, is not relevant. If OTOH you read in the repos for each completion then it of course is going to be slow. Your profile run doesn't indicate either way as it seems to be from just one completion attempt, that's why I asked that, because depending on that I have to give different advice. So: if you are reading in all repos each time an completion is attempted, this won't ever fly, reading in the .solv files is fast, but not _interactively_ fast. If you are _not_ reading in all repos each time, you need to do some other measurement (i.e. attempt multiple completions so that the first time hit for loading the repos becomes irrelevant). -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c35 --- Comment #35 from Pavol Rusnak <prusnak@novell.com> 2008-10-21 08:48:53 MDT --- SOLV files are read for each completion. Is daemon holding all solvdata in memory (I assume max 20 MBs for typical user) an option? It would also need to detect changes in solv files and reload the data. I could implement it, but I don't know if this approach is reasonable. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c36 --- Comment #36 from Michael Schröder <mls@novell.com> 2008-10-21 09:21:47 MDT --- No, daemons suck. Avoid them if you can. Let's first see how much we can make the solv read faster. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c37 --- Comment #37 from Michael Schröder <mls@novell.com> 2008-10-21 09:36:54 MDT --- Do you have any "monster repositories" in your repo list, i.e. repos with more than 30000 solvables? How big are your solv files? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c38 --- Comment #38 from Pavol Rusnak <prusnak@novell.com> 2008-10-21 09:49:17 MDT --- I have total 20000 solvables according to comment #32. I have these repositories: http://download.opensuse.org/factory/repo/oss/ http://download.opensuse.org/repositories/home:/matejcik/openSUSE_Factory/ http://ftp.gwdg.de/pub/linux/misc/packman/suse/11.1/ http://download.opensuse.org/repositories/devel:/languages:/python:/Factory/... http://download.opensuse.org/repositories/home:/prusnak:/scout/openSUSE_Fact... http://download.opensuse.org/repositories/home:/prusnak:/scout/data/ SOLV files have ~10MB -rw-r--r-- 1 root root 7626858 2008-10-21 15:14 ./solv/factory/solv -rw-r--r-- 1 root root 11121 2008-10-21 15:14 ./solv/home-matejcik/solv -rw-r--r-- 1 root root 2164751 2008-10-21 15:14 ./solv/packman/solv -rw-r--r-- 1 root root 15773 2008-10-21 15:14 ./solv/python/solv -rw-r--r-- 1 root root 8800 2008-10-21 15:14 ./solv/scout-data/solv -rw-r--r-- 1 root root 10578 2008-10-21 15:14 ./solv/scout/solv -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c39 --- Comment #39 from Michael Schröder <mls@novell.com> 2008-10-21 11:09:21 MDT --- Ah, found the reason why it is so slow. I'm to blame for it, the new solv format parser has a bug that makes it move some data much to often. Fixed version is committed to svn... -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c40 --- Comment #40 from Pavol Rusnak <prusnak@novell.com> 2008-10-21 11:32:47 MDT --- It seems a little but faster, but now it also prints to stderr: data_read_idarray: id too large (9/6) unexpected EOF, depth = 2 data_read_idarray: id too large (17/8) unexpected EOF, depth = 2 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User mls@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c41 --- Comment #41 from Michael Schröder <mls@novell.com> 2008-10-22 02:27:27 MDT --- For which solv file? Can you make one of them available to me, please? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User bili@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c42 Li Bin <bili@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |bili@novell.com --- Comment #42 from Li Bin <bili@novell.com> 2008-10-23 03:23:50 MDT --- *** Bug 438004 has been marked as a duplicate of this bug. *** https://bugzilla.novell.com/show_bug.cgi?id=438004 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User sdelcroix@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c43 --- Comment #43 from Stephane Delcroix <sdelcroix@novell.com> 2008-10-23 03:24:33 MDT --- This regressed in beta3, I'm now down to 1.6sec. What about not activating this by default ? -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c44 --- Comment #44 from Dr. Werner Fink <werner@novell.com> 2008-10-23 03:32:37 MDT --- Simply do unset command_not_found_handle and be fine, you may add this to your personal ~/.bashrc file -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User sdelcroix@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c45 --- Comment #45 from Stephane Delcroix <sdelcroix@novell.com> 2008-10-23 04:37:12 MDT --- (In reply to comment #44 from Werner Fink)
Simply do unset command_not_found_handle and be fine, you may add this to your personal ~/.bashrc file
oh, am fine. I was speaking of the distribution default, not my egotist one. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User werner@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c46 --- Comment #46 from Dr. Werner Fink <werner@novell.com> 2008-10-23 05:00:03 MDT --- Disabling this be default would make the package scout useless because this one is especially for novice also called newbe or green horn ;) -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User aj@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c47 Andreas Jaeger <aj@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |aj@novell.com --- Comment #47 from Andreas Jaeger <aj@novell.com> 2008-10-23 05:02:24 MDT --- (In reply to comment #15 from Pavol Rusnak)
Regexp for filelists is
^/(s?bin|usr/(s?bin|games)|opt/(kde3|gnome)/s?bin)/%s$
which matches files in
{ /bin, /sbin, /usr/bin, /usr/sbin, /usr/games, /opt/kde3/bin, /opt/kde3/sbin, /opt/gnome/bin, /opt/gnome/sbin }
With openSUSE 11.1 there should be no /opt/gnome/ anymore and there's neither /opt/kde3/sbin So, let's simplify the regex to look only in: { /bin, /sbin, /usr/bin, /usr/sbin, /usr/games, /opt/kde3/bin} What about the following (untested): ^/(s?bin|usr/(s?bin|games)|opt/kde3/bin)/%s$ -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User prusnak@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c48 Pavol Rusnak <prusnak@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- Status|NEW |RESOLVED Resolution| |FIXED --- Comment #48 from Pavol Rusnak <prusnak@novell.com> 2008-12-15 06:43:41 MST --- Regex match is now performed only on a few strings so it really does not matter. Closing this as fixed. We did almost everything to squeeze down the "lag" and will try to make some other improvements in the future, but the current state is certainly not a bug. -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
https://bugzilla.novell.com/show_bug.cgi?id=435160 User froh@novell.com added comment https://bugzilla.novell.com/show_bug.cgi?id=435160#c49 Susanne Oberhauser <froh@novell.com> changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |froh@novell.com --- Comment #49 from Susanne Oberhauser <froh@novell.com> 2009-02-04 02:54:23 MST --- I've filed a feature request to discuss an alternative, faster approach: https://features.opensuse.org/305803 -- Configure bugmail: https://bugzilla.novell.com/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are on the CC list for the bug.
participants (1)
-
bugzilla_noreply＠novell.com