On Thursday 19 April 2007 14:01, Duncan Mac-Vicar Prett wrote:

On Wednesday 18 April 2007 18:01:44 Duncan Mac-Vicar Prett wrote:

...

I am not happy with performance yet. To create basic resolvables objects I need 19 (s) for 19.500 packages. I got this number down from 40. It is much faster than zypp 2.x, but not the numbers I am looking for.

Got 19.000 packages in less than 1 second, denormalizing 1 capablities table.

Wow. That's what I call speedup. Just watch out not to get into any kind of speed intoxication. You might overcome the sound barrier, but getting across the barrier of lightspeed is a lot harder still. ;-) CU -- Stefan Hundhammer Penguin by conviction. YaST2 Development SUSE LINUX Products GmbH, GF: Markus Rex, HRB 16746 (AG Nürnberg) Nürnberg, Germany -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

On Thursday 19 April 2007 16:08:44 Benji Weber wrote:

Presumably this is querying the cache? What sort of times are we looking at for converting metadata -> database at the moment? I would expect query times to be fast but cache creation time I'm interested in.

Not just that, it is loading all the required objects in memory. That is NVRA and capabilities. Queries are not fast by nature, the amount of time you need to spend playing with schema, indexes, etc is not trivial. Launching a source went down from 35 to 19 and to 1 second. Half of the time before was parsing capabilities, we now store them ready to be read. A horrible test program without any optimization takes like 35 seconds right now, to write a 19.000 packages cache. I have some ideas how to start reducing it later. What we dont have, I am not sure we will have time, is a rpm -> cache store loader, that would turn the 20 seconds reading rpm database into half. -- Duncan Mac-Vicar Prett Novell :: SUSE R&D, Maxfeldstr. 5, 90409 Nürnberg GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe, e-mail: zypp-devel+unsubscribe@opensuse.org For additional commands, e-mail: zypp-devel+help@opensuse.org

If you look at this graph, the problem is not even the database it self, but other ZYpp bottlenecks. For example, filterExtraSupplemments is 5% of the total. rewriteKernelDeps alone, 1% Edition::Impl validateVR is another bottleneck If you just loop the capabilities, it is: 1.95user 0.04system 0:01.99elapsed Still denormalizing the capabilities made writing faster. Launching 19.000 packages and capabilities real 0m20.069s user 0m15.417s sys 0m0.252s Disabling rewriteKernelDeps and other "hacks" real 0m15.803s user 0m15.057s sys 0m0.108s disabling Edition::Impl::validateVR (return the same string) real 0m11.071s user 0m10.793s sys 0m0.088s And so on.... My proposal, we need backdoors here and skip to much regexp validation, we are supposed to save only validated data in the db. The CacheStore can take care of doing it once. Duncan