В Tue, 16 Sep 2014 23:49:53 +0200 Stefan Brüns <stefan.bruens@rwth-aachen.de> пишет:
On Tuesday 16 September 2014 23:01:40 Stefan Brüns wrote:
On Tuesday 16 September 2014 16:54:13 Claudio Freire wrote:
That's not the case. It does use cached reads, but it takes about a minute to cache the whole thing in random order, whereas it takes only a few seconds in sequential order.
I'm having a hard time following rpmdb.c's code. I see it uses plain db3 cursors, which should be sequentially scanning the file instead of hopping all over the place. If it's truly the case, then it's db3 the one that needs fixing. If it's rpmdb.c creating other cursors in parallel and seeking other parts of the packages database, which I can't rule out because I couldn't fully figure out the code yet, but seems unlikely, it's rpmdb.c the one in need of fixing.
I think a simple test case should clear this. I'll try to make one with python (it has a nice and neat interface to db3 that's easier to use than C for this).
The Packages db is in DB_HASH format - this has several implications:
1) A linear scan of the database is a random access pattern of the backing store, i.e. the disk.
As we do not care about real order in which entries are returned - is it possible to scan in order in which they are stored? I.e. iterate over one hash bucket then next etc?
2) bdb *does* a mmap of database files, but not for DB_HASH databases.
I do not see how mmap is relevant or can help here. If anything it can make things worse by pretending you can do fast access in random size and order.
Regards,
Stefan
According to /usr/lib/rpm/rpmdb_stat -d /var/lib/rpm/Packages the hash has 19 buckets, and the database contains 3042 keys. Thus every bucket has 150 elements average. I doubt this hash has better access times than a BTree ...
The disk access pattern is horrible, see the attached graphic ...
This seems to confirm what you said above and that hash function seems to be good :) It looks like it cycles through hash buckets on linear scan.
Regards,
Stefan
-- To unsubscribe, e-mail: opensuse-factory+unsubscribe@opensuse.org To contact the owner, e-mail: opensuse-factory+owner@opensuse.org