On Wed, Sep 17, 2014 at 12:57 PM, Claudio Freire <klaussfreire@gmail.com> wrote:
On Tue, Sep 16, 2014 at 11:33 PM, Andrei Borzenkov <arvidjaar@gmail.com> wrote:
Regards,
Stefan
According to /usr/lib/rpm/rpmdb_stat -d /var/lib/rpm/Packages the hash has 19 buckets, and the database contains 3042 keys. Thus every bucket has 150 elements average. I doubt this hash has better access times than a BTree ...
The disk access pattern is horrible, see the attached graphic ...
This seems to confirm what you said above and that hash function seems to be good :) It looks like it cycles through hash buckets on linear scan.
Or it could be that it's scanning the hash index sequentially but accessing something else in tandem with a join, and that's random.
Ok, I reproduced it with a test python script, it's a bsddb3 thing. ~> python bsdcreate.py Berkeley DB 4.8.30: (July 23, 2013) ~> echo 3 | sudo tee /proc/sys/vm/drop_caches 3 ~> time python bsdtest.py real 0m28.912s user 0m0.220s sys 0m0.595s claudiofreire@klaumpp:~> time python bsdtest.py real 0m0.165s user 0m0.100s sys 0m0.064s Attached is the script. If I change HASH into BTREE, I get: ~> rm fruit ~> python bsdcreate.py Berkeley DB 4.8.30: (July 23, 2013) ~> echo 3 | sudo tee /proc/sys/vm/drop_caches 3 ~> time python bsdtest.py real 0m17.458s user 0m0.180s sys 0m0.450s ~> time python bsdtest.py real 0m0.164s user 0m0.092s sys 0m0.072s So, some improvement, but not much. If I do an fadvise first for the whole file (you gotta install python-fadvise[0] for that), I get: ~> echo 3 | sudo tee /proc/sys/vm/drop_caches 3 ~> time python bsdtest.py real 0m3.560s user 0m0.100s sys 0m0.115s ~> time python bsdtest.py real 0m0.164s user 0m0.080s sys 0m0.084s So, fadvise works for full scans. :-) [0] https://github.com/lamby/python-fadvise