Well, it's the memory bandwith, stupid... What we observe appears to be essentially the performance drop on the way from 10^5 to 10^6 unknwons which one can also see in Georg Hagers blog: https://blogs.fau.de/hager/archives/6883 So this can be closed.