Hey, On 04.10.21 10:15, Andreas Schwab wrote:
On Okt 04 2021, Adrian Schröter wrote:
Dunno what was the situation on saturday, atm it looks fine for me.
It lasted until late Sunday evening.
We had multiple spiders (bing, yandex, unicom, sprinthost.ru) crawling 'expensive' routes like /package/view_file/... Every once in a while their frequencies align and then it becomes a bit too much. But in general we want them to index things of course. But it wasn't *that* bad, 95% of all requests still where answered faster than 256 milliseconds. We are looking into a more clever robots.txt now and some request throttling in the mid-term future.... Henne -- Henne Vogelsang http://www.opensuse.org Everybody has a plan, until they get hit. - Mike Tyson