On Sun, Feb 15, 2009 at 3:35 PM, David C. Rankin
Listmates:
On a couple of occasions today, I have had machine gun speed googlebot crawling all over my site. tcpdump shows the
Well as Anders says, (and I'm sure you know) Google spiders web sites. However they honor robots.txt and they spider in such a way as to cause very little load. There is no reason to undertake heroic measures to block them, because they can spider you from any on of several hundred thousand addresses. They do honor robots.txt but if they follow a link into your subdirectory they will not always check the root of the web, so a robots.txt in each directory keeps them at bay. I doubt its spoofed. You really can't spoof an address off your local subnet for anything but denial of service attacks, unless you have control of upstream routers. But its not their practice to swamp a server. -- ----------JSA--------- Someone stole my tag line, so now I have this rental. -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org