6 Jun
2008
6 Jun
'08
00:39
On Thu, Jun 5, 2008 at 4:36 PM, Carlos E. R. <robin.listas@telefonica.net> wrote:
A robot can simply ignore what the robot.txt file says. It is not mandatory, it is not forced by the server.
To be pedantic, you can set a flag in Apache to prevent it from generating indexes for directories which robots use to crawl webs. That leaves you only at risk for the links you built into your web. -- ----------JSA--------- -- To unsubscribe, e-mail: opensuse+unsubscribe@opensuse.org For additional commands, e-mail: opensuse+help@opensuse.org