Relax. It's just a 'bot trying to index your site: http://info.webcrawler.com/mak/projects/robots/faq.html
My webserver logs certainly show a substantial effort on the part of some visitors to carry out what must be hostile cgi-bin tricks - I disabled the whole of cgi-bin because we don't need it, but some folk are sure trying hard to get system information by this means. I'd rather they didn't get into this server, which is a polite and harmless little machine telling the world some basic facts about our free charity library.
What I'd like to know is
a) what robots.txt really does b) does the entry GET robots.txt in an httpd access log actually mean someone is deliberately accessing the text with a specific request, and if they are would you think it sinister? c) or does Apache access this file under given conditions automatically, creating a GET entry without the user having explicitly requested the file? d) what are the security implications, if any, of what these folk might read there? e) is there a 'more secure' kind of robots.txt I should have? f) does it need to be world-readable g) are there any deleterious effects if I remove it or make it non-readable?
That's a hell of a big list of questions, thanks for reading this far ...
-- To unsubscribe send e-mail to suse-linux-e-unsubscribe@suse.com For additional commands send e-mail to suse-linux-e-help@suse.com Also check the FAQ at http://www.suse.com/Support/Doku/FAQ/