Mailinglist Archive: opensuse (3104 mails)

[fwilde@xxxxxxxxxxxxxxx (Fergus Wilde)]

Another dumb newbie query, no doubt, and not SuSE specific, but would anyone
be prepared to offer a view on the security aspects of people reading the
file robots.txt from the httpd documents directories on a SuSE based
webserver?

My webserver logs certainly show a substantial effort on the part of some
visitors to carry out what must be hostile cgi-bin tricks - I disabled the
whole of cgi-bin because we don't need it, but some folk are sure trying
hard to get system information by this means.  I'd rather they didn't get
into this server, which is a polite and harmless little machine telling the
world some basic facts about our free charity library.

What I'd like to know is

a) what robots.txt really does
b) does the entry GET robots.txt in an httpd access log actually mean
someone is deliberately accessing the text with a specific request, and if
they are would you think it sinister?
c) or does Apache access this file under given conditions automatically,
creating a GET entry without the user having explicitly requested the file?
d) what are the security implications, if any, of what these folk might read
there?
e) is there a 'more secure' kind of robots.txt I should have?
f) does it need to be world-readable
g) are there any deleterious effects if I remove it or make it non-readable?

That's a hell of a big list of questions, thanks for reading this far ...

Fergus Wilde
Chetham's Library
Long Millgate
Manchester
M3  1SB
UK

Tel: +44 (0)161 834 7961
Fax: +44 (0)161 839 5797


-- 
To unsubscribe send e-mail to suse-linux-e-unsubscribe@xxxxxxxx
For additional commands send e-mail to suse-linux-e-help@xxxxxxxx             
Also check the FAQ at http://www.suse.com/Support/Doku/FAQ/