|
Authored by: Anonymous on Monday, November 05 2012 @ 03:31 PM EST |
~$ curl http://www.apple.com/robots.txt
# robots.txt for http://www.apple.com/
User-agent: *
Disallow:
~$ curl http://www.apple.co.uk/robots.txt
<HTML><HEAD><TITLE>Found</TITLE></HEAD><BODY>
;
<H1>Found</H1> This document has moved to a new <a
href="http://www.apple.com/uk/robots.txt/?cid=oas-emeia-domains-
apple.co.uk"> location</a>.
Please update your documents and hotlists accordingly.
http://www.apple.com/uk/robots.txt/?cid=oas-emeia-domains-apple.co.uk
<!--httphost www.apple.co.uk
URI /robots.txt
qs
env
st11p01ww-redx002.apple.com-->
</BODY></HTML>
~$ curl http://www.apple.com/uk/robots.txt
~$ curl http://www.apple.com/uk/robots.txt/?cid=oas-emeia-domains-
apple.co.uk
both return a complex java/css ridden 404.
I'll take the top para. as authoritative, robots not welcome.
[ Reply to This | Parent | # ]
|
|
|
|
|