I put this robots.txt about one month ago:
-------
User-agent: ia_archiver
Disallow: /
-------
it's intended to disallow the webarchive robot (archive.org). I've seen it on many sites (e.g. Ugas and Mancheck) and it shouldn't affect other search engines. I don't think this has any effect on Google, because the warning page was introduced about 3 months ago, and that's when normal indexing stopped.





Reply With Quote
Bookmarks