You do realize by 'gay' I mean a man who has sex with other men?
Robots.txt - The Basics
By writing a structured text file you can indicate to robots that certain parts of your server are off-limits to some or all robots. It is best explained with an example:
# robots.txt file for general use on web servers.
The first line, starting with '#', specifies a comment.
The first paragraph specifies that the robot called 'webcrawler' has nothing disallowed: it may go anywhere.
The second paragraph indicates that the robot called 'googlebot' has all relative URLs starting with '/' disallowed. Because all relative URL's on a server start with '/', this means the entire site is closed off.
The third paragraph indicates that all other robots should not visit URLs starting with /cgi-bin or /log. Note the '*' is a special token, meaning "any other User-agent"; you cannot use wildcard patterns or regular expressions in either User-agent or Disallow lines.
Two common errors:
Wildcards are not supported: instead of 'Disallow: /tmp/*' just say 'Disallow: /tmp'.
You shouldn't put more than one path on a Disallow line (this may change in a future version of the spec)
Ultimately, without the use of robots.txt files on your servers/domains, you are risking a variety of potential problems including, unauthorized access to your cgi directory, unauthorized viewing of your site stats, possible spamming of the search engines by accidental crawling of doorway pages.
One distinct advantage however of having a robots.txt file on your server is that, quite simply, you will be able to tell when and where your site has been indexed or potentially indexed as, all robots will automatically call for the robots.txt file BEFORE any other page on your server so, as long as you keep an eye open for any calls of this file, you can see who is knocking at your site for indexing purposes.
Below is a robots.txt example that you can copy and paste into a text document to use on your own server:
<!--Start Copy Below This Line-->
<!--End Copy Above This Line-->
The above will allow all spiders to crawl all of your site except the subdirectory's 'cgi-bin' and 'logs' which, may be altered to suit any subdirectory's you do not wish the spiders to crawl on your server.
Article written by Lee.
Robot.txt file is the main constituent that enables various web crawlers and search bots to cache your website. You can always make use of this file to allow or keep robots from caching your website.
affordable web design india
robots.txt. document may be the main major component that permits different web spiders in addition to seek robots to be able to cache your site. You possibly can constantly take advantage of this document to allow or maybe retain bots via caching your site.
Robots.txt is a file in the root directory of your web site that instructs web crawlers what parts, or all, or none of your site they are allowed examine.
Good to know about the Robots.txt in detail description.