robots.txt. document may be the main major component that permits different web spiders in addition to seek robots to be able to cache your site. You possibly can constantly take advantage of this document to allow or maybe retain bots via caching your site.