Robot.txt file is the main constituent that enables various web crawlers and search bots to cache your website. You can always make use of this file to allow or keep robots from caching your website.