It specifies for search engines’ crawlers and spiders what website pages and files you want or don’t want them to visit.