SEO term
Robots.txt
A small text file at the root of your site that tells search engine bots which areas they may or may not crawl. It is useful for keeping crawlers away from private or low-value sections. Misconfiguring it can accidentally hide your whole site.