Robots.txt

Scaled On Learning and Education Center

A file that is important for SEO. It lets the search engine bots know which parts of your website they should crawl and which are restricted. This can be useful for hiding duplicate content and admin pages, or to improve the crawl budget.