Robots.txt
A text file telling crawlers which parts of a site they may access.
What is Robots.txt?
Robots.txt is a plain-text file placed at the root of a website that gives search engine crawlers instructions on which pages or directories they may or may not access. It uses simple directives like User-agent, Disallow, and Allow, and often points to the XML sitemap. It controls crawling but does not reliably prevent indexing, so sensitive pages need other protections.