Robots.txt Generator — Visual, Free, Sitemap-Aware

Disallow does not mean deindex

The most damaging misunderstanding about robots.txt is that Disallow: /path removes a URL from search results. It does not. Disallow prevents crawlers from fetching the page — indexing is a separate decision.

If other pages link to a blocked URL, a search engine can still index it from those links alone. And because it is forbidden from fetching the page, it cannot read the content to write a description, so the result appears as a bare URL with no useful summary. Blocking a page you wanted removed makes the listing worse rather than absent.

Use the right tool for the intent you actually have:

• To keep a page out of the index — allow crawling and serve a noindex meta robots tag or an X-Robots-Tag: noindex header. Note the trap: the crawler must fetch the page to see that instruction, so blocking it in robots.txt actively prevents the removal you want. If a page is both disallowed and carries noindex, the noindex is never read.
• To remove something urgently — the removal tool in Google Search Console.
• To stop crawl budget draining into infinite faceted or filter URLs — this is what robots.txt is genuinely for.

robots.txt is advisory, not a lock

The file sits at your domain root, publicly readable by anyone who types the path. It is not access control. Well-behaved crawlers honour it; scrapers built to ignore it ignore it.

Which means Disallow: /admin does not hide /admin — it advertises to anyone curious that the path exists and that you would rather they did not look. Never use robots.txt to protect admin panels, staging environments, backups or private files. That needs authentication and server-level restrictions; a text file asking politely is not a security control.

The syntax

Rules are grouped by User-agent. User-agent: * applies to every crawler, and you can target a specific one such as User-agent: Googlebot — a crawler follows only the most specific group that matches it, not several at once.

Within a group, Allow and Disallow paths are prefix matches, which surprises people. Disallow: /admin blocks /admin, /admin/users and also /admin-panel and /administrator, because all of them begin with that string. Add the trailing slash — Disallow: /admin/ — if you meant only the directory.

An empty Disallow: with no path means allow everything. Allow carves exceptions out of a broader block: Disallow: /temp/ followed by Allow: /temp/public/ blocks /temp/archive while permitting /temp/public/, because the longer matching rule wins.

Crawl-delay

The optional Crawl-delay: 5 asks crawlers to wait five seconds between requests. Google's crawlers ignore it entirely — Google determines its own crawl rate and adjusts automatically when a server responds slowly or returns errors. Several other crawlers, including Bing's, do honour it, so it is not useless, just not a lever over Google. If Googlebot is genuinely overloading your server, the effective responses are server-side rate limiting or returning 503 for a period, not this directive.

Pointing to your sitemap

A Sitemap: directive can be listed in robots.txt with an absolute URL, and unlike the other lines it is not tied to any User-agent group — put it anywhere in the file and every crawler reads it. It is the simplest way to make a sitemap discoverable without submitting it anywhere. Generate one with the sitemap.xml generator.