Robots.txt Generator

How to use Robots.txt Generator

1. Choose which crawlers to allow or block.
2. Add any paths to disallow — and check none of them are pages you want indexed.
3. Include your full sitemap URL.
4. Save it at the site root, at /robots.txt, and fetch it in a browser to confirm it serves.
5. It must live at the root of the domain; a subdirectory location is ignored entirely.
6. Each subdomain needs its own file — one on the apex domain does not cover them.
7. Disallow prevents crawling, never indexing; use a noindex tag to remove a page from results.

Disallow is not noindex, and this trips up everyone

robots.txt controls crawling, not indexing. A page you disallow can still appear in search results — typically with no description, because Google was told not to read it but found it linked elsewhere. Worse, if you disallow a page you also want de-indexed, the crawler cannot see your noindex tag, so the page stays indexed indefinitely. To remove a page: let it be crawled and serve a noindex meta tag or header. To save crawl budget on pages already out of the index: disallow.

The rule that takes a site offline

User-agent: * followed by Disallow: / blocks everything. It is the correct content for a staging site and catastrophic on production, and it has repeatedly gone live during a deploy when the staging file was copied across. If organic traffic disappears overnight, this is the first file to check. Note that a blank Disallow value means allow everything, which is the opposite of what it looks like.

What it cannot do

It is a request, not an access control. Well-behaved crawlers honour it; malicious scrapers and email harvesters ignore it entirely. Worse, robots.txt is public, so listing /admin/ or /private-backups/ as disallowed advertises exactly where the interesting directories are. Never put anything in it that you would not want a stranger to read as a map. Protect sensitive paths with authentication.

Crawlers worth naming explicitly

Googlebot and Bingbot for search. GPTBot, ClaudeBot, CCBot, PerplexityBot and Google-Extended for AI training and retrieval — each honours robots.txt and each has its own token, so blocking AI crawlers while allowing search is a deliberate and common choice worth making consciously rather than by default. Also include the full sitemap URL, which is the one line in robots.txt that actively helps discovery rather than restricting it.