Skip to main content
ToolsBay

Robots.txt Generator

Write crawler allow/disallow rules without syntax mistakes.

Runs entirely in your browser — nothing is uploaded

disallow paths
allow paths
Common paths to block

robots.txt

User-agent: *
Disallow:

Frequently asked questions

Does robots.txt keep a page out of Google?

No — and this catches people out. It stops crawling, not indexing. A blocked page can still appear in results if other sites link to it, just without a description. To keep a page out of the index, let it be crawled and serve a noindex meta tag.

Why does an empty "Disallow:" mean allow everything?

Because Disallow with no value blocks nothing. It is the standard way to write "this user agent may crawl the whole site", and it keeps the group valid rather than empty.

Does Google respect Crawl-delay?

No. Googlebot ignores it entirely — adjust the crawl rate in Search Console instead. Bing, Yandex and several smaller crawlers do respect it.

Where does the file go?

Only at the root of the domain: https://example.com/robots.txt. A robots.txt in a subdirectory is ignored, and each subdomain needs its own.

What robots.txt does and does not do

robots.txt tells well-behaved crawlers which paths they may fetch. It is a request, not an access control — the file is public, and a crawler that ignores it faces no technical obstacle. Never use it to hide anything sensitive; listing Disallow: /secret-admin publishes the location of your admin panel to anyone who reads the file.

The most common misunderstanding is the difference between crawling and indexing. Blocking a URL here stops Google fetching it, which means Google cannot see a noindex tag on that page either. If a blocked page is linked from elsewhere, it can still be indexed — as a bare URL with no description. To remove a page from search results, allow crawling and use noindex.

Matching rules

  • Paths are matched as prefixes: /admin also blocks /administrator.
  • Add a trailing slash to target a directory only: /admin/.
  • * matches any sequence, and $ anchors the end — /*.pdf$ blocks PDFs.
  • When rules conflict, the most specific match wins, and Allow beats Disallow at equal length.

Pair it with a sitemap

The Sitemap: line is how crawlers that have never seen your site find your URL list. Generate one with the XML sitemap generator, and check the meta tags on the pages you are allowing with the meta tag generator.

Related tools

All seo tools