What robots.txt does and does not do
robots.txt tells well-behaved crawlers which paths they may fetch. It is a request, not an access control — the file is public, and a crawler that ignores it faces no technical obstacle. Never use it to hide anything sensitive; listing Disallow: /secret-admin publishes the location of your admin panel to anyone who reads the file.
The most common misunderstanding is the difference between crawling and indexing. Blocking a URL here stops Google fetching it, which means Google cannot see a noindex tag on that page either. If a blocked page is linked from elsewhere, it can still be indexed — as a bare URL with no description. To remove a page from search results, allow crawling and use noindex.
Matching rules
- Paths are matched as prefixes:
/adminalso blocks/administrator. - Add a trailing slash to target a directory only:
/admin/. *matches any sequence, and$anchors the end —/*.pdf$blocks PDFs.- When rules conflict, the most specific match wins, and Allow beats Disallow at equal length.
Pair it with a sitemap
The Sitemap: line is how crawlers that have never seen your site find your URL list. Generate one with the XML sitemap generator, and check the meta tags on the pages you are allowing with the meta tag generator.