About Robots.txt Generator
robots.txt tells crawlers which parts of your site they may request. The format looks trivial — a few lines of `User-agent` and `Disallow` — and it has a handful of traps that have taken entire sites out of search results. `Disallow:` with nothing after it means allow everything. `Disallow: /` means block everything. One character separates a normal site from an invisible one.
The most consequential misunderstanding is what the file controls. robots.txt governs *crawling*, not *indexing*. A blocked page can still appear in search results — Google simply shows it without a description, because it was not permitted to look. To keep a page out of the index you need a `noindex` directive, and for Google to see that directive it must be allowed to crawl the page. Blocking a page in robots.txt is therefore the one reliable way to guarantee your noindex is never read.
The validator here reports that contradiction, along with rules written before any `User-agent` line, relative sitemap URLs, and the `Noindex:` directive that robots.txt has never supported and Google formally dropped in 2019.
How to create a robots.txt
Choose which crawlers the rules apply to
Start with * for all of them. Add specific agents only when you genuinely need different rules.
Add your disallow paths
Paths start with / and are relative to your site root. Leave the list empty to allow everything.
Add your sitemap
This must be an absolute URL, unlike the rule paths. It is the cheapest way to help crawlers find every page.
Test before you publish
Use the path tester on a few real URLs. Longest-match resolution surprises people, so checking beats assuming.
The rules that decide conflicts
- The most specific rule wins — not the first, and not the last. Longer matching patterns take priority.
- On an exact tie between Allow and Disallow, Allow wins.
- * matches any sequence of characters; $ anchors the match to the end of the URL.
- Rules are case-sensitive. /Admin and /admin are different paths.
- A blank line ends a group. A stray newline in the middle of your rules attaches the rest to no crawler at all.
What not to put in robots.txt
Never use it to hide sensitive URLs. The file is public, permanently, at a completely predictable address — listing your admin panel there advertises it to anyone curious enough to look. Protect private areas with authentication, not with a request that well-behaved crawlers stay away.
Do not block CSS or JavaScript. Google renders pages to evaluate them, and a page it cannot style or script looks broken to the crawler even though it looks fine to you.