robots.txt Generator
Generate robots.txt files online
Frequently Asked Questions
What is a robots.txt file?
It is a plain text file placed in your website's root directory to instruct search engine crawlers which pages to index or skip.
How do I generate a robots.txt file?
Select the crawl rules, specify allowed/disallowed paths, add your sitemap URL, and the tool builds the file syntax.
Where do I upload the robots.txt file?
Upload the file to the absolute root of your domain (e.g., `https://example.com/robots.txt`).
What is the difference between Allow and Disallow?
`Disallow` blocks crawlers from accessing specific paths. `Allow` overrides a disallow rule for nested subdirectories.
What does the user-agent wildcard "*" mean?
The asterisk wildcard `*` applies the specified crawling rules to all search engine bots and crawlers.
Can I block specific crawlers (like Baidu or Yandex)?
Yes. You can define specific rules for individual crawler names (user-agents) in the file.
Should I put my sitemap location in robots.txt?
Yes. Adding `Sitemap: https://example.com/sitemap.xml` helps crawlers find your sitemap quickly.
Does robots.txt guarantee page privacy?
No. robots.txt is a public directive. To secure private pages, use password authentication or noindex meta tags.