Robots.txt Generator
Set crawler rules and generate a ready-to-upload robots.txt.
About the Robots.txt Generator
The Robots.txt Generator builds a robots.txt file, the small text file that tells search engine crawlers which parts of your site they may visit. You can list disallowed paths, add your sitemap address and set an optional crawl delay, then copy or download the file.
Robots.txt is placed at the root of a site, for example example.com/robots.txt. A line User-agent: * applies to all crawlers, Disallow: /private/ blocks a folder and Sitemap: points to your sitemap. Robots.txt is a request that well-behaved crawlers follow, not a security feature; blocked pages can still appear in search if other sites link to them, and you should never use it to hide sensitive information. Some crawlers, including Google, ignore crawl-delay.
How to use it
- Enter paths you do not want crawled, one per line.
- Add your sitemap URL.
- Optionally enter a crawl delay.
- Copy or download robots.txt and upload it to your site root.
Tips and good to know
- Do not block CSS and JavaScript files that pages need to render.
- Test the file in a search console tool.
- A mistaken "Disallow: /" blocks the entire site.
Frequently asked questions
Where do I put the file?
In the root directory of your domain.
Does it stop pages being indexed?
Not reliably; use a noindex tag.
Do I need one?
It is optional but recommended.