SEO
Robots.txt Generator
Prepare clear robots.txt rules for common site sections, crawlers, and sitemap references.
Generated robots.txt
User-agent: * Allow: / Sitemap: https://workflowtools.space/sitemap.xml
How to use this tool
- Enter the full website URL and sitemap URL.
- Choose allow all for a normal public site, block all for private staging, or custom for specific paths.
- Add one disallow path per line when using custom mode.
- Copy the generated text and place it in the robots.txt file at your domain root.
Practical examples
A public blog might allow all crawling and include the sitemap URL.
A staging site can block all crawlers while the team prepares a launch.
A site can disallow internal search pages or admin-style paths that should not appear in search.
Common mistakes to avoid
- Using robots.txt to hide private information instead of securing the page.
- Blocking important CSS, JavaScript, or page paths that search engines need to render content.
- Forgetting to update the sitemap URL after changing domains.
- Leaving block-all rules in place after a staging site becomes public.
Useful internal links
Frequently Asked Questions
Can robots.txt force every crawler to obey the rules?
No. Reputable crawlers usually respect robots.txt, but it is not an access control system and should not protect private data.
Where should robots.txt be placed?
Place it at the root of the domain, such as https://example.com/robots.txt.
Should I block all crawlers on a live website?
Only do that when you intentionally do not want pages crawled. Blocking all crawlers on a public site can prevent search discovery.