Tools menu
Free tool
Robots.txt Generator
Create robots.txt files with user-agent rules, allow/disallow directives, and sitemap references.
Robots Config
Crawler Governance
Your generated code will appear here
Crawler Safety Warning
Misconfiguring your robots.txt can accidentally de-index your entire website. Always test your generated code in the Google Search Console Robots Tester before deployment.
- Do not disallow CSS or JS files (essential for rendering).
- Ensure the Sitemap URL is absolute (includes https://).
- Use * for general rules, and specific bots for custom logic.
How to use this tool
Follow these steps to get the most out of the Robots.txt Generator.
Select a User-Agent (e.g., Googlebot, Bingbot, or *).
Add 'Allow' or 'Disallow' directives for specific paths.
Include the absolute URL to your Sitemap index.
Set a Crawl-Delay if your server needs throttling.
Copy the formatted text to your robots.txt file.
Want it done for you?
Crawling and indexing problems quietly cap your rankings. Our technical SEO audits find and fix them.
SEO ServicesMore in Sitemaps & Robots