Free Robots.txt Generator
Create Robots.txt for SEO in seconds
Generate a clean and SEO-friendly robots.txt file. Choose your bot rules,
block private directories, and add your sitemap URL.
- Allow or block all crawlers
- Disallow specific folders (admin, private, search, etc.)
- Add sitemap URL
- Copy + download robots.txt
Place the file at: https://yourdomain.com/robots.txt
Preview: common robots.txt setup.
Choose your robots.txt rules
Allow all is recommended for most websites. Use Block all only for staging/dev sites.
If you have multiple sitemaps, add them one per line (advanced section).
Tip: start with
/. Use trailing slash for folders.
Advanced options
Leave empty to apply rules to
User-agent: *
Allow rules can override a disallow rule for specific files/folders.
Not supported by Google, but some bots respect it.