Set crawl rules for every bot or for specific crawlers, block private paths, add your sitemap, then copy or download a valid robots.txt for your site root. Free, no sign-up, and nothing leaves your browser.
Applied to every group. Bing and some other bots follow it, but Google ignores it.
A robots.txt file is a plain text file at the root of your website that tells crawlers which URLs they may or may not request. It follows the Robots Exclusion Protocol, and well-behaved bots from Google, Bing and others check it before crawling.
It is mainly a crawl-management tool. Use it to keep bots out of admin areas, internal search results and endless filter URLs, and to point them to your sitemap.
It is not a security feature and it does not hide pages from Google. A blocked URL can still be indexed if other sites link to it, and the file is public, so never list secrets in it.
User-agent: * Disallow: /wp-admin/ Disallow: /cart/ Allow: /wp-admin/admin-ajax.php User-agent: GPTBot Disallow: / Sitemap: https://www.example.com/sitemap.xml
Most sites need only three: User-agent, Disallow and Sitemap. Add Allow when you need an exception.
| Directive | What it does | Priority |
|---|---|---|
User-agent | Names the crawler a group of rules applies to, or * for all | Always include |
Disallow | Blocks a path. An empty value allows everything | Always include |
Sitemap | Gives the full URL of an XML sitemap | Strongly recommended |
Allow | Re-opens a path inside a blocked folder | When you need an exception |
Crawl-delay | Asks a bot to wait between requests. Google ignores it | If it applies |
* and $ | Wildcards for patterns, and end-of-URL matching | If it applies |
They solve different problems. One controls crawling, the other controls indexing.
To stop crawling
To keep a page out of search
The generator writes the file. These four steps get it live and checked.
Start from a preset, or build your own rules.
Add bots, block paths and list your sitemap.
Save it with the exact name robots.txt.
Place it at your site root, then check it in Search Console.
The file must live at the root, for example https://www.example.com/robots.txt. On WordPress, Shopify and Wix the platform or an SEO plugin often manages it, so check your current file before overwriting it. Never disallow CSS or JavaScript files that your pages need to render.
Short answers on rules, placement and AI crawlers.
A robots.txt file tells crawlers which parts of your site they may request. It helps you keep bots out of low-value or private areas and point them to your sitemap.
Upload it to the root of your domain, so it loads at yourdomain.com/robots.txt. Crawlers do not look for it in subfolders, and each subdomain needs its own file.
No. Blocking a URL stops crawling, but the URL can still be indexed if other pages link to it. To keep a page out of search, use a noindex tag and leave the page crawlable.
It blocks the whole site for the bots in that group. It is useful for staging sites and dangerous on a live one, so check it twice before you upload.
Add a group for each crawler with Disallow: /. The Block AI bots preset does this for common ones. Reputable crawlers follow robots.txt, but it is a request, not enforcement.
No. Google ignores the Crawl-delay directive. Bing and some other crawlers respect it. Use it only if a specific bot is overloading your server.
Yes. A Sitemap line with the full URL helps crawlers discover it. You can list more than one sitemap, and the line works regardless of the user-agent groups.
Yes. Google Search Console shows how Google fetched your file and any errors, and you can use its URL Inspection tool to check whether a page is blocked.
Yes. It is completely free, with no sign-up and no limit. Everything runs in your browser, so what you type is never uploaded to a server.
Robots.txt controls crawling. Meta tags and schema describe what gets crawled.
Search Savvy’s technical SEO team audits your crawl rules, sitemaps and indexing, and fixes the issues that waste crawl budget.
