Free SEO & AI crawl tool
Free Custom Robots Txt Generator
Create a production-ready robots.txt for WordPress, Blogger, Wix, Shopify, and any CMS. Control search crawlers and modern AI bots (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more), add sitemaps, and download or copy in one click.
Sitemap(s)
Allow Directories
Disallow Directories
Allow User-agents
Select All to include every crawler below (they will be checked and locked). Or pick specific search and AI bots.
Search engines & SEO bots
AI crawlers
Disallow User-agents
Block training crawlers (e.g. GPTBot, ClaudeBot, Google-Extended) while still allowing search bots if you want AI citations without model training use.
Search engines & SEO bots
AI crawlers