robots.txt Generator
Create a proper robots.txt file with allow/disallow rules, crawl delays, and sitemaps. 100% client-side.
ℹ️ Note: robots.txt is a request, not a security mechanism. Well-behaved bots honor it; malicious ones ignore it. Use authentication for truly private content.
⚡ Templates
User-agent group #1
🗺️ Sitemaps
📖 Common path reference
/— Block/allow everything/admin/— Admin panel/wp-admin/— WordPress admin/login— Login page/private/— Private content/staging/— Staging site/dev/— Development/api/— API endpoints/search— Internal search/cart/— Shopping cart/checkout/— Checkout/account/— User account/includes/— Server includes/cgi-bin/— Legacy CGI scripts/tmp/— Temporary files/backup/— Backup files/*.pdf$— All PDF files/*.jpg$— All JPG images/*?*— URLs with query strings/wp-content/uploads/— Allow WordPress uploadsGenerated robots.txt
Sitemap: https://example.com/sitemap.xml User-agent: * Disallow: /admin/ Disallow: /private/
🔒 100% private: All generation happens in your browser. Your rules are never sent to any server. Follows RFC 9309 (Robots Exclusion Protocol).
About robots.txt Generator
robots.txt is a plain-text file placed at the root of a website (example.com/robots.txt) that tells search engine crawlers which pages they are allowed to index and which they should skip. It uses the Robots Exclusion Protocol (RFC 9309). The file is not a security mechanism — it is a request that well-behaved crawlers honor, but malicious bots may ignore it. For true access control, use authentication. A well-written robots.txt prevents wasted crawl budget on duplicate or irrelevant pages, keeps admin and staging areas out of search results, and points crawlers to your sitemap. This generator helps you build it correctly with proper syntax, common patterns, and presets.
How It Works
- Add user-agent groups (Googlebot, Bingbot, or wildcard *).
- For each group, list Allow and Disallow paths.
- Optionally set a Crawl-delay (seconds between requests).
- Add Sitemap URL(s) at the end.
- Copy the generated robots.txt file to your site root.
✨ Key Features
- ✓Multiple user-agent groups
- ✓Allow and Disallow rules per group
- ✓Crawl-delay directive
- ✓Sitemap URL(s)
- ✓Host directive (Yandex-specific)
- ✓20+ common path presets
- ✓20+ full robots.txt templates (WordPress, e-commerce, AI-block)
- ✓Live preview of generated robots.txt
- ✓Path validation warnings
- ✓Comments for readability
- ✓Copy to clipboard
- ✓100% client-side — no data sent to any server
🎯 Common Use Cases
- →Setting up robots.txt for a new website
- →Blocking admin panels from being indexed
- →Preventing duplicate content from being crawled
- →Allowing only Googlebot while blocking others
- →Blocking AI training crawlers (GPTBot, ClaudeBot, etc.)
- →Pointing crawlers to your sitemap
- →Fixing "Google indexed my staging site" issues
- →Debugging crawl errors in Google Search Console
💡 Advanced Tips & Pro Insights
- ▸robots.txt must be at the site root: https://example.com/robots.txt. It cannot be in a subfolder.
- ▸Only the most specific matching rule applies per user-agent. A longer path match wins over a shorter one.
- ▸User-agent matching is case-insensitive. Use lowercase for readability.
- ▸The wildcard * matches any sequence of characters. Example: Disallow: /*.pdf$ blocks all PDFs.
- ▸The $ anchors the end of a URL. Disallow: /private/$ blocks only /private/, not /private/subpage.
- ▸Crawl-delay is NOT supported by Google. It is honored by Bing (partially), Yandex, and some others. For Google, use Google Search Console → Crawl Rate.
- ▸Sitemap directives are independent of user-agent — place them at the top or bottom, they apply globally.
- ▸robots.txt only blocks crawling, not indexing. To prevent indexing, use <meta name="robots" content="noindex"> on the page itself.
- ▸Blocking a page with robots.txt and using noindex together can be counterproductive — Google cannot read the noindex tag if it cannot crawl the page.
Frequently Asked Questions
Is this robots.txt generator free?▾
Yes, this robots.txt generator is completely free with no signup, email, or usage limits.
Is my data safe?▾
Yes, 100% safe. All generation happens in your browser. Nothing is uploaded to any server.
What is robots.txt?▾
robots.txt is a plain-text file at the root of a website that tells crawlers which URLs they may or may not access. It follows the Robots Exclusion Protocol (RFC 9309).
Is robots.txt a security mechanism?▾
No. robots.txt is an honor-system convention. Malicious bots ignore it. To truly protect private pages, use authentication (password protection) or a staging environment.
What is the difference between Disallow and noindex?▾
Disallow in robots.txt tells crawlers not to VISIT a URL. noindex (a meta tag or HTTP header) tells crawlers not to INDEX a URL if they have already visited it. They serve different purposes — often you want to keep a page crawlable but marked noindex, or vice versa.
Which crawlers should I allow?▾
Allow Googlebot and Bingbot at minimum (they drive most traffic). Consider allowing others based on your audience. Block scrapers and AI training crawlers if you do not want your content used for AI training.
How do I block AI crawlers?▾
Add specific user-agent blocks for GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google AI), CCBot (Common Crawl), and others. Use Disallow: / under each user-agent.
Should I include a Sitemap directive?▾
Yes. Adding "Sitemap: https://example.com/sitemap.xml" helps crawlers find your sitemap faster. Google also allows submitting sitemaps via Google Search Console.