All processing happens locally in your browser · No data uploaded
Robots.txt Generator
Build a robots.txt file visually without memorizing syntax. Add rules for different crawlers (Googlebot, Bingbot, all), allow/disallow specific paths, and set your sitemap URL.
Rules
BotTypePath
robots.txt
User-agent: * Disallow: /admin/ Disallow: /private/ Allow: / Sitemap: https://example.com/sitemap.xml
Robots.txt Syntax
User-agent: * (applies to all bots) User-agent: Googlebot (specific bot) Disallow: /admin/ (block this path) Allow: /admin/public (override for sub-path) Sitemap: https://example.com/sitemap.xml Crawl-delay: 10 (seconds between requests)
Examples
Basic website
Result: User-agent: * Disallow: /admin/ Disallow: /wp-admin/ Allow: / Sitemap: https://example.com/sitemap.xml
Allow all crawlers except private admin areas.
Frequently Asked Questions
Does robots.txt guarantee pages are not indexed?
No! Robots.txt only prevents crawling. For guaranteed non-indexing, use <meta name="robots" content="noindex"> or X-Robots-Tag HTTP header.
Can I block specific bots?
Yes. Use User-agent: BotName to target specific crawlers. However, malicious bots often ignore robots.txt.