Robots.txt Generator
Choose which search and AI crawlers can access your site.
robots.txt · 3 lines
User-agent: * Allow: /
FAQ
Does robots.txt actually stop bots?+
Well-behaved crawlers respect it. Bad bots can ignore it, so protect private content with a login.
Does blocking a page remove it from Google?+
No. Use a noindex tag to remove pages from search results.
Does Crawl-delay work for Google?+
No. Googlebot ignores it. Bing and Yandex respect it.