Robots.txt Tester
Check if a URL is allowed or blocked for any crawler.
Load a robots.txt and enter a URL to test.
FAQ
Which rules apply to a bot?+
Its own User-agent group if one exists, otherwise *. The longest matching rule wins; ties go to Allow.
What if no rule matches?+
The URL is allowed.
Are wildcards supported?+
Yes. * matches anything and $ marks the end of the URL.