Robots.txt Tester
Enter a site, path, and crawler user agent to see whether robots.txt allows or blocks crawling.
Free Robots.txt Tester
Robots.txt rules decide which crawlers can request which paths. This tester parses Allow and Disallow rules with longest-match precedence, then compares common search and AI crawlers so you can catch accidental blocks before they cost traffic.
How robots.txt matching works
Crawlers choose the most specific matching user-agent group, then apply the longest matching Allow or Disallow rule for the requested path. If Allow and Disallow tie, Allow usually wins.
Crawler rules deserve version control
Robots rules change during staging launches, migrations, and AI crawler policy updates. Testing specific paths helps avoid blocking valuable pages while still protecting private or duplicate areas.
Frequently Asked Questions
Does robots.txt remove pages from Google?
No. robots.txt controls crawling. To remove a page from search results, use noindex on a crawlable page or remove the URL and return an appropriate status.
What user agent should I test for Google?
Use Googlebot for general web search crawling. Some Google products use specialized crawlers, but Googlebot is the main SEO check.
Can I block AI crawlers with robots.txt?
Many AI crawlers publish user-agent names and respect robots.txt, but behavior varies by provider. Use explicit groups for the crawlers you care about.
What does an empty Disallow mean?
An empty Disallow line means crawling is allowed for that group because no path is being disallowed.
Настав час позбутися Google Аналітики.
Втомилися від розчарувань, складності та проблем з приватністю Google Аналітики? Ми також. Саме тому ми створили Swetrix - етичну, відкриту і повністю cookie-free альтернативу.