← Blogby Marcell Almeida

How to test robots.txt rules against URL paths

Paste your robots.txt, enter a path and User-agent, and see Allowed or Blocked — before Googlebot finds a surprise.

One wrong Disallow line can block your entire site from Google. The Robots.txt Tester simulates crawler rules against any URL path so you can verify access before deploying a change.

Paste-only limitation: the tool cannot fetch your live robots.txt. Visit yoursite.com/robots.txt in a browser, copy the text, and paste it here. Browsers block cross-origin file fetches from JavaScript.

Try it — paste robots.txt and test a path

Open full tool →

How it works

  1. Paste robots.txt — the full file contents, including User-agent blocks and Allow/Disallow rules.
  2. Enter a path — e.g. /blog/my-post or /admin/dashboard.
  3. Pick a User-agent — Googlebot, Bingbot, or * for the catch-all block.

The tester uses longest-match logic: when multiple rules could apply, the most specific path wins — similar to how Google interprets robots.txt.

What it does and does not do

  • Does — parse User-agent groups, match Allow/Disallow rules, and report Allowed or Blocked for a given path.
  • Does not — fetch your live robots.txt, check sitemap references, or validate against Google Search Console's exact crawler behavior.

For production sites, always double-check in Google Search Console's robots.txt tester after using this for quick local validation.

Common use cases

  • Verifying that Disallow: /staging/ blocks staging URLs but not production paths.
  • Testing whether a new Allow override unblocks a subdirectory inside a broader Disallow.
  • Checking bot-specific rules — e.g. blocking AI crawlers while allowing Googlebot.

Related tools

Build a robots.txt from scratch with the Robots.txt Generator, or validate your sitemap XML with the Sitemap XML Validator.

The Robots.txt Tester is free and runs entirely in your browser. More SEO audit tools at maratool.com/developer/audit.