Test Robots.txt Rules Online

Paste robots.txt and test whether a URL path is allowed or blocked for a given crawler. Does not fetch live files.

Paste only — cannot fetch live URLs.The robots.txt tester analyzes content you paste into the box. Browsers block websites from loading other sites' HTML (CORS), so we cannot enter a URL and pull your page automatically. Copy HTML from View Source (Ctrl+U / Cmd+Option+U), your CMS, or a saved file.
—Result
Paste robots.txt to testMatched rule

How to use

  1. Copy your robots.txt from yoursite.com/robots.txt or your CMS.
  2. Paste it into the robots.txt box.
  3. Enter a URL path and User-agent, then check Allowed or Blocked.

Frequently asked questions

Can this fetch live robots.txt?

No. Paste the file contents — browsers block cross-origin fetches.

How do I get robots.txt?

Open yoursite.com/robots.txt in a browser and copy the text.

Which bots are supported?

Enter any User-agent string (Googlebot, Bingbot, or * for all).

Need to create robots.txt?

Use the Robots.txt Generator.

Last updated

Powered by maratool