Robots.txt Generator

Generate a valid robots.txt file for your website. Control which bots can crawl your pages, block AI scrapers, and add your sitemap URL.

Presets:
robots.txt

How to use

  1. Choose a preset (Allow All, Block All, Block AI bots) or build rules manually with the rule builder.
  2. Optionally add your sitemap URL and enable a crawl delay.
  3. Copy the generated robots.txt and upload it to the root of your website.

Frequently asked questions

What is a robots.txt file?

A plain-text file at your site root that tells crawlers which pages to skip. It follows the Robots Exclusion Protocol — well-behaved bots respect it, malicious ones may not.

Does robots.txt prevent pages from appearing in Google?

It prevents crawling, not indexing. Google may still index a blocked URL if other pages link to it. Use noindex to prevent indexing.

How do I block AI training crawlers?

Use the "Block AI bots" preset to add Disallow rules for GPTBot, CCBot, anthropic-ai, Claude-Web, and Google-Extended.

What should I put as the Sitemap URL?

The full absolute URL, e.g. https://example.com/sitemap.xml. It helps search engines discover your pages faster.

Last updated

Powered by maratool