robots.txt Generator

Create robots.txt files to control search engine crawlers.

Helpful?

Thanks for your feedback!

You have already rated this tool.

Quick Templates

User Agents

All bots (*)

Googlebot

Bingbot

Googlebot-Image

GPTBot (OpenAI)

Claude-Web (Anthropic)

CCBot (Common Crawl)

Disallow Paths One path per line

Allow Paths (overrides disallow) One path per line

Sitemap URL

Crawl Delay (seconds) Leave empty for no delay

Preferred Host Specify canonical domain (optional)

Location: Place robots.txt in your website's root directory (e.g., example.com/robots.txt)

Note: robots.txt is advisory only. Malicious bots may ignore it.