Robots.txt Generator — Free Online Tool

What is robots.txt and Why Does It Matter for SEO?

A robots.txt file sits at the root of your domain (e.g. https://yoursite.com/robots.txt) and tells crawlers which pages they are and aren't allowed to visit. It is one of the first files any search engine bot reads when it arrives at your site.

While robots.txt does not directly affect your rankings, it has a powerful indirect impact on SEO through crawl budget management. By blocking low-value URLs — such as admin pages, internal search results, and duplicate content — you ensure Googlebot spends its crawl budget on the pages that actually matter.

A misconfigured robots.txt can accidentally block Googlebot from your entire site, causing pages to drop out of the index overnight. That is why getting it right is critical.

• Crawl budget optimisation — Direct Googlebot only to pages you want indexed, preventing wasteful crawls of pagination, filters, or staging URLs.
• Protecting sensitive areas — Keep /admin, /checkout, and internal tooling out of search engine indices.
• Sitemap declaration — Pointing crawlers to your sitemap.xml speeds up content discovery.

Should You Block AI Scrapers?

Since 2023, AI companies have deployed large-scale bots to scrape the web for training data. The most prominent ones — GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), and Google-Extended (Google's AI training crawler) — respect the robots.txt standard.

Whether to block them is a strategic decision for your site:

• Block if you produce original content and don't want it used for AI model training without compensation or attribution.
• Allow if you want your content to appear in AI-generated answers (e.g. Google AI Overviews) and benefit from the referral traffic.
• Note: Blocking these bots has no negative effect on your standard Google Search rankings — Google-Extended is separate from Googlebot.

Our generator lets you add a single-click block for all four major AI scrapers, with correct user-agent names that these crawlers officially recognise.

Common Robots.txt Mistakes That Kill Your Rankings

Even experienced developers make errors in robots.txt that have devastating SEO consequences. Here are the most common pitfalls:

• Blocking the entire site: Disallow: / under User-agent: * blocks every crawler from every page. This is the single most catastrophic robots.txt error and it's surprisingly common after CMS migrations.
• Blocking CSS and JS files: Google needs to render your pages to understand them. Disallowing stylesheet or script directories prevents proper rendering and can hurt rankings.
• Disallowing canonicalised pages: If you block a URL but it has a canonical pointing to it from an allowed page, Google gets conflicting signals. Always align your canonical tags and robots.txt rules.
• Forgetting to update after migration: A staging robots.txt with Disallow: / copied to production is a silent ranking killer.
• No Sitemap declaration: While not mandatory, including a Sitemap: line is a best-practice signal that helps crawlers discover all your content faster.