Skip to content
Collabs Tool
SEORuns in your browser

Robots.txt Generator

Create a correct robots.txt with a preview of what each rule allows.

Input
No upload needed — instant
Privacy
Nothing is uploaded
Cost
Free · no sign-up · no watermark

Loading tool…

The tool is loading its code on your device. This happens once and is cached for later visits.

Processed entirely on your device

Everything you type or paste is handled by JavaScript running in this tab. No request is sent, nothing is logged and nothing is stored. Close the page and it is gone.

Overview

About the Robots.txt Generator

Generate a robots.txt file free. Per-user-agent rules, allow and disallow paths, crawl-delay, sitemap declarations and a plain-English explanation of every line.

robots.txt is the first file a crawler reads and the easiest one to get subtly wrong. It sits at the root of your domain, it is plain text, and a single misplaced slash can either expose a directory you meant to hide or block your entire site from search.

The structure

A robots.txt is a sequence of groups. Each group begins with one or more User-agent lines and is followed by the directives that apply to those agents:

User-agent: *
Disallow: /admin/
Disallow: /cart
Allow: /admin/public/

User-agent: Googlebot
Disallow:

Sitemap: https://example.com/sitemap.xml

The blank line separates groups. A crawler uses the most specific matching group only — it does not combine rules from * and from Googlebot. That single fact causes most misconfiguration: people add a * block and a Googlebot block expecting Googlebot to inherit both.

Path matching rules

  • Disallow: /admin blocks /admin and everything starting with that string, including /administrator.
  • Disallow: /admin/ blocks the directory and its contents but not /administrator.
  • Disallow: with an empty value means "nothing is disallowed" — the standard way to fully allow a specific agent.
  • Allow: is not in the original REP specification but is supported by Google, Bing and most major crawlers. When Allow and Disallow both match, Google applies the longest matching rule.
  • * matches any sequence of characters and $ anchors the end of a path. Both are Google extensions, widely but not universally supported.

Crawl-delay

Crawl-delay: 10 asks a crawler to wait ten seconds between requests. Googlebot ignores it entirely — use Search Console's crawl rate setting instead. Bingbot and Yandex respect it. Include it only if your server is genuinely struggling and you are not primarily dependent on Google.

What robots.txt cannot do

It cannot keep a page out of the index. It cannot protect content — anyone can read the file and see exactly what you asked crawlers to avoid, which is why Disallow: /secret/ is a well-known anti-pattern. It cannot stop a determined scraper.

For de-indexing, use noindex in a meta tag or an X-Robots-Tag header, and make sure the page is not disallowed — a crawler that cannot fetch the page cannot see the noindex directive.

For protecting content, use authentication. There is no alternative.

AI crawlers

A growing number of sites now block AI training crawlers explicitly. The common ones are GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended and PerplexityBot. Note that Google-Extended controls AI training only and does not affect regular Google Search indexing — blocking Googlebot to stop AI training would remove you from search entirely.

Deploying and verifying

Upload to the domain root, request /robots.txt in a browser and confirm a 200 response with text/plain. Then check Google Search Console's robots.txt report, which shows syntax errors, warnings and the last fetch time. A 404 on robots.txt is treated as "no restrictions" — which is fine if that is your intention, but it should be a decision rather than an accident.

Step by step

How to use the Robots.txt Generator

  1. Choose which crawlers to address — all, Googlebot only, or a custom user agent.

  2. Add allow and disallow rules for the paths you want to control.

  3. Set a crawl delay if your server is sensitive, and add your sitemap URL.

  4. Read the plain-English explanation the generator produces for each line.

  5. Copy the file or download robots.txt, then upload it to your domain root.

Why use it

Benefits and common use cases

What this tool is good for, and what it deliberately does not try to do.

Explains what it writes

Each rule is annotated in plain language, so you can see that Disallow: /wp-admin/ blocks that path but not its subpaths the way you assumed.

Correct directive syntax

Produces standards-conforming output including the wildcards and end-anchors that Google supports, without the syntax errors that silently invalidate a rule.

Multiple user-agent groups

Address all crawlers, or give Googlebot, Bingbot and AI scrapers different rules — increasingly necessary rather than optional.

Test before you deploy

A path tester shows whether a sample URL would be allowed or blocked under the current rules, catching mistakes before they cost you indexation.

Questions

Frequently asked questions

Short, honest answers about quality, limits and privacy.

Where does robots.txt go?

In the root of your domain, served at https://example.com/robots.txt. It must return HTTP 200 with a text/plain content type. A robots.txt in a subdirectory is ignored — only the one at the host root has any effect.

Does Disallow stop a page from being indexed?

No, and this is the most common misunderstanding. robots.txt controls crawling, not indexing. If other sites link to a disallowed URL, Google can still index it — with no description, because it could not read the page. To keep a page out of the index, use a noindex meta tag or header, which requires the page to be crawlable.

What does User-agent: * mean?

It addresses every crawler that has no more specific group. Rules are matched by the most specific user-agent group available, so a Googlebot-specific group overrides the * group for Googlebot. Note that a group applies in full — crawlers do not merge rules from multiple groups.

Do all crawlers obey robots.txt?

Reputable ones do — Googlebot, Bingbot, DuckDuckBot and the major AI crawlers all respect it. Malicious scrapers ignore it entirely. robots.txt is a polite convention, not a security control, so never use it to protect sensitive content. Use authentication.

Should I include a Sitemap directive?

Yes, it is good practice. Sitemap: https://example.com/sitemap.xml at the end of the file helps all crawlers discover your content, and it is independent of any allow or disallow rules. You can list multiple sitemaps, and they can point to other hosts.