Skip to content
QuikTools Hub

Robots.txt Guide: Rules, Examples and AI Crawlers

Published: 15 November 2026 · by Umar Farooq Iftikhar

A robots.txt file is a small plain-text file that tells crawlers which parts of a website they may request. It sits at the root of a domain, at /robots.txt, and well-behaved crawlers read it before they crawl. It is a request, not a lock: it guides cooperative bots, and it cannot keep a page out of search results on its own. The free Robots.txt Generator on this site builds a valid file from your rules — one user-agent group, allow and disallow paths, an optional crawl-delay and sitemap line — entirely in your browser.

This guide explains how crawlers actually read the file: how groups are chosen, which rule wins when two rules match, what wildcards do, and what the file cannot do. It covers the AI training crawlers that now have their own controls, the file's size and placement limits, and worked examples you can adapt — including which examples the generator produces directly and which need two blocks merged by hand. The rules described here follow RFC 9309, the Robots Exclusion Protocol standard published by the IETF in September 2022, and Google's robots.txt documentation.

How a crawler reads the file

The convention started in 1994, when the original robots.txt proposal asked crawlers and sites to agree on one simple file. A robots.txt file is organized into groups. Each group starts with one or more User-agent lines naming the crawlers the group applies to, followed by the rules for those crawlers: Allow and Disallow lines, one rule per line. Consecutive User-agent lines before the rules all belong to the same group; a new group begins only when a User-agent line appears after the preceding group's rules, and the last group runs to the end of the file. Comments start with # and are ignored. The file must be named exactly robots.txt, encoded as UTF-8, and served from the root of the host it applies to.

A crawler looks for the group that names its own product token, matched case-insensitively. If a group names it, the crawler follows that group and ignores the wildcard group entirely. Only when no group names it does it fall back to the User-agent: * group. If several groups name the same crawler, Google combines them into one group before applying the rules. And the default is open: anything not blocked by a Disallow rule is allowed, so a file with no Disallow lines places no restrictions at all.

Which rule wins: the longest match

When several rules match a URL, the crawler uses the most specific one — the rule whose pattern matches the greatest number of characters in the path, octets in the standard's wording, which is the same count for ordinary ASCII paths. Line order does not matter: an Allow line above or below a Disallow line changes nothing, unlike a firewall where the first match wins. Take a file that blocks /blog/ (6 characters) but allows /blog/case-studies/ (19 characters): for the URL /blog/case-studies/client-a, the Allow rule matches more characters, so the page is crawlable, while the rest of /blog/ stays blocked.

The same arithmetic decides the other cases. Disallow: /images/ (8 characters) with Allow: /images/logo.png (16 characters) keeps the logo crawlable inside a blocked folder. And a longer Disallow beats a shorter Allow: Allow: /downloads/public/ (18 characters) with Disallow: /downloads/public/internal/ (27 characters) blocks the internal folder while the rest of the public folder stays open. When an Allow and a Disallow match with exactly equal length, the Allow rule wins: Disallow: /page and Allow: /page together mean /page is allowed. Count the characters, not the lines, when two rules disagree.

Wildcards and the end anchor

Two special characters extend prefix matching. An asterisk matches any sequence of characters, including none, and a dollar sign anchors the match to the end of the URL. Disallow: /*.pdf$ blocks every URL ending in .pdf, whatever folder it sits in. Disallow: /*? blocks every URL containing a question mark — the usual way to keep crawlers out of query-string variations such as faceted search or tracking parameters. Disallow: /foo$ blocks exactly /foo and nothing longer: /foo/bar does not match, because the dollar sign requires the URL to end there.

Both characters are defined in RFC 9309 and supported by Google's crawlers, so they work for the major search engines. Because the standard dates from 2022, a crawler built to older behaviour may treat them literally; test wildcard rules against the crawlers that matter to you. The generator passes paths through exactly as typed, so a wildcard rule typed into the Disallow box is emitted verbatim in the file; nothing about the generator's behaviour changes for wildcard patterns.

The trailing slash, prefix matching, and the empty-value trap

Every path rule is a prefix match: read it as 'everything starting with this'. Disallow: /private/ blocks the folder and everything beneath it. Without the trailing slash, Disallow: /private also matches /privacy-policy and /private-files.html — paths that merely start with the same letters. Google's documentation states the rule directly: a rule for a directory must end with the slash mark. The same prefix logic explains why Disallow: /tmp/ is the safe form and Disallow: /tmp can surprise you.

The empty value is the opposite trap. Disallow: with nothing after the colon places no restriction at all — it means allow everything — while Disallow: / blocks the entire site. The two differ by a single character, and confusing them is one of the most consequential typos on the web: a staging file copied to a live site, or a blank rule meant as a placeholder, can remove a whole domain from crawling with no error message anywhere. If a rule looks empty, read it as permission, not as a block.

What robots.txt cannot do

The standard states it plainly: these rules are not a form of access authorization, and the protocol is not a substitute for valid content security measures. The file itself is public — anyone can fetch /robots.txt — so listing a sensitive path there advertises it rather than hiding it. Reputable crawlers obey the file; scrapers, spam bots and attackers routinely ignore it, and some read it specifically to find the paths a site hoped to hide. Private areas need passwords and proper access controls, never robots.txt alone.

Robots.txt also controls crawling, not indexing. A disallowed page cannot be crawled, but if other sites link to it, search engines may still list its URL in results, shown without a description because the content could not be read. To keep a page out of results, allow the crawl and use a noindex tag on the page itself, or protect it with a login. There is no supported noindex directive for robots.txt: Google's documentation lists only user-agent, disallow, allow and sitemap as supported rules, and lines that match none of them are ignored — a Noindex: line in robots.txt does nothing for Google.

Groups: one file, several crawlers

A file can hold several groups, each starting with its own User-agent lines. A crawler follows only the group that names it, so rules meant for every crawler must be repeated in each group or left to the wildcard group for crawlers with no named group. Multiple User-agent lines at the top of one group share that group's rules: Google's own example blocks both Googlebot and AdsBot-Google with two User-agent lines above a single Disallow: / line. Note the AdsBot exception in Google's documentation: the * wildcard matches all crawlers except the AdsBot crawlers, which must be named explicitly to be covered.

The generator builds one group at a time, which covers the most common case: identical rules for everyone. For genuinely different rules per crawler, generate one file per crawler and merge the blocks together in a text editor, keeping each group's User-agent lines above its rules. One warning: the generator's user-agent field takes a single name. Typing two names on one line produces a single User-agent line containing both names, which matches no crawler — generate once per name, or add the extra User-agent lines by hand above the shared rules.

AI crawlers: training, search, and user fetches

AI companies now run three kinds of crawler, and robots.txt treats them as three separate decisions. Training crawlers collect content that may be used to train future models. Search crawlers fetch pages so AI answers can cite them. User-triggered fetchers retrieve a page because a person asked about it or pasted a link. Blocking the first kind is a policy choice about training; blocking the second removes the site from AI answers. The third kind differs by operator: Anthropic documents all three of its bots, including the user-triggered Claude-User, as controllable through robots.txt — disabling Claude-User prevents retrieval in response to a user query — while OpenAI says ChatGPT-User requests, because they are initiated by a user, may not be governed by robots.txt. Do not generalize one operator's behaviour to another's.

The vendor-specific training controls are GPTBot from OpenAI, ClaudeBot from Anthropic, and Google-Extended from Google — three separate controls, not interchangeable ones. OpenAI states that each of its crawler settings is independent of the others, so blocking GPTBot does not remove a site from ChatGPT search: that is controlled separately by OAI-SearchBot, and sites opted out of OAI-SearchBot are not shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic separates ClaudeBot (training) from Claude-SearchBot (search) and Claude-User (user fetches), and supports the Crawl-delay extension. Google-Extended is a robots.txt product token, not a separate HTTP crawler identity: it has no separate user-agent string, and crawling is done with existing Google user-agent strings. It controls whether content Google crawls may be used to train future generations of Gemini models and to ground answers in Gemini Apps and Vertex AI. Blocking it does not affect inclusion in Google Search or ranking — but it can affect whether the site's content grounds Gemini answers, so it does not preserve every form of Gemini visibility.

Alongside the vendor-specific controls, a shared cross-vendor signal now exists in draft form. The IETF Internet-Draft "Machine-Readable Training Preference Signal", published 30 October 2026, defines the policy token __usage_ai-training: a robots.txt group keyed by that token in the User-agent line expresses a preference about AI training for the paths it covers, using the ordinary longest-match rules. It is an evolving Internet-Draft, not a finalized RFC, and it is not universally supported. Anthropic began recognizing the signal on 30 October 2026, treating rules that name ClaudeBot directly as more specific than the general signal. Google announced on the same day that it plans to support the shared control in 2027, interpreting an opt-out through the token as if the publisher had disallowed Google-Extended, unless Google-Extended is explicitly specified. Critically, the token expresses a usage preference, not a crawling rule: the draft states it is not a directive for crawling behaviour, and it does not block any crawler from fetching a page. To limit crawling itself, use groups naming the crawlers' own product tokens.

There is no single switch that blocks all AI training while preserving every form of AI visibility, so treat each control on its own terms. Blocking the vendor-specific training controls — GPTBot, ClaudeBot and Google-Extended — signals no training use to those operators while leaving their search crawlers alone, but Google-Extended also governs Gemini grounding, so blocking it can cost grounding visibility even though Google Search inclusion and ranking are unaffected. The shared __usage_ai-training token offers one broader preference, but it is a draft signal with partial support: Anthropic recognizes it, Google plans to support it in 2027, and it blocks no crawler. The practical approach is to decide per operator and per purpose — training, search, grounding, user fetches — using the vendor-specific controls where they exist, and to add the shared token as an additional preference rather than a replacement. Generate one block per named agent with the tool and merge them, or add the lines by hand. Crawler names, tokens and purposes change: check each operator's current crawler documentation before copying any list of tokens, and re-check it when this guide's examples stop matching what the operators publish.

Where the file lives, and how big it can be

The file must sit at the root of the host it applies to, named exactly robots.txt in lowercase, reachable at yoursite.com/robots.txt. It cannot live in a subdirectory. Its rules apply only to the protocol, host and port where it is posted: the file on https://example.com does not cover http://example.com, m.example.com, or example.com:8181 — each needs its own file. Each subdomain needs its own file for the same reason.

Size has a floor, not just a ceiling: RFC 9309 requires crawlers to parse at least the first 500 kibibytes of the file, and beyond that a crawler may stop reading — so nothing important belongs at the bottom of a very long file. In practice most files are a few kilobytes; a file approaching the limit usually means generated rules that should be replaced with a few wildcard patterns. Crawlers may also cache the file: the standard says a cached copy should not normally be relied on for more than 24 hours, unless the file is unreachable. That is caching guidance, not a guaranteed propagation deadline: it describes how long a cached copy should be trusted, not a promise that every crawler will pick up a change within a day.

Examples you can adapt

Four examples, each showing the complete file. The first two are generated directly by the tool; the last two need blocks merged by hand, as noted.

Block every PDF file, in any folder. Type the pattern in the Disallow box; the generator emits it verbatim.

  • User-agent: *
  • Disallow: /*.pdf$

Block query-string URLs, but keep one parameter crawlable. The Allow rule matches 16 characters against the Disallow rule's 6 for the URL /page?ref=public, so the longer rule wins.

  • User-agent: *
  • Disallow: /*?
  • Allow: /*?ref=public

Express a no-training preference to the three operators, and add the shared token. The first group names the vendor-specific training controls; the second group is the evolving shared signal, which states a usage preference and blocks no crawler. Blocking Google-Extended can affect Gemini grounding. Generate once per named agent and merge the blocks, or add the lines by hand — the shared token is typed into the user-agent field exactly as shown.

  • User-agent: GPTBot
  • User-agent: ClaudeBot
  • User-agent: Google-Extended
  • Disallow: /
  • User-agent: __usage_ai-training
  • Disallow: /

Allow one crawler and block the rest. Googlebot follows its named group; every other crawler follows the wildcard group. Two blocks merged by hand. Google's AdsBot crawlers do not match the wildcard and would need their own named group.

  • User-agent: Googlebot
  • Allow: /
  • User-agent: *
  • Disallow: /

Test cases: ten rules, checked

  • Tie: Allow: /page and Disallow: /page — /page is ALLOWED. Equal-length rules: Allow wins (RFC 9309 §2.2.2).
  • Longer rule wins: Allow: /private with Disallow: /private/page — /private/page is DISALLOWED.
  • Wildcard and anchor: Disallow: /*.pdf$ — /docs/file.pdf is DISALLOWED, but /docs/file.pdf?x=1 is ALLOWED: the $ anchor requires the address to end at .pdf.
  • End anchor: Disallow: /page$ — /page is DISALLOWED, /page2 is ALLOWED.
  • Empty value: Disallow: with nothing after the colon — everything is ALLOWED.
  • Trailing slash: Disallow: /docs/ — /docs and /docs2/ are ALLOWED; /docs/x is DISALLOWED.
  • Specific group replaces generic: a Bingbot group with Disallow: /b alongside a * group with Disallow: /a — Bingbot may fetch /a (ALLOWED) but not /b (DISALLOWED); other crawlers fall back to the * group.
  • Case matters: Disallow: /Page — /page is ALLOWED, /Page is DISALLOWED.
  • Percent-encoding compared literally: Disallow: /foo%20bar — /foo%20bar is DISALLOWED. Crawlers differ on normalization; the matcher compared the encoded form as written.
  • Broad Allow, narrow Disallow: Allow: / with Disallow: /private — / and /other are ALLOWED; /private/x is DISALLOWED.

Outcomes checked 18 November 2026 with a matcher implemented from RFC 9309 §2.2.2 (Python 3.12.3). CPython's urllib.robotparser was cross-checked and disagreed on four cases: it applies the first matching rule in file order and does not honor the $ anchor, so it was not used as the authority.

To confirm a request really is Bingbot, check the reverse DNS of its IP address: it should resolve to a search.msn.com hostname, and a forward lookup of that hostname should return the same IP. Bing publishes its crawler IP list as a machine-readable file for this check.

How to use the Robots.txt Generator

The Robots.txt Generator on this site assembles the file as you type and runs entirely in your browser. Generate it, read the result once with fresh eyes, upload it to the root of your domain, and confirm it loads at yoursite.com/robots.txt.

Two honest limits shape how to use it. The generator writes exactly what is typed — paths are trimmed and blank lines dropped, but nothing is checked against the real site, and no warning appears if a path does not exist or a rule blocks more than intended. And it produces one user-agent group per file, so multi-crawler files are assembled by merging generated blocks by hand. Generate, read the file once with fresh eyes, upload it, then verify it live and in Search Console's robots.txt report, which shows how Google interprets the actual rules against real addresses. Everything runs in the browser: nothing is uploaded or stored.

Mistakes that block more than intended

  • Disallow: / on a live site: the staging file copied to production blocks every crawler from every path.
  • A missing trailing slash: Disallow: /private matches /privacy-policy as well as the folder.
  • An empty Disallow: meant as a block: with nothing after the colon it allows everything.
  • Two crawler names on one User-agent line: the line matches no crawler at all.
  • Trusting robots.txt for privacy: the file is public, and bad bots ignore it.
  • Blocking the CSS, scripts or images a crawler needs to render pages: test the file in Search Console before relying on it.

Each of these is a reading error, not a syntax error: the file parses cleanly and the crawler obeys it exactly — it just does not do what was meant. Read every rule as 'everything starting with this', count characters when rules overlap, and test the file against real addresses before trusting it.

Conclusion

A robots.txt file is five ideas: groups name the crawler, rules name the paths, the longest match wins, the default is open, and the file is a request rather than a lock. Everything else — wildcards, sitemap lines, AI training controls, crawl-delay — builds on those five. Write the fewest rules the site needs, keep each rule's prefix in mind, and verify the file where it lives.

The free Robots.txt Generator produces the file in seconds: one group, allow and disallow paths, an optional crawl-delay and sitemap line, copied or downloaded as robots.txt. Generate it, read it once, upload it to the root, and check it in Search Console. The file is small; the consequences of getting it wrong are not.

Sources

1. IETF, RFC 9309: Robots Exclusion Protocol (September 2022) — https://www.rfc-editor.org/rfc/rfc9309.html — groups and rule matching; most-specific (longest) match wins; Allow wins equal-length ties; * and $ special characters; 500 KiB parsing limit; 24-hour caching guidance; rules are not access authorization. Accessed 15 November 2026.

2. Google Search Central, Create and Submit a robots.txt File — https://support.google.com/webmasters/answer/6062596?hl=en — default allow; root location and protocol/host/port scope; UTF-8; case-sensitive rules; group selection and combining; AdsBot wildcard exception; sitemap rules; wildcard support; testing via the Search Console robots.txt report. Accessed 15 November 2026.

3. Google for Developers, Overview of Google Crawlers and Fetchers — https://developers.google.com/crawling/docs/crawlers-fetchers/overview-google-crawlers — common crawlers always respect robots.txt; special-case crawlers; user-triggered fetchers. Accessed 15 November 2026.

4. Google for Developers, Google's Common Crawlers — https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers — Google-Extended has no separate HTTP user-agent string; standalone product token controlling training of future Gemini generations and grounding in Gemini Apps and Vertex AI; no effect on Google Search inclusion or ranking. Accessed 15 November 2026.

5. OpenAI, Overview of OpenAI Crawlers — https://developers.openai.com/api/docs/bots — GPTBot and OAI-SearchBot settings independent; OAI-SearchBot opt-out removes ChatGPT search answers (navigational links remain); ChatGPT-User requests may not be governed by robots.txt. Accessed 15 November 2026.

6. Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler? — https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler — ClaudeBot, Claude-SearchBot and Claude-User roles; Crawl-delay support; recognition of the shared training-preference signal from 30 October 2026; rules naming ClaudeBot take priority over the general signal. Accessed 15 November 2026.

7. IETF Internet-Draft, draft-kelley-aipref-training-preference-signal-00, Machine-Readable Training Preference Signal (30 October 2026; work in progress, expires 3 May 2027) — https://datatracker.ietf.org/doc/draft-kelley-aipref-training-preference-signal/ — __usage_ai-training policy token; usage preference, not a directive for crawling behaviour; independent of crawler rules; wildcard groups do not match the token. Accessed 15 November 2026.

8. Google Search Central Blog, Supporting a shared AI training control for websites (30 October 2026) — https://developers.google.com/search/blog/2026/10/ai-training-control — planned support for the shared control in 2027; token opt-out interpreted as disallowing Google-Extended unless explicitly specified; Google-Extended offered since 2023. Accessed 15 November 2026.

9. The original robots.txt proposal (1994) — https://www.robotstxt.org/orig.html — the convention's origin; one simple file agreed between crawlers and sites. Accessed 18 November 2026.

10. Google, Robots.txt Specification — https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec — longest-match rule; wildcard and end-anchor behavior. Accessed 18 November 2026.

11. Bing Webmaster Help, Which Crawlers Does Bing Use? — https://www.bing.com/webmasters/help/which-crawlers-does-bing-use-8c184ec0 — Bingbot and other Bing crawler tokens. Accessed 18 November 2026.

Frequently asked questions

If an Allow and a Disallow rule are equally specific, which wins?

The Allow rule. Crawlers use the most specific matching rule, measured by the number of characters matched, and when an Allow and a Disallow match with exactly equal length, the standard says the Allow rule should be used. Disallow: /page together with Allow: /page means /page is allowed. When the lengths differ, the longer rule wins whichever kind it is.

Do wildcards work in robots.txt?

Yes, for crawlers that follow the current standard. An asterisk matches any sequence of characters and a dollar sign anchors the match to the end of the URL, so Disallow: /*.pdf$ blocks every URL ending in .pdf. Both are defined in RFC 9309 and supported by Google's crawlers. A crawler built to older behaviour may treat them literally, so test wildcard rules against the crawlers that matter to you.

Can I put several crawler names on one User-agent line?

No. A User-agent line holds one name, and a line containing two names matches no crawler. To give several crawlers the same rules, write one User-agent line per name at the top of a single group, followed by the shared rules. The generator takes one name per file, so generate once per crawler and merge the blocks, or add the extra User-agent lines by hand.

Does blocking GPTBot remove my site from ChatGPT answers?

No. GPTBot collects content that may be used to train models, and OpenAI states that each of its crawler settings is independent of the others. ChatGPT search is controlled separately by OAI-SearchBot: blocking that crawler removes the site from ChatGPT search answers, while blocking GPTBot only signals that the content should not be used for training. The same training/search split applies to Anthropic's ClaudeBot and Claude-SearchBot.

How big can a robots.txt file be?

RFC 9309 requires crawlers to parse at least the first 500 kibibytes of the file, and beyond that a crawler may stop reading — so keep important rules near the top and keep the file well under that size. Most files are a few kilobytes. A file approaching the limit usually means many generated rules that a few wildcard patterns would replace.

Do I need a robots.txt file if I want everything crawled?

No. With no file, crawlers assume everything is allowed, because the default is open. Many sites still publish a minimal file — a User-agent: * group with no Disallow lines and a Sitemap line — to advertise where the sitemap lives. The generator produces exactly that file when the Allow and Disallow boxes are left empty and only a sitemap address is entered.

More guides on the QuikTools Hub blog.