robots.txt Builder & Tester
Test 13 AI crawlers against your robots.txt, see the exact rule that decided, then fix it.
Paste mode runs entirely in your browser. Fetch mode asks our server to retrieve one robots.txt — nothing is stored either way.
About robots.txt Builder & Tester
robots.txt is the shortest file on your site and the one most likely to be quietly wrong. A single `Disallow: /` under a `User-agent: *` group blocks every AI crawler that has no group of its own, which is most of them. This tool takes a robots.txt — pasted, or fetched from a URL — and reports a verdict for each of the thirteen AI user-agents we track, with the rule line that produced it quoted verbatim.
Matching follows Google's documented semantics rather than a naive prefix check: the longest matching rule wins, an `Allow` beats a `Disallow` of equal length, `*` matches any run of characters and a trailing `$` anchors the end of the path. That matters — `Disallow: /` plus `Allow: /` in a GPTBot group resolves to allowed, and a tool that reports otherwise will send you chasing a problem you don't have.
The builder half preserves what you already wrote. It keeps your existing groups, keeps your `Sitemap:` lines, and appends explicit groups for the agents you admit and the ones you turn away. The output round-trips: parse the generated file and every bot lands on the same verdict the matrix showed.
Deciding which crawlers to admit is a business call, not a technical one. Training crawlers like GPTBot and CCBot take content and give nothing directly back. Search crawlers like OAI-SearchBot and PerplexityBot are what put a citation and a link in front of a reader. Plenty of sites reasonably allow the second group and block the first — the tool lets you express exactly that, per agent.
Frequently asked
›What is the difference between GPTBot and OAI-SearchBot?
GPTBot collects content for training OpenAI's models. OAI-SearchBot fetches pages so ChatGPT can cite them in a search answer, with a link back to you. Blocking GPTBot keeps you out of training data; blocking OAI-SearchBot keeps you out of answers. Most sites want those two decisions to be different.
›Does blocking a crawler in robots.txt actually stop it?
robots.txt is a request, not a fence. The major vendors documented here honour it, and Google, OpenAI, Anthropic and Perplexity all publish their tokens so you can name them. Anything that ignores robots.txt has to be stopped at the edge — a WAF rule or a firewall — not in a text file.
›Why does my `Disallow: /admin` rule not block `/admin`?
It should — check whether a longer `Allow` rule in the same group is winning. Longest match takes precedence, so `Allow: /admin/public` beats `Disallow: /admin` for any path under it. The matrix quotes the winning line, which is usually enough to spot the conflict.
›Where does robots.txt have to live?
At the root of the host, exactly `/robots.txt`, served as text with a 2xx status. It applies per host and per port and per scheme — `https://example.com/robots.txt` says nothing about `https://blog.example.com`. A robots.txt served as an HTML error page counts as no robots.txt at all.
›Will a `Sitemap:` line survive the generated file?
Yes. Sitemap declarations are non-group fields, so they're parsed out separately and re-emitted at the end of the generated file, deduplicated.