Can AI crawlers read you? Check GPTBot, ClaudeBot and PerplexityBot
Open your-site.md/robots.txt in a browser. If you see the name of an AI crawler — GPTBot, ClaudeBot, PerplexityBot — followed by „Disallow: /”, that crawler is not reading your site. If the file is missing or does not name them, all of them have access by default.
This guide is the practical half of „GEO: how ChatGPT, Perplexity and Google AI decide to recommend you”, which explains why crawler access matters. Here it is just the how — check, then fix, step by step.
Where to find the file
The file always sits at the root of the site, never in a subfolder: your-site.md/robots.txt. Open it directly in a browser — it is plain text, no account or tool required.
Three possible results:
| What you see | What it means |
|---|---|
| A 404 error, the page does not exist | The file is missing. By default, every crawler has open access — AI included. |
| An empty file, or only comments | Same thing: open access, nothing blocked. |
| A file with rules | It needs reading line by line — see below. |
If you have several subdomains — blog.your-site.md, shop.your-site.md — each has its own robots.txt, separately. A blog hosted on a subdomain does not inherit the rules from the main domain.
How to read it, briefly
The file is organised in blocks. Each block starts with User-agent: — the name of the crawler it applies to — followed by Disallow: (forbidden) or Allow: (permitted) lines.
User-agent: GPTBot
Disallow: /
The block above forbids exactly one crawler, GPTBot. It does not affect any other crawler, including those not named explicitly.
User-agent: *
Disallow: /
The asterisk * means „every crawler that does not have its own block, above or below, under its exact name”. If you see this, and no AI crawler has its own block, you are blocked for everyone — Google included.
The general rule: a block with an exact name takes priority over the general * rule, regardless of the order in the file. If the file has Disallow: / for * but also Allow: / for OAI-SearchBot, that specific crawler still has access.
The crawlers that matter, with exact names
Here is the current list, checked directly against each company's own documentation — do not assume, check, because names change.
OpenAI
| Crawler | Role | Respects robots.txt |
|---|---|---|
GPTBot |
Collects text for model training | Yes |
OAI-SearchBot |
Fetches pages for sourced answers in ChatGPT | Yes |
OAI-AdsBot |
Checks ad landing pages | Yes |
ChatGPT-User |
Opens a page when a person explicitly asks, mid-conversation | Documented as generally outside the automatic crawling mechanism |
Anthropic (Claude)
| Crawler | Role | Respects robots.txt |
|---|---|---|
ClaudeBot |
Collects text for model training | Yes |
Claude-SearchBot |
Analyses content for the quality of search-backed answers | Yes |
Claude-User |
Fetches a page at a person's explicit request, mid-conversation | Documented as generally ignoring robots.txt, because the request comes directly from a person |
Perplexity
| Crawler | Role | Respects robots.txt |
|---|---|---|
PerplexityBot |
Indexes pages for Perplexity's search results — not used for model training | Yes |
Perplexity-User |
Fetches a page at a person's explicit request | Documented as generally ignoring robots.txt |
Unlike OpenAI and Anthropic, Perplexity does not document a separate training crawler — PerplexityBot is the search one.
| Crawler | Role | Affects Google Search ranking |
|---|---|---|
Google-Extended |
Controls whether your text helps train Gemini models | No — Google states explicitly it is not a ranking signal |
Googlebot |
The regular search crawler, unrelated to AI | Yes, it is the basis of indexing itself |
A robots.txt template, ready to copy
If you want access for search and citation crawlers but do not want your text used for model training, this is a reasonable starting point:
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Disallow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /
A few notes, so you do not copy it blindly:
- An explicit
Allow: /is optional, but clearer. Without any rule, a crawler has open access anyway. Writing it explicitly shows intent — useful to you a year from now, when you no longer remember why the file looks this way. - Blocking training (
GPTBot,ClaudeBot) does not remove you from cited answers. The search crawlers —OAI-SearchBot,Claude-SearchBot,PerplexityBot— are separate and stay active in the template above. - There is no single „correct” version. If you would rather your text was never used for training and never cited either, swap the
Allowlines forDisallow. That is your choice, not a technical one.
How to check the rules actually work
Writing the file is step one. Checking that crawlers follow it is step two, and it is often skipped.
- Re-read the live file, not the one in your editor. Open
your-site.md/robots.txtagain after publishing the change. Hosting or CDN caching can serve an old version for minutes or hours. - Check the robots.txt report in Search Console. Google removed the old „robots.txt Tester” tool; in its place, the robots.txt report in Search Console shows the latest version Google has read and flags syntax errors.
- For a specific page, use URL Inspection. Search Console tells you directly whether a page is „blocked by robots.txt” for Googlebot.
- For AI crawlers, no equivalent official tool exists. The safest method remains reading the file manually, against the exact names in the tables above.
Common problems
A security panel blocks crawlers without writing anything into robots.txt. Some security plugins or firewalls (WAFs) stop crawlers at the network level, by agent name, and this never shows up in robots.txt. The file can look wide open while the crawler still never reaches the page. If you suspect this, check the hosting's access logs, not just the text file.
Cloudflare can add rules automatically. The managed robots.txt feature writes directives without notifying you — details are in the GEO article.
An old plugin can overwrite the file on every save. Some WordPress SEO modules regenerate robots.txt from their own settings, ignoring anything you wrote by hand. If your edits keep vanishing, check there first.
An important nuance: robots.txt is a request, not a lock
Worth saying plainly, because it changes how you should look at the whole file.
robots.txt works through voluntary cooperation. The large companies — OpenAI, Anthropic, Perplexity, Google — publicly document their crawlers and confirm they honour the rules. But the standard, technically, stops nothing: an ad-hoc script that does not identify itself under any of the names above can read the page regardless.
The practical consequence: robots.txt is the right tool for expressing your preference toward the large, documented crawlers that actually respect it. It is not a security system and does not protect sensitive information — that needs authentication, not a public text file.
The first thing you can do today
Open your-site.md/robots.txt. Compare what you see against the tables above. If you find a name from the list followed by Disallow: / and cannot remember deciding that, you have found exactly why you are missing from an AI answer.
Sources
- OpenAI, official crawler documentation — GPTBot, OAI-SearchBot, OAI-AdsBot, ChatGPT-User.
- Anthropic (Claude), crawler documentation — ClaudeBot, Claude-SearchBot, Claude-User.
- Perplexity, crawler documentation — PerplexityBot, Perplexity-User.
- Google, crawler overview — Google-Extended and the confirmation that it does not affect Search ranking.
- Google, robots.txt report in Search Console — how to check the latest version Google has read.
- Google, guide to creating a robots.txt file — the full syntax and special cases.
Frequently asked questions
Allowed. A missing file means open access for every crawler, AI included. A missing robots.txt is not an access problem — but it can be a sign that nobody has ever looked at what crawlers read on the site.
Usually one of three sources: the site's theme, a security plugin, or an automatic feature from hosting or Cloudflare, such as managed robots.txt. None of them notify you when they change the file — which is why it is worth checking manually rather than assuming.
Not necessarily. Robots.txt is a polite request, followed voluntarily. The large crawlers — GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot — document that they honour it. But the file is not a technical lock like a password; it will not stop an ad-hoc script that ignores the rule.
Google removed it. Instead: the robots.txt report in Search Console shows whether Google read your file correctly, and the URL Inspection tool tells you, for a specific page, whether it is „blocked by robots.txt”. For AI crawlers, the safest method remains reading the file directly.
No. Every subdomain has its own robots.txt, at its own root — blog.your-site.md/robots.txt is separate from your-site.md/robots.txt. If your blog sits on a subdomain, check it separately.
MR. SEO · updated 18 September 2026