ClaudeBot: What It Is and How to Control It

What is ClaudeBot?

ClaudeBot is the web crawler operated by Anthropic to collect publicly accessible content for building and improving Claude's AI models. Per Anthropic's own crawler documentation (checked June 2026), ClaudeBot "helps enhance the utility and safety of our generative AI models by collecting web content." It reads public pages the way a search-engine crawler does, but the destination is a training corpus rather than a search index.

Two operating principles are consistent with Anthropic's documentation. First, ClaudeBot only touches publicly accessible content and does not attempt to access content behind paywalls or authentication. Second, it honors robots.txt rules addressed to the ClaudeBot token.

ClaudeBot is not the only Anthropic bot, and confusing the three is the most common mistake in making a block-or-allow decision. The full picture is in the next section.

For where ClaudeBot fits in the wider taxonomy of AI crawlers, training scrapers, search indexers, and autonomous agents, see our four classes of AI bots breakdown.

Anthropic's three crawlers and what each one does

Anthropic operates three distinct bots, and robots.txt treats each independently. Per Anthropic's crawler documentation (checked June 2026):

Bot	Purpose	robots.txt token	What blocking costs you
ClaudeBot	Collects content for Claude model training	ClaudeBot	Your content stays out of future Claude training runs. No effect on Claude search visibility.
Claude-SearchBot	Indexes sites for Claude's web search features	Claude-SearchBot	Your pages stop appearing in Claude's web search results.
Claude-User	Fetches a specific page when a real user's Claude session needs it	Claude-User	User-initiated page fetches via Claude fail.

The row that most guides miss: blocking ClaudeBot does nothing to your visibility in Claude's web search answers, and allowing it does not improve your Claude search presence. Training collection and search indexing run through separate crawlers with separate tokens.

Compare this structure to OpenAI's, where GPTBot handles training, OAI-SearchBot handles ChatGPT search, and ChatGPT-User handles live page fetches. Anthropic follows the same architectural split with its own three-bot system. For the full cross-vendor taxonomy, see four classes of AI bots.

The ClaudeBot user-agent string and how to verify real visits

The identifying token in the ClaudeBot user-agent string is ClaudeBot. For the current full user-agent string, check Anthropic's crawler documentation at support.claude.com/en/articles/8896518. In any log filter or firewall rule, match on ClaudeBot rather than the full string, since version numbers in the UA string update over time.

A user-agent string is plain text, and anyone can fake one. Scrapers impersonate ClaudeBot to borrow its welcome at sites that allow Anthropic's bots. Anthropic publishes the current IP ranges at https://claude.com/crawling/bots.json. A request claiming to be ClaudeBot from an IP outside those ranges is an impostor and a legitimate target for your firewall.

Verification workflow:

1. Pull a suspect request's IP from your logs.
2. Fetch claude.com/crawling/bots.json and check whether the IP falls inside any listed CIDR block.
3. In range: genuine ClaudeBot. Out of range: an impostor wearing the user-agent string.

Claude-SearchBot and Claude-User are covered by the same JSON file; it lists IP ranges for all three Anthropic crawlers. Keep the three tokens separate in your log queries to understand which bot is doing what.

How to check ClaudeBot activity in your logs

One grep against your access logs answers whether ClaudeBot visits and how frequently. On any server with standard combined-format logs:

grep -i "claudebot" access.log | wc -l

To break out the requesting IPs:

grep -i "claudebot" access.log | awk '{print $1}' | sort | uniq -c | sort -rn

Cross-reference those IPs against claude.com/crawling/bots.json. Anything claiming ClaudeBot from outside the listed ranges is spoofed traffic.

To check all three Anthropic crawlers in a single pass:

grep -iE "claudebot|claude-searchbot|claude-user" access.log | awk '{print $1, $7, $9}' | sort

On Shopify or other hosted platforms where raw logs are not exposed, a CDN analytics layer (Cloudflare's bot reports, for example) gives the same view with less effort.

How to block or allow ClaudeBot in robots.txt

To block ClaudeBot from your entire site:

User-agent: ClaudeBot
Disallow: /

To keep it out of specific paths only:

User-agent: ClaudeBot
Disallow: /members/
Disallow: /research/

To block all three Anthropic crawlers at once:

User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /

Three operational notes before you ship that file.

First, a disallow is forward-looking. It stops future crawls but does not pull content out of models already trained on it.

Second, be precise with tokens. A Disallow rule for ClaudeBot does not touch Claude-SearchBot or Claude-User. They have separate tokens and require separate rules.

Third, it is a request, not a wall. Compliant crawlers honor it; spoofed ones don't. For enforcement beyond robots.txt, IP-range verification at the edge is the reliable layer.

Should you block ClaudeBot in 2026?

For most DTC stores and content sites that want any AI visibility, blocking ClaudeBot alone is low priority. For businesses whose content is the product, it may be a reasonable default. The honest answer is a cost-benefit question.

The case for blocking is real for content businesses: original research, journalism, licensed creative work, or anything that earns money by controlling access to information. ClaudeBot collects content for Claude's training runs, which transfers value to Anthropic without compensation. A disallow rule is the documented way to refuse.

The case for allowing comes from visibility. Content that enters training data shapes what Claude knows about your brand, your product category, and your claims. A model that has never read you cannot describe you accurately when a user asks. Beyond training, the separate Claude-SearchBot determines whether your pages appear in Claude's web search results - and if you want that visibility, a blanket "block all Anthropic bots" rule takes down the search crawler alongside the training one.

Make the decision per-crawler rather than per-vendor. The configuration that allows Claude web search while opting out of training:

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

This keeps you visible in Claude's search answers while stopping future training data collection from your site.

A note on Claude-User. This bot operates differently from the other two. It fetches a specific page when a real user's Claude session requests it - closer to a browser visit than to crawling. Blocking Claude-User means that when someone asks Claude about a specific URL on your site, Claude cannot fetch the current page to answer. For most merchants, allowing Claude-User while controlling ClaudeBot is the right default.

FAQ

Does ClaudeBot respect robots.txt?

Yes. Anthropic's documentation states ClaudeBot honors robots.txt directives. Compliance is voluntary; enforcement against spoofed or non-compliant traffic requires IP-level verification against claude.com/crawling/bots.json.

Does blocking ClaudeBot affect Claude's web search results?

No. ClaudeBot collects training data; Claude-SearchBot indexes pages for web search. Blocking ClaudeBot with a robots.txt rule does nothing to your visibility in Claude's search answers. To opt out of Claude's web search, use a Disallow rule for Claude-SearchBot separately.

Does blocking ClaudeBot remove content already in Claude's training data?

No. A robots.txt disallow stops future crawls. Content collected before the rule existed stays in whatever training data was built from it. There is no retroactive removal mechanism via robots.txt.

How is ClaudeBot different from PerplexityBot and GPTBot?

All three are crawlers operated by separate AI companies. GPTBot (OpenAI) and ClaudeBot (Anthropic) both collect training data; blocking either opts you out of those companies' future training runs. PerplexityBot and Claude-SearchBot index pages for search answers; blocking either removes you from those platforms' search results. The GPTBot guide, PerplexityBot guide, and OAI-SearchBot guide cover the OpenAI and Perplexity bots at the same depth as this article.

What should a DTC merchant's default Anthropic bot policy be?

Allow Claude-SearchBot and Claude-User (these drive citation visibility and user-initiated access), and make a separate call on ClaudeBot based on whether your content is a licensable asset. For most DTC brands whose product pages and blog content exist to be found and recommended, allowing ClaudeBot is fine. For brands with genuinely proprietary content they want to protect, a path-scoped or full disallow on ClaudeBot is the right call.