Anthropic crawler controls · reviewed July 2026

Control Claude search, retrieval, and training separately.

Anthropic documents three crawler tokens with different jobs. Set a rule for each one instead of treating every Claude request as the same bot.

Free interactive generator

Choose each Claude use independently.

The default allows Claude search and user-requested retrieval while blocking model-training collection. Change each choice to match your publishing policy.

01 Claude search

Claude-SearchBot supports search discovery and result quality.

02 User retrieval

Claude-User supports a person asking Claude to retrieve a page.

03 Model training

ClaudeBot controls collection that may contribute to model training.

Generated robots.txt blockReview before publishing
# Claude search discovery

User-agent: Claude-SearchBot
Allow: /

# User-requested retrieval

User-agent: Claude-User
Allow: /

# Anthropic model training

User-agent: ClaudeBot
Disallow: /

Add this block to the existing root /robots.txt file without deleting unrelated rules. Then verify the public response.

Check the live rules

Three tokens, three decisions.

Claude-SearchBotAnthropic says this bot navigates the web to improve search-result quality and relevance. Blocking it may reduce your site's visibility in Claude search results.
Claude-UserSupports a person asking Claude to retrieve web content. Blocking it can prevent user-directed access to your pages.
ClaudeBotCollects public web content that may contribute to model training. This training control is independent from search and user retrieval.

Purpose and behavior are based on Anthropic's current crawler guidance, reviewed July 11, 2026.

Choose the policy that matches your intent.

Discovery onlyUse the generator defaults to allow Claude-SearchBot and Claude-User while blocking ClaudeBot.
Allow all threeGive each token its own group with Allow: /. Separate groups keep later policy changes explicit.
Block all threeGive each token its own group with Disallow: /. Repeat the policy on every subdomain you intend to cover.
Slow ClaudeBotAnthropic says it supports the non-standard Crawl-delay directive. Use it only when you want crawling with a lower request rate.

Verify the rule that is actually public.

  1. 1

    Edit the root file: update https://your-domain.com/robots.txt, not an example file hidden in a CMS or repository.

  2. 2

    Keep groups separate: explicit per-token groups make search, retrieval, and training choices independently readable.

  3. 3

    Check the live response: test the public hostname after caches and deployments have updated.

  4. 4

    Inspect other access layers: a permissive robots rule does not bypass authentication, a WAF, CDN bot challenge, rate limit, or network block.

Using Cloudflare? Compare the robots decision with a synthetic Claude-SearchBot response. The result can expose a mismatch, but it cannot authenticate Anthropic or identify the blocking rule.

What robots.txt can—and cannot—do.

Robots rules communicate crawler preferences; they are not access control. Keep private content behind authentication. An allowed result also cannot prove that Anthropic will crawl, index, retrieve, cite, recommend, rank, or send traffic to a page.

Anthropic says its bots honor robots.txt and warns that blocking provider IP addresses can interfere with its ability to read the policy file. It does not currently publish stable IP ranges for these bots.

Claude crawler questions

What is the difference between ClaudeBot and Claude-SearchBot?

Anthropic describes ClaudeBot as a crawler for content that may contribute to model training. Claude-SearchBot supports search discovery and result quality. They use separate robots.txt tokens.

Can I allow Claude search and block model training?

Yes. Put Claude-SearchBot and ClaudeBot in separate robots.txt groups, allow Claude-SearchBot, and disallow ClaudeBot. Claude-User is a third independent choice for user-requested retrieval.

Does an Allow rule guarantee that Claude will cite my site?

No. A permissive robots.txt result only removes that policy restriction. It does not guarantee crawling, indexing, retrieval, citation, recommendation, ranking, or traffic.