Free Cloudflare diagnostic · reviewed July 2026

Does robots.txt allow AI search—but Cloudflare still blocks it?

Compare the public rule with the homepage response seen by synthetic OAI-SearchBot, Claude-SearchBot, and PerplexityBot requests. The mismatch is the evidence; no Cloudflare login is required.

Robots policy + synthetic response

Check the two layers together.

Enter one public homepage. The free check compares the robots decision with the response seen by synthetic OpenAI, Anthropic, and Perplexity search-crawler requests.

No account. Public homepage and robots.txt only. The request is synthetic and does not impersonate an authenticated provider bot.

Two controls can disagree.

robots.txtPublishes a crawler preference by user agent and path. It is not firewall enforcement and does not authenticate the requester.
Cloudflare WAFCloudflare documents that AI Crawl Control blocks use WAF custom rules. Those rules run before Cloudflare bot products and can return a restricted response even when robots.txt permits the path.
Bot identityA provider can verify its crawler through published IP ranges or other signals. This free check sends only a synthetic user-agent request, so it cannot prove how a verified provider crawl would be classified.

Check the current Cloudflare controls.

  1. 1

    Review AI Crawl Control → Crawlers. Cloudflare lets you allow or block individual AI crawlers and shows unsuccessful response classes.

  2. 2

    Review Security Settings → Configure AI bot policies. Search, Agent, and Training are separate behaviors. Do not assume a legacy “Block AI bots” setting represents every current policy.

  3. 3

    Inspect WAF custom rules and bot settings. A separate custom rule, Bot Fight Mode, rate limit, origin rule, or application check can still change the response.

  4. 4

    Run the comparison again. A reachable synthetic response confirms only this external path at this time; it does not prove provider identity, crawling, indexing, citation, or traffic.

Current primary references: Cloudflare’s Manage AI crawlers, AI Crawl Control with WAF, and AI bot policy configuration. The controls were reviewed July 13, 2026 and can change.

Why this checks search crawlers first.

OAI-SearchBot, Claude-SearchBot, and PerplexityBot support search or answer discovery. Their access is a different policy decision from model-training crawlers such as GPTBot and ClaudeBot. Use the broader checker when you need all eight current policy tokens.

Check all eight crawler policies

A mismatch narrows the investigation. It does not name the cause.

If robots permits a crawler and the synthetic response is 401, 403, or 429, inspect Cloudflare and origin controls. The response could still come from another CDN, a hosting firewall, rate limiting, authentication, or application logic. The checker does not inspect your account or alter any setting.

If both layers appear open, the site is reachable along this synthetic path. That still does not guarantee a provider will crawl, index, cite, rank, recommend, or send traffic.