Which AI crawlers can access your website?
Search bots, user-requested fetchers, training crawlers, and ad validators do different jobs. Check the public rules provider by provider before changing robots.txt.
Check the live homepage rules.
Fetch the public /robots.txt response and resolve the longest matching rule for each crawler token. No account and no private pages.
Allow discovery. Restrict training.
This conservative example separates automated search and user retrieval from training controls. Merge it with existing rules; do not overwrite the full file.
# Search and answer discovery
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# User-requested retrieval
User-agent: Claude-User
Allow: /
# Model training and Gemini grounding control
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
One “AI bot” switch is the wrong model.
OAI-SearchBot, Claude-SearchBot, and PerplexityBot support search or answer discovery. Blocking them can reduce eligibility for those providers' search experiences.Claude-User supports a person's request to retrieve a page. Other providers also use user-directed fetchers, but robots.txt behavior is not uniform.GPTBot and ClaudeBot represent training-related collection controls. They are independent from the corresponding search bots.Google-Extended controls certain Gemini training and grounding uses of content Google already crawls. Google states that it does not affect Google Search inclusion or ranking.OAI-AdsBot checks landing pages submitted for ChatGPT advertising. Most sites do not need to allow it unless they use that product.Generate the policy for one provider.
Use the broad checker to see what is live. Use a focused generator when you are ready to change one provider without copying an all-purpose AI-bot block.
Official sources used for this checker
- OpenAI publisher guidance separates OAI-SearchBot discovery from GPTBot training controls.
- Anthropic crawler guidance defines ClaudeBot, Claude-User, and Claude-SearchBot.
- Perplexity crawler documentation defines PerplexityBot and distinguishes its user-requested fetcher.
- Google crawler documentation defines the Google-Extended control token and its limits.
Crawler names and purposes can change. This page records its review date, links to primary sources, and tests only the public robots response—not IP authenticity or network-level reachability.
Run the same check from a terminal.
The dependency-free open-source CLI returns the same eight policy decisions in readable or JSON form. It rejects private-network targets, limits responses to 1 MB, and times out bounded fetches.
brew install unitedideas/tap/actablesite-checkThen run
actablesite-check example.com. The formula passed installation, its built-in test, and a live eight-crawler check on GitHub's macOS runner. Review the tap and workflow.npx github:unitedideas/actablesite-check#v1.3.0 example.comRuns the pinned v1.3.0 release without a global install. Review the source, tests, and MIT license.
uses: unitedideas/actablesite-check@v1Adds a scheduled or pull-request policy check with optional fail-on-blocked enforcement. Copy the workflow and review the operating modes.
Both paths use the same public release and require no ActableSite account. The Homebrew path installs the command for repeated checks; the tagged npx path is useful for a single run or CI pin.
What the result does not prove
An allowed result means no matching robots.txt restriction won for that token on the homepage. It does not prove that a provider can pass your firewall, that the requester is an authentic provider bot, or that any system will crawl, index, cite, recommend, rank, or send traffic. A blocked result is also not access control; keep private content behind authentication.
If the site uses Cloudflare, run the Cloudflare AI crawler checker to compare the robots decision with synthetic OAI-SearchBot, Claude-SearchBot, and PerplexityBot homepage responses.