Perplexity crawler controls · reviewed July 2026

Control Perplexity search indexing without confusing it with user fetches.

Perplexity documents an automated search crawler and a separate user-requested fetcher. Generate the search rule first, then verify robots, WAF, and IP controls as different layers.

Free PerplexityBot generator

Choose automated search indexing.

This rule addresses PerplexityBot. It does not promise to control every user-requested fetch or network request associated with Perplexity.

01 Perplexity search

PerplexityBot indexes pages for Perplexity search results and is documented as separate from model training.

Generated robots.txt blockReview before publishing
# Perplexity automated search crawler

User-agent: PerplexityBot
Allow: /

Add this block to the existing root /robots.txt file without deleting unrelated rules. Then verify the public response.

Check PerplexityBot live

PerplexityBot and Perplexity-User are different access paths.

PerplexityBotAutomated crawler used to surface and link websites in Perplexity search results. Perplexity recommends allowing it for search visibility and says it is not used for foundation-model training.
Perplexity-UserFetcher used when a person asks Perplexity to access a page. Perplexity's crawler documentation says this user-requested path generally ignores robots.txt, while its help center says an older direct-URL summarization behavior was disabled.

The provider's current pages describe both the crawler distinction and a product-policy change. Review Perplexity crawler documentation and its robots.txt help article before relying on one rule for every request.

Robots permission does not bypass your WAF.

Perplexity recommends combining a matching user-agent with its current published IP ranges when configuring a firewall. A permissive robots rule can still fail behind a CDN challenge, bot rule, rate limit, authentication gate, or stale allowlist.

RobotsResolve the exact public path against the PerplexityBot group and any wildcard rules.
Bot IPsUse the live PerplexityBot IP range file; do not paste a one-time address from a log into a permanent rule.
User IPsPerplexity publishes a separate Perplexity-User IP range file for user-requested traffic.
ResponseTest the real public page without a login and inspect the status, redirects, cache, and challenge response.

Using Cloudflare? Compare the robots decision with a synthetic PerplexityBot response. The comparison can reveal a mismatch, but it does not authenticate Perplexity or identify the blocking rule.

Verify the public result in four steps.

  1. 1

    Publish the root file: update https://your-domain.com/robots.txt and preserve unrelated crawler groups.

  2. 2

    Wait for propagation: Perplexity says crawler-setting changes can take up to 24 hours to appear in its systems.

  3. 3

    Check the live rule: resolve the homepage and any important path against the public file.

  4. 4

    Check the edge: verify the current published IP ranges and review WAF, CDN, authentication, and rate-limit logs.

What an allowed result cannot guarantee

Allowing PerplexityBot removes that robots restriction for automated search crawling. It does not guarantee a crawl, index entry, citation, ranking, recommendation, answer placement, or traffic. It also does not authenticate a requester or make private content safe to expose.

Perplexity crawler questions

Does PerplexityBot respect robots.txt?

Perplexity says its automated search crawler, PerplexityBot, respects robots.txt and will not index page text when the crawler is disallowed.

Is Perplexity-User the same as PerplexityBot?

No. Perplexity documents PerplexityBot as an automated search crawler and Perplexity-User as a fetcher used for a person's request. The provider documents different behavior for those paths.

Does allowing PerplexityBot permit AI model training?

Perplexity states that PerplexityBot is for search indexing and is not used to crawl content for AI foundation-model training.