PerplexityBot: How Perplexity Crawls Your Site

What is PerplexityBot?

PerplexityBot is the web crawler operated by Perplexity AI that indexes pages so they can appear as sources and links in Perplexity's answers. According to Perplexity's crawler documentation, its job is to "surface and link websites in search results," it honors robots.txt directives, and it does not collect content for training AI foundation models.

That last point separates it from training crawlers like GPTBot. In the taxonomy we use across this blog, PerplexityBot belongs to the search-index class of AI bots, the ones that determine whether you exist inside an answer engine. Our four classes of AI bots guide maps all four bot types side by side; this page goes deep on Perplexity's pair only.

The pair part is what the typical "what is PerplexityBot" page misses.

PerplexityBot vs Perplexity-User: two agents, two rule sets

Perplexity operates two distinct agents, and they follow different rules. PerplexityBot crawls proactively to build the index. Perplexity-User fetches a specific page reactively, at the moment a user's question needs it.

	PerplexityBot	Perplexity-User
Trigger	Automatic crawling	A real user's question
Purpose	Index pages for Perplexity search results	Fetch a page to answer one query
robots.txt	Respected	"Generally ignores robots.txt rules" (official docs)
Used for model training	No	No
IP list	perplexity.com/perplexitybot.json	perplexity.com/perplexity-user.json

The robots.txt line deserves emphasis because it comes from Perplexity itself, not from critics. The official docs say Perplexity-User "generally ignores robots.txt rules" because the fetch represents a human asking for a specific page, closer to a browser visit than to crawling.

The practical consequence: a Disallow rule for Perplexity affects indexing, not access. You can remove yourself from Perplexity's search index, but if a user pastes your URL or asks about your product directly, Perplexity-User can still fetch the page. Site owners deciding policy need to understand they are setting visibility, not building a wall. The same crawl-versus-fetch split exists at OpenAI, where GPTBot handles training data and OAI-SearchBot handles search indexing, each with its own controls.

User-agent strings and how to verify real visits

Both agents identify themselves in the user-agent header. From Perplexity's documentation, verbatim:

PerplexityBot:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Perplexity-User:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

A user-agent string is plain text, and anyone can send it. Scrapers impersonate well-known AI bots to borrow their welcome. Perplexity publishes the source IP addresses for both agents as JSON endpoints (linked above), so the reliable verification is two-step: match the UA string, then confirm the request IP appears in the published list. Cloudflare's verified-bots program automates exactly this pairing, and other major WAFs offer similar checks.

How to find Perplexity traffic in your logs

A case-insensitive search for perplexity across your access logs catches both agents in one pass, since both UA strings contain the company name. Then split the results:

• PerplexityBot hits look like crawling: multiple pages per visit, recurring over days, no referrer. Volume scales with your site size and how often your content changes.
• Perplexity-User hits look like single shots: one URL, then silence. Each one is a person whose question pulled your page into an answer in real time. These are arguably the more interesting line in the log, because they represent live demand.

If you see neither, that is also data. Either Perplexity has no reason to visit yet, or something upstream blocks it. WAF defaults are the usual upstream suspect; our breakdown of Cloudflare Bot Fight Mode and AI agents on Shopify stores shows how stores block AI bots without ever deciding to.

The Cloudflare dispute, with both sides

In August 2025, Cloudflare published research accusing Perplexity of evading crawl restrictions. Any article about PerplexityBot that skips this episode is incomplete, so here it is with both positions sourced.

Cloudflare's claim (August 4, 2025): after sites blocked Perplexity's declared crawlers, Cloudflare observed requests answering Perplexity queries arriving from a generic Chrome user agent, on IPs and networks outside Perplexity's published ranges. Their test used fresh domains with robots.txt disallowing all bots; Perplexity still returned detailed content about them. Cloudflare estimated the declared crawler at 20-25 million daily requests and the undeclared traffic at 3-6 million, de-listed Perplexity as a verified bot, and added blocking rules for customers.

Perplexity's rebuttal, published days later and covered by SDxCentral among others: the company argued Cloudflare conflated user-driven fetching with crawling, and misattributed traffic from BrowserBase, a third-party cloud browser service Perplexity says it uses for under 45,000 daily requests, a fraction of the 3-6 million figure. The deeper argument: when a human asks about a page, fetching it on their behalf is the user browsing, not a bot crawling, so robots.txt does not apply.

We are not the referee here. Two takeaways survive regardless of whose framing you accept. First, user-agent blocking has structural limits; identity on the web is verified by IP, not by self-declared strings. Second, the industry has no settled definition of which automated visits robots.txt governs, and Perplexity's own docs (the "generally ignores" line) state the ambiguity plainly. Set your policy knowing the boundary is contested.

Should you block PerplexityBot?

For most DTC merchants, no. Blocking PerplexityBot removes your store from the index behind Perplexity's answers, and Perplexity's whole pitch is research with cited sources: the product-comparison and "best X for Y" questions shoppers ask before buying. Absence there is a silent loss; no analytics line shows you the recommendation you didn't get.

A decision framework in three questions:

1. Does Perplexity send or influence revenue? Check logs for both agents and analytics for perplexity.ai referrals. Visits from Perplexity-User mean your pages already appear in answers.
2. What would blocking actually achieve? Neither agent trains models on your content, per Perplexity's docs. The usual reasons to block training crawlers do not apply; you would be giving up answer-engine visibility to stop indexing.
3. Is there a real cost concern? A store with heavy crawl load and zero Perplexity-attributed value can rate-limit rather than block outright, and revisit quarterly.

If you do decide to block, do it explicitly in robots.txt with User-agent: PerplexityBot and Disallow: /, and know its limits: it asks the index crawler to stay out, while user-triggered fetches continue under Perplexity's stated policy.

If instead you want to be cited more, the bottleneck is usually not bot access but whether your pages are extractable once fetched. A free Crawloria audit checks both layers, crawl access and content extraction, in about a minute.

FAQ

Does PerplexityBot collect content for AI training?

No. Perplexity's documentation states neither PerplexityBot nor Perplexity-User collects content for training AI foundation models. PerplexityBot indexes pages for search results; Perplexity-User fetches pages to answer individual questions.

Does Perplexity respect robots.txt?

Split answer, from Perplexity's own docs: PerplexityBot respects robots.txt; Perplexity-User generally does not, because it acts on a specific user request. Cloudflare's 2025 research disputed how cleanly the declared behavior held in practice, and Perplexity contested the findings.

Can I block the crawler but stay citable in Perplexity?

Not really. Citations in Perplexity answers come from the index PerplexityBot builds, so blocking the crawler shrinks your citable footprint to pages users request directly. If citation visibility matters to you, the crawler needs access.

How is PerplexityBot different from GPTBot?

Purpose. GPTBot gathers training data for OpenAI's models; PerplexityBot builds a search index and is not used for training, per each company's documentation. Blocking GPTBot is a stance on training-data use; blocking PerplexityBot is opting out of an answer engine's results. Our GPTBot guide covers the OpenAI side in the same depth.