OAI-SearchBot: The Crawler Behind ChatGPT Search

What is OAI-SearchBot?

OAI-SearchBot is the web crawler OpenAI uses to build and refresh the index behind ChatGPT Search. When ChatGPT answers a question with web results and linked citations, the sites it can cite are, in large part, the sites OAI-SearchBot was able to crawl. OpenAI's crawler documentation describes it as "used to surface websites in search results in ChatGPT's search features" (checked June 11, 2026); training collection is documented as the job of a different crawler, GPTBot.

That single sentence carries the whole strategy question. Site owners spent 2023 and 2024 deciding whether to block OpenAI over training-data concerns. OAI-SearchBot is not that fight. It is a search engine crawler in the classic sense, much closer to Googlebot than to a training scraper: it visits so that your pages can appear, with links, in front of people asking questions. Blocking it does not keep your content out of model training; that lever is GPTBot's robots.txt entry. Blocking OAI-SearchBot only makes you invisible in ChatGPT Search.

For where this bot sits in the wider zoo of AI crawlers, agents, and fetchers, see our four classes of AI bots breakdown.

OAI-SearchBot vs GPTBot vs ChatGPT-User

OpenAI operates three distinct bots, and robots.txt treats each independently. The split, per OpenAI's documentation (checked June 11, 2026):

Bot	What it does	If you block it
OAI-SearchBot	Crawls to index pages for ChatGPT Search	Your pages stop appearing in ChatGPT Search results
GPTBot	Crawls content used to train OpenAI's foundation models	Your content stays out of future training runs; search visibility unaffected
ChatGPT-User	Fetches a specific page in real time when a user's ChatGPT session needs it	Live page fetches fail; note OpenAI says robots.txt rules may not apply to these user-initiated visits

The pairing that matters: a Disallow rule for GPTBot does nothing to OAI-SearchBot, and vice versa. A site can stay out of training data while remaining fully visible in ChatGPT Search; that is exactly the two-line robots.txt setup shown below. We cover GPTBot's own behavior, IP ranges, and the block-or-allow tradeoff in depth in the GPTBot guide.

The OAI-SearchBot user agent (and how to verify it is really OpenAI)

The full user-agent string, verbatim from OpenAI's crawler documentation as of June 11, 2026:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible;
OAI-SearchBot/1.3; +https://openai.com/searchbot

Two things to note. First, the string opens with a normal Chrome desktop signature; if your log tooling matches on "Chrome" before it matches on "OAI-SearchBot", the bot lands in the wrong bucket. Match on the OAI-SearchBot token. Second, the version number changes over time. Several bot directories still publish OAI-SearchBot/1.0, and at least one bot-protection vendor page circulates a string pointing to openarchives.org, which is a different OAI entirely (the Open Archives Initiative, a metadata standard from the library world). When strings disagree, trust openai.com's own documentation page and nothing else.

A user-agent string is plain text, and scrapers fake reputable ones daily. The reliable check is the source IP. OpenAI publishes the current ranges OAI-SearchBot crawls from in a machine-readable file at https://openai.com/searchbot.json. Verification workflow:

1. Pull a suspect request's IP from your logs.
2. Fetch openai.com/searchbot.json and check whether the IP falls inside any listed CIDR block.
3. In range: genuine OAI-SearchBot. Out of range: an impostor wearing the string, and a legitimate target for blocking.

GPTBot and ChatGPT-User have their own files (gptbot.json and chatgpt-user.json), so keep the three separate when you automate the check.

How to find OAI-SearchBot in your server logs

One grep answers whether ChatGPT Search can see you at all. On an nginx or Apache box:

grep "OAI-SearchBot" /var/log/nginx/access.log | tail -20

On Cloudflare, filter your traffic analytics or Logpush data on the user-agent containing OAI-SearchBot. On Shopify and similar hosted platforms, raw logs are not exposed; a server-side analytics tool or Cloudflare in front of the custom domain is the workaround.

How to read what you find:

• Steady hits returning 200 on your key pages: you are indexable in ChatGPT Search. The frequency varies by site; OpenAI publishes no crawl schedule.
• Hits returning 403 or 429: something on your edge (WAF, rate limiter, bot rule) is refusing the bot. This is the silent failure mode covered below.
• No hits at all over several weeks: either robots.txt already disallows it, your firewall drops it before logging, or the index simply has not reached you. Check the first two before assuming the third.
• Hits only on robots.txt: the bot checks permissions but never proceeds, which usually means a Disallow rule it is respecting.

In our audit work at Crawloria, bot-access checks like this one are a recurring failure point e-commerce sites hit without knowing it, which is why the free audit runs the access test against AI crawler user agents automatically.

Should you block OAI-SearchBot?

For most sites the answer is no, and the reasoning is short: OAI-SearchBot is how you exist inside ChatGPT Search, and ChatGPT Search sends referral traffic. We can offer one verified data point instead of an industry projection: ChatGPT is the #2 referrer of human traffic to crawloria.com, behind only Google (GA4, June 2026). A small site with no paid acquisition gets its second-largest visitor stream from the surface this crawler feeds.

The decision framework, by situation:

• Content business (DTC store, blog, SaaS marketing site): allow. You want to be cited, linked, and visited. There is no training-data cost; this bot does not feed training.
• Worried about model training, not visibility: block GPTBot, allow OAI-SearchBot. The two-line robots.txt below does exactly that.
• Paywalled or licensed content: a real case for blocking, with the understanding that you disappear from ChatGPT Search citations too.
• Server load concerns: rate-limit rather than block. Search crawlers generally back off on 429s; scrapers generally don't.

The robots.txt for the most common setup, visible in search but out of training:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

And the full block, if that is your call:

User-agent: OAI-SearchBot
Disallow: /

One operational note from OpenAI's documentation: their systems take roughly 24 hours to pick up a robots.txt change. A fix you ship today shows up in crawler behavior tomorrow, so do not judge the result from the next hour of logs. If your pages are missing from ChatGPT entirely, work through why ChatGPT isn't showing your website, where robots.txt is only the first suspect on the list.

Common ways sites block OAI-SearchBot by accident

In the audits we run, deliberate blocks are the exception and accidental ones come up constantly, and they share a cause: OAI-SearchBot is young enough that security defaults written before 2024 do not recognize it as a search crawler.

The patterns we see most:

1. WAF managed rules. "Block AI bots" toggles and managed rule sets often bundle search crawlers like OAI-SearchBot with training scrapers into one category. If the toggle predates the bot or the vendor lumps everything AI together, you are blocking your own ChatGPT visibility with one checkbox.
2. Cloudflare Bot Fight Mode. The free tier's bot mitigation challenges automated traffic broadly, and AI crawlers fail browser challenges by definition. We documented the e-commerce version of this failure in Cloudflare Bot Fight Mode and AI agents on Shopify.
3. Rate limiters tuned for humans. A crawler fetching 50 URLs in a minute trips thresholds a human never would. The bot gets 429s, retries later, and your freshest pages stay out of the index.
4. Geo or datacenter IP blocks. Rules that drop cloud-provider IP ranges drop crawlers too, since no crawler browses from a residential connection.

The diagnostic is the log read from the previous section: a recurring OAI-SearchBot user agent paired with 403/429/challenge responses is an accidental block, and the fix lives in your edge configuration rather than robots.txt.

FAQ

Does OAI-SearchBot respect robots.txt?

Yes. OpenAI documents robots.txt support for OAI-SearchBot, including the roughly 24-hour propagation delay after you change the file. Claims that it ignores robots.txt circulate on bot-protection vendor pages, but published large-scale evidence for OAI-SearchBot specifically is absent, and OpenAI's documentation says the opposite (checked June 11, 2026). If you see traffic that ignores your rules, verify the IP against searchbot.json first; impostor traffic wearing the user agent is the usual explanation.

Does blocking GPTBot remove my site from ChatGPT Search?

No. GPTBot and OAI-SearchBot are separate crawlers with separate robots.txt identities. Blocking GPTBot keeps your content out of future model training while leaving ChatGPT Search visibility untouched. The reverse is also true: blocking OAI-SearchBot removes you from search results without affecting training collection at all.

How frequently should I expect OAI-SearchBot visits?

There is no published crawl-frequency schedule for OAI-SearchBot. The cadence varies site by site, so the grep from the logs section above is the only number that describes yours; figures quoted in bot directories describe someone else's traffic.

Can I see which of my pages ChatGPT Search has indexed?

There is no public index-coverage console for ChatGPT Search comparable to Google Search Console. The available proxies: search for your brand and key topics inside ChatGPT Search and note which URLs it cites, and watch chatgpt.com referrals in your analytics to see which landing pages receive clicks.