A bot that indexes your catalog to answer a question has nothing in common with a bot that scrapes your content to train a model, or an agent acting in real time on behalf of a shopper. In 2026, the right question is no longer "should I block AI?" but "which bot, for what purpose?".
AI agent traffic has changed scale
Automated reading of the web is no longer marginal. According to DataDome's quarterly report (data from April to June 2026, a network of 400+ enterprises), AI agent requests jumped 45% in a single quarter, from 12.2 billion in Q1 to 17.7 billion in Q2 2026.
The report highlights a crucial point for merchants: crawl volume and referral value are moving in opposite directions. ChatGPT-User (the user-triggered agent) saw its request volume drop 6% over the quarter, even as ChatGPT remains, by far, the leading source of AI-driven referral traffic, commanding 80–88% of all AI referrals every month. In other words: the bot that crawls the most is not the one that sends the most visitors. Hence the need to reason by purpose, not by volume.
The purpose-based matrix: Search, Agent, Training
Cloudflare, which protects a large share of the web, formalized in July 2026 a three-purpose taxonomy rather than a binary "AI / not AI" label. The question is no longer "is this an AI bot?" but "what does it do with my content?":
- Search — the bot collects or indexes your content so it can answer questions about it later. This is what makes you citable and recommendable.
- Agent — automated behavior acting, usually in real time, on a person's behalf, to get something done right now (compare, add to cart, buy).
- Training — a crawler taking your content to train or fine-tune a model.
This distinction has a concrete consequence: from September 15, 2026, on pages that display ads, Cloudflare will block the Training and Agent categories by default for new domains, while leaving Search allowed. Owners can change these preferences before that date. A merchant must therefore know exactly what they allow.
The bots to know (and sort) for e-commerce
OpenAI separated its agents by purpose, each controllable independently in robots.txt. The principle is explicit in their documentation: you can allow search while disallowing training.
| Bot | Purpose | Stake for a merchant |
|---|---|---|
OAI-SearchBot | Search (ChatGPT Search) | Allow it: otherwise absent from ChatGPT search answers |
GPTBot | Training | Strategic choice: future visibility vs content control |
OAI-AdsBot | Ad landing page validation | Only visits pages submitted as ads |
ChatGPT-User | User-triggered action | Real-time agent: tied to assisted buying journeys |
On Google's side, the equivalent is Google-Extended (Gemini training), distinct from Googlebot; on Perplexity's side, PerplexityBot. Each can be allowed or disallowed separately. OpenAI notes that after a robots.txt update, it takes about 24 hours for its search systems to adjust.
The trap to avoid
Blocking "all AI bots" in one sweep — via a generic rule or a miscalibrated Cloudflare setting — can cut off search crawlers. The result: your store becomes invisible in ChatGPT, Perplexity and Gemini, the very engines that send buyers. Protecting against training should not cost you a disappearance from answers.
The video below (Ahrefs channel) walks through the technical checks — robots.txt, GPTBot, llms.txt, and Cloudflare's default setting — that can quietly stop an AI from accessing your site:
Source: Ahrefs — "Technical SEO for AI: Robots.txt, GPTBot & llms.txt Explained" (May 2026).
A commerce-oriented robots.txt
For a store that wants to be discovered by AI while keeping control over training, a reasonable baseline looks like this (adapt it to your policy):
# AI search — allow to stay citable
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
# Training — strategic choice (here: disallowed)
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Never block product/catalog pages from search
Sitemap: https://your-store.com/sitemap.xmlCaution: robots.txt expresses a declared policy. Observed reality may differ (WAF rules, firewall, CDN, bots that ignore the standard). That's why you must verify what actually happens, not just what the file claims.
Auditing your store's access for AI crawlers
Knowing which bots you allow — and verifying that policy is consistent across every surface — is the first, most blocking factor of a GEO audit. If search crawlers are blocked, no other optimization matters. In the Shopify ecosystem, VerityScore's GEO audit checks exactly this crawler access: reading robots.txt, detecting rules that block GPTBot, OAI-SearchBot or Google-Extended, and the consistency of discovery surfaces (sitemap, agent-card.json, UCP/ACP manifests).
You can test a Shopify store's AI accessibility for free (preview, no email) to spot, before anything else, whether a setting silently blocks your visibility. That's check number one: making the store readable by the right bots.
Crawler control and UCP: readability first
This access hygiene is the foundation of the Universal Commerce Protocol. An agent cannot understand, trust or transact with a store it is not allowed to read. Allowing search bots, exposing a readable catalog (Catalog MCP, schema.org) and publishing the right discovery surfaces: this is the entry condition for existing in agentic commerce. Purpose-based control is not a brake on AI — it is how you take part without being scraped away.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT?
Not from search. GPTBot concerns training; it's OAI-SearchBot that governs appearance in ChatGPT's search results. You can disallow GPTBot and stay visible via OAI-SearchBot.
Is robots.txt enough to control AI agents?
No. robots.txt is a declared policy that well-behaved bots follow, but network-level controls (Cloudflare, WAF) and cryptographic agent verification (Web Bot Auth) increasingly take over for real-time actions.
Should a small merchant care?
Yes, especially them. For a small store, the risk is less training than invisibility: an overly restrictive default can prevent it from appearing in the AI answers that steer purchases.