Increasio Logotype

Do Big Retailers Block ChatGPT Search? A robots.txt Audit

We parsed robots.txt for 50 of the NRF Top 100 retailers. Only Amazon blocks ChatGPT search; GPTBot is blocked five times as often.

Only one of the 50 big US retailers whose robots.txt we could read blocks ChatGPT search in robots.txt, and that retailer is Amazon. We parsed the robots.txt files of 50 companies on the NRF Top 100 Retailers 2026 list on November 2, 2026. OAI-SearchBot, the crawler OpenAI uses to show sites in ChatGPT search, is blocked by 1 of them. GPTBot, the training crawler, is blocked by 5. Most retailers block nothing at all: 40 of the 50 files never mention an AI crawler by name.

For an ecommerce tech lead, the useful part is not the averages. It is how the blocks got there. Three grocery chains serve a byte-identical robots.txt that Cloudflare generates, and one discount grocer runs an allowlist that shuts out every Anthropic crawler by default. Both patterns are easy to inherit without anyone deciding anything about AI search.

Key takeaways

  • Blocks on AI search crawlers are rare: OAI-SearchBot 1 of 50, Claude-SearchBot 2, PerplexityBot 2. Googlebot: 0.
  • GPTBot is blocked five times as often as OAI-SearchBot (5 vs 1). ClaudeBot, Google-Extended and CCBot are also blocked by 5 of 50 each.
  • WinCo Foods, Stater Bros. and Weis Markets serve the same file, which disallows 32 named agents and ends in a "Cloudflare Managed Content" marker, with no Sitemap line.
  • Aldi's file disallows everything for unnamed bots, then names OAI-SearchBot as allowed and PerplexityBot as blocked. Claude-SearchBot is never named, so it falls into the block.
  • Blocking Google-Extended is not only a training opt-out. Google says the token also controls grounding in Gemini Apps.
  • 63 of the 100 sites did not return a robots.txt to a plain HTTPS request from our office. robots.txt is only one of the layers that decide what an AI crawler can reach.

Bar chart: full-site robots.txt blocks across 50 NRF Top 100 retailers. OAI-SearchBot 1, Claude-SearchBot 2, PerplexityBot 2, GPTBot 5, ClaudeBot 5, Google-Extended 5, CCBot 5, Applebot-Extended 4, Googlebot 0.

How did we run the audit?

We took the 100 companies on the NRF Top 100 Retailers 2026 list, mapped each one to its main US storefront domain, and requested /robots.txt over HTTPS on November 2, 2026. For holding companies we used the flagship banner (Ahold Delhaize USA as giantfood.com, TJX as tjmaxx.tjx.com, Tapestry as coach.com, Signet as kay.com).

37 files came back live. For the 63 that did not, we pulled the most recent Internet Archive capture and kept it only if it was taken after August 4, 2026 and contained a real robots.txt rather than a block page. That added 13 files, captured between September 29 and October 28, 2026. The other 50 retailers are excluded, including Costco, Home Depot, CVS and Lowe's. Of the top 10, we have Walmart, Amazon, Kroger, Walgreens, Target and Albertsons.

Each file was evaluated per crawler token with the matching rules from RFC 9309: a crawler obeys the group that names it, falls back to the * group only when no group names it, and the longest matching path rule wins. We count a crawler as blocked only when both / and a product-style path are disallowed. Partial blocks on carts, search pages or APIs are not counted.

Which retailers block AI search bots?

Two retailers block AI search crawlers outright, and only Amazon blocks OpenAI's. Ten of the 50 files name at least one AI crawler. Every one of them is below.

RankRetailerSourceWhat the file says about AI crawlers
2amazon.comliveNames and blocks OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot
14aldi.uslive* gets Disallow: /. OAI-SearchBot and ChatGPT-User allowed by name; PerplexityBot and Perplexity-User blocked by name. GPTBot and all Claude agents fall into the * block
25macys.comarchive, Oct 3Names ClaudeBot with only Crawl-delay: 5, which gives it no path rules at all
47wincofoods.comliveCloudflare-managed file: blocks GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and 27 more agents
54harborfreight.comliveExplicit Allow: / for OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot, ClaudeBot and CCBot
56petsmart.comliveOAI-SearchBot and PerplexityBot get their own groups that open inventory endpoints such as /api/pdp/inventories?*
63qvc.comliveOAI-SearchBot and PerplexityBot allowed by name
86staterbros.comliveSame Cloudflare-managed file as WinCo
88overstock.comliveGPTBot named, with one results-page rule
92weismarkets.comliveSame Cloudflare-managed file as WinCo

PetSmart is the most deliberate file in the set. It gives the two AI search crawlers access to store-inventory API paths, which suggests someone wants "is this in stock near me" answers to come from PetSmart's own data. Harbor Freight takes the opposite of Amazon's position and allows every major AI crawler by name, training included.

One rule catches teams out here. A group that names a crawler replaces the * group for that crawler; it does not add to it. Macy's ClaudeBot group contains only a crawl delay, so under RFC 9309 ClaudeBot is not bound by any of the Disallow lines Macy's wrote for *.

Why do three grocers have the same robots.txt?

WinCo Foods, Stater Bros. and Weis Markets all run behind Cloudflare and serve a robots.txt with the same MD5 hash. The file opens with Cloudflare's Content Signals text, sets Content-Signal: search=yes,ai-train=no,use=reference for *, disallows 32 named agents and ends with # END Cloudflare Managed Content. There is no site-specific rule and no Sitemap line after the marker.

That matches what Cloudflare's robots.txt setting documentation describes. With Bot Preference Sync on and the Training policy set to Disallow, Cloudflare prepends managed directives to the origin's robots.txt, and if the origin has no robots.txt, Cloudflare creates one. The docs also say that setting Training to Disallow this way enables edge blocking for AI training crawlers. If that is the setting these grocers used, their file is backed by a block at the CDN, not only a request.

The managed list does not touch OAI-SearchBot, Claude-SearchBot or PerplexityBot, so these grocers stay eligible for AI search. The catch is the sitemap: the served file has nothing after the managed block, so a crawler reading it finds no Sitemap line. Two more retailers, AT&T and Wegmans, publish Content-Signal: search=yes, ai-input=yes, ai-train=no but block no crawler by name.

Does blocking GPTBot keep a store out of ChatGPT?

No. OpenAI's crawler documentation treats each token as independent: GPTBot governs training, and OAI-SearchBot governs whether a site is shown in ChatGPT search answers. A site that blocks OAI-SearchBot can still appear as a navigational link. ChatGPT-User covers user-initiated visits and, per OpenAI, is not used to decide what appears in search.

Anthropic and Perplexity split their crawlers the same way. ClaudeBot collects content that may be used for training, and Claude-SearchBot is used to improve search result quality. PerplexityBot surfaces sites in Perplexity search and, according to Perplexity, is not used to train foundation models.

Google-Extended is the token that behaves differently from its name. Google's crawler documentation says it controls training of future Gemini models and also grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. It does not affect inclusion or ranking in Google Search. So the five retailers blocking Google-Extended, including the three on Cloudflare's managed file, are opting out of more than training. The same page notes that Google intends to support the AIPREF training-preference signal in early 2027.

Our earlier piece on how AI assistants choose which brands to recommend covers the full crawler list. The short version for a retailer: if you want product answers in ChatGPT, allow OAI-SearchBot even if you block GPTBot.

What happened with the other 63 sites?

They answered a plain HTTPS request for /robots.txt with something other than the file. 52 returned 403. The rest split into two 404s (one of them Costco), two 406s, a 418, a 429, a 462, two HTML pages served with a 200, and two timeouts. We did not try to get around those responses, and we draw no conclusion from them about how AI crawlers are treated, because these systems typically verify real crawlers by IP range and our request came from neither OpenAI nor Anthropic.

The point for a tech lead is narrower. A robots.txt that allows OAI-SearchBot does not prove OAI-SearchBot gets a 200. Bot management and WAF rules at the CDN sit in front of the file, and OpenAI explicitly asks sites to allow requests from its published IP ranges as well as the robots.txt token. If your security team owns the WAF and your SEO team owns robots.txt, nobody may own the answer to "can ChatGPT read our product pages". Rendering is a third layer: on Next.js storefronts we found that AI crawlers can miss page titles even when they are allowed in.

How do you check your own storefront?

Pipe your live robots.txt into this script with a real product path. It applies the same RFC 9309 rules we used for the audit and reports, per crawler, whether the path is allowed and whether the crawler is named or inherits the * group. It uses only the Python standard library (tested with Python 3.10.2).

# Usage: curl -s https://www.example.com/robots.txt | python3 check_ai_bots.py /products/some-item
import re, sys

BOTS = ["OAI-SearchBot", "ChatGPT-User", "GPTBot", "Claude-SearchBot",
        "Claude-User", "ClaudeBot", "PerplexityBot", "Google-Extended", "Googlebot"]

def groups(text):
    out, cur, in_ua = [], None, False
    for line in text.splitlines():
        line = line.split("#", 1)[0].strip()
        if ":" not in line:
            continue
        key, val = (s.strip() for s in line.split(":", 1))
        key = key.lower()
        if key == "user-agent":
            if not in_ua:
                cur = {"agents": [], "rules": []}
                out.append(cur)
            cur["agents"].append(val.lower())
            in_ua = True
        elif key in ("allow", "disallow") and cur:
            cur["rules"].append((key, val))
            in_ua = False
    return out

def allowed(rules, path):
    best = None  # RFC 9309: longest match wins, allow wins a tie
    for kind, pat in rules:
        if not pat:
            continue
        rx = "^" + ".*".join(map(re.escape, pat.rstrip("$").split("*")))
        if re.match(rx + ("$" if pat.endswith("$") else ""), path):
            if best is None or len(pat) > best[0] or (len(pat) == best[0] and kind == "allow"):
                best = (len(pat), kind)
    return best is None or best[1] == "allow"

path = sys.argv[1] if len(sys.argv) > 1 else "/"
gs = groups(sys.stdin.read())
for bot in BOTS:
    rules = [r for g in gs if bot.lower() in g["agents"] for r in g["rules"]]
    named = any(bot.lower() in g["agents"] for g in gs)
    if not named:
        rules = [r for g in gs if "*" in g["agents"] for r in g["rules"]]
    print(f"{bot:17} {'allowed' if allowed(rules, path) else 'BLOCKED':8} {'(named)' if named else '(via *)'}")

Run against aldi.us on November 2, it reports OAI-SearchBot as allowed (named) and Claude-SearchBot as BLOCKED (via *), which is exactly the allowlist gap described above. It checks the file only. To test the network layer, look at your CDN or WAF logs for verified OpenAI, Anthropic and Perplexity requests and their status codes.

What should an ecommerce team decide, and who should own it?

Decide search and training separately, write both decisions into robots.txt by name, and give one person ownership of the file and the bot rules in the CDN. In practice that means:

  1. Name OAI-SearchBot, Claude-SearchBot and PerplexityBot explicitly if you want AI search visibility. Relying on * works until someone adds a catch-all block, as Aldi's file shows.
  2. Decide on GPTBot, ClaudeBot, CCBot and Google-Extended as a content-licensing question, and remember Google-Extended also covers Gemini grounding.
  3. If a CDN writes managed directives, read the served file after every change and confirm your Sitemap line and your own rules survived.
  4. Check that verified AI search crawlers get 200 responses on product and category pages in your WAF logs, not only in robots.txt.

For the citation side of the same question, our AI Visibility Audit measures how ChatGPT, Perplexity, Gemini and AI Overviews describe and cite you across your priority prompts, with a competitor benchmark and a step-by-step action plan. It is a fixed fee and takes two weeks.

Method note: data collected November 2, 2026 by increas.io. Domain mapping, raw files and archive timestamps are kept on file. Robots.txt rules change often; Amazon and Aldi in particular should be re-checked before you quote them.

Do you want secure, fast website or application?

Get an estimate: