Increasio Logotype

Why AI Crawlers Miss Your Next.js Page Titles (and the Fix)

We tested Next.js 16.4.0: on dynamic pages GPTBot and ClaudeBot get an empty head. Byte offsets from our test and the htmlLimitedBots fix.

On dynamic Next.js App Router pages, GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot get an HTML response whose <head> has no <title> and no meta description. We reproduced it on Next.js 16.4.0 (the current release, October 6, 2026): Next.js streams the output of generateMetadata into the end of <body>, and its built-in list of bots that get metadata in the head covers Bingbot and Google's crawlers but none of the AI crawlers. The fix is one htmlLimitedBots entry in next.config.ts, shown below.

Key takeaways

  • Since Next.js 15.2 (February 2025), metadata from an async generateMetadata is streamed. Browsers and Googlebot, which render JavaScript, handle that. Crawlers that read raw HTML get the title and description near the end of the body, after the page content.
  • The bots that still get metadata in <head> are matched by one regex in Next.js. In 16.4.0 it has 27 entries and no OpenAI, Anthropic or Perplexity user agent.
  • Vercel's December 2024 crawler study found that GPTBot, ClaudeBot and PerplexityBot do not execute JavaScript. Anything that only exists after hydration does not exist for them.
  • Pages that prerender (static or ISR without dynamic data) are not affected. Product pages that read cookies, search params or uncached fetches usually are.
  • Content inside <Suspense> is a second issue. It arrives in a hidden div at the end of the document while the fallback text sits where the content should be.

Diagram of the HTML GPTBot receives from a dynamic Next.js 16.4.0 page: an empty head, the product content, and the title and meta description streamed at byte 5,421 of 6,496

What does an AI crawler actually receive from a Next.js page?

It receives the server HTML and nothing else. In December 2024 Vercel and MERJ published crawler data from Vercel's network: GPTBot made 569 million fetches in a month and Claude's crawler 370 million. Both downloaded JavaScript files (11.50% of ChatGPT requests, 23.84% of Claude requests) but neither ran them. Their summary was blunt: "none of the major AI crawlers currently render JavaScript." Google's Gemini (through Googlebot) and AppleBot were the exceptions. Read the Vercel study.

The study is from late 2024 and the vendors have not documented a change since. OpenAI's bot documentation, Anthropic's crawler help page and Perplexity's bot guide (all checked October 12, 2026) describe user agents and robots.txt handling but say nothing about rendering. Until a vendor says otherwise, assume raw HTML is all you get.

Why is the title missing from the head?

Because Next.js streams metadata by default, and AI crawlers are not on its exception list. The generateMetadata docs for 16.4.0 say that once it resolves, the metadata tags are appended to the body, and that Next.js verified this with bots that execute JavaScript and read the full DOM, such as Googlebot. For bots that cannot run JavaScript, Next.js waits for metadata and puts it in <head>. It finds those bots with a user-agent regex.

We read that regex in the published next@16.4.0 package (dist/shared/lib/router/utils/html-bots.js). It matches Google's crawlers, Bingbot, applebot, DuckDuckBot, Yandex, Baidu and social preview bots like facebookexternalhit, Twitterbot, LinkedInBot and Slackbot. GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User are absent. The comment above it in the source explains why the list exists: some bots look for metadata only inside <head>, so it cannot be streamed to them.

How did we test it?

We built a test app on next@16.4.0 with react@19.3.0, ran next build && next start, and fetched each route with curl using each vendor's documented user-agent string. No JavaScript ran, which is what these crawlers do. The test routes:

  • /ssr: a server component product page with force-dynamic and a generateMetadata that awaits 300 ms (a stand-in for a product API call).
  • /csr: a client component that fetches the same product in useEffect.
  • /stream: a server-rendered product with a reviews block in <Suspense> that takes 1.5 s.
Route and user agentResponse size<title> and meta descriptionProduct name and price
/ssr, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot6,496 BIn <body>, from byte 5,421 (head ends at byte 590)In body, near the top
/ssr, Bingbot5,296 BIn <head>In body, near the top
/ssr, GPTBot after the fix below5,296 BIn <head>In body, near the top
/csr, every user agent5,337 BAbsentAbsent. The page reads "Loading…"
/stream, every user agent6,314 to 6,386 BNot set on this routePrice in place. Reviews in <div hidden id="S:0"> at byte 5,286 or later, "Loading reviews…" in their place

Bingbot and GPTBot request the same URL and get different documents. Bing gets the head it expects. OpenAI's crawler gets an empty head, then the body, then the title in the last 1,100 bytes. Whether a given AI pipeline still picks up a <title> from the body is not documented by any vendor. We would not bet a product page's citation on it.

How do you fix it in next.config.ts?

Add the AI user agents to htmlLimitedBots. Setting the option replaces the default list, so copy the defaults in first or you lose head metadata for Bingbot and Google's other crawlers. This compiles under tsc --noEmit against Next.js 16.4.0, and it moved the title back into <head> for GPTBot and ClaudeBot in our test:

import type { NextConfig } from 'next'

// Setting htmlLimitedBots replaces the Next.js default list, so the
// defaults (from html-bots.ts in next@16.4.0) are copied in first.
const NEXT_DEFAULT_BOTS =
  /[\w-]+-Google|Google-[\w-]+|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti|googleweblight/

const AI_BOTS =
  /GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-User|Claude-SearchBot|PerplexityBot|Perplexity-User/

const config: NextConfig = {
  htmlLimitedBots: new RegExp(`${NEXT_DEFAULT_BOTS.source}|${AI_BOTS.source}`, 'i'),
}

export default config

Re-check the default list whenever you upgrade Next.js, because your copy will not pick up new entries. The cost is small: AI crawlers wait for generateMetadata before the first byte, which was 300 ms in our test. Browsers keep streaming. The blunt alternative from the htmlLimitedBots docs is htmlLimitedBots: /.*/, which turns streaming metadata off for everyone and gives up the faster first paint for real shoppers.

Which storefront pages are affected?

Only routes whose metadata cannot be prerendered. The docs say that if a page can be prerendered and generateMetadata does nothing dynamic, the metadata lands in the initial HTML. Static and ISR product pages are fine. The risk sits in routes that opt into dynamic rendering, which in commerce is common:

  • Product pages that read cookies for currency, market or a logged-in price list.
  • Search and collection pages driven by searchParams.
  • Any generateMetadata that makes an uncached fetch to a commerce API or PIM.

Pages Router sites are a different path. Our own blog runs on it, and the title sits at byte 112 of the head for both GPTBot and Bingbot. Shopify's Hydrogen demo store (React Router) also served its title in the head to both user agents when we checked today. The streaming behavior is specific to the App Router.

What else on a product page disappears for AI crawlers?

Two more patterns, both visible in the table above.

Client-fetched content. If price, availability or the description load in useEffect, an AI crawler gets the loading state. Our /csr route sent 5,337 bytes and none of them were the product. Typical offenders on headless storefronts are review widgets, inventory badges and "complete the look" blocks. Move what you want quoted into a server component.

Content behind Suspense. React streams it into a hidden div at the end of the document and swaps it in with a script. The text is in the bytes, but out of place, inside an element marked hidden, with the fallback text sitting in its slot. A crawler that drops hidden elements or reads in document order sees "Loading reviews…". Keep the rating summary and the facts you want cited outside the boundary, and suspend only the parts that are slow and not worth quoting.

How do you check your own site in five minutes?

Fetch a product URL with an AI user agent and see where the meta description lands (we grep for it rather than <title>, because inline SVG icons carry their own <title> tags):

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" \
  https://your-store.com/products/some-handle | grep -o -b -E '</head>|<meta name="description"'

If the description's byte offset is larger than the </head> offset, your metadata is streaming past AI crawlers. Run the same command with a Bingbot user agent to see the difference. Then search the response for the price and for one line of your review summary. Our free Next.js site and AI-visibility auditor reads only the server HTML and runs no JavaScript, which is the same view an AI crawler gets. For the bigger picture of which crawler feeds which assistant, see how AI assistants decide which brands to recommend.

What does fixing it cost?

The config change is the file above, plus a check after each Next.js upgrade. Moving client-fetched product data into server components depends on how much of the page renders in the browser, so we scope it per project. If you want this measured across your priority prompts, our AI Visibility Audit is a fixed fee and takes 2 weeks, and our Next.js team does the remediation. Test date and versions for everything above: October 12, 2026, Next.js 16.4.0, React 19.3.0.

Do you want secure, fast website or application?

Get an estimate: