Building Websites AI Crawlers Can Actually Read

The short answer: make the content exist in the HTML response. Most AI crawlers don't execute JavaScript, so anything rendered client-side — a React app that fetches its content after mount, a tab panel populated on click, an accordion whose text arrives with the bundle — is effectively not there. Server-render (or statically generate) everything you want retrieved, describe entities with connected JSON-LD using @id references, keep important pages within about three clicks of the homepage, write in self-contained paragraphs that survive being quoted out of context, and publish llms.txt plus a feed as cheap discovery infrastructure. This overlaps almost entirely with good technical SEO — the difference is that AI crawlers are less forgiving.

There's a strange asymmetry in how sites get built right now. Googlebot has rendered JavaScript for years, which quietly made client-side rendering safe enough that a generation of frameworks defaulted to it. Then a new class of crawler arrived — the ones feeding ChatGPT, Perplexity, Claude, Gemini, and the AI answer layers on top of search — and most of them fetch your HTML, parse it, and leave. No headless browser. No waiting for hydration. No second pass.

Which means a site that scores fine in Search Console can be substantially invisible to the systems that increasingly sit above search. Not penalised — just absent from the input. This post is the architecture side of that problem: what to build so retrieval works. (The local-business side — profiles, reviews, citations — is over here.)

1. Server-render anything you want retrieved

This is the whole ballgame, and everything else is a refinement.

The test is easy and I'd run it on any site you own right now: fetch the page without JavaScript and read what comes back. curl -s https://yoursite.com/page | less, or disable JS in devtools and reload. Whatever isn't in that response is what AI crawlers don't see.

The usual offenders, in rough order of how often I find them:

• A single-page app that fetches content client-side. The HTML is a <div id="root"> and a script tag. The crawler gets an empty page.
• Content behind interaction. Tabs, accordions, "read more" toggles, modals. If the text only enters the DOM on click, and it isn't in the initial HTML, it isn't retrievable — even though a human can obviously reach it.
• Infinite-scroll or "load more" listings. Only the first batch exists. Everything below is unreachable without pagination that's actually in the markup.
• Text baked into images. Prices on a graphic, hours on a JPEG of the front door, a menu as a PDF screenshot. Unreadable, and this one is remarkably common on restaurant and trades sites.

The fix is architectural rather than a plugin: render on the server. With Next.js App Router that's the default — React Server Components render on the server and stream HTML, so content is present in the response unless you actively opt out with a client component that fetches after mount. Static generation is even better where the content permits it. If you're on WordPress, you're already server-rendered; your risk is page builders that lazy-inject sections client-side.

The mental rule I use: 'use client' is for interactivity, not for content. A component whose job is to display text should render on the server. A component whose job is to respond to clicks can be a client component — but the text it reveals should already be in the HTML. That distinction is most of the work, and it's the same instinct that keeps sites fast, which I wrote about in architecture for speed.

2. Describe your entities, and connect them

Structured data isn't new. What's changed is what it's used for: AI systems use JSON-LD to resolve what a thing is rather than to draw a rich snippet. That makes the baseline set worth getting right — Organization or LocalBusiness, Person, Article or BlogPosting, FAQPage, Service, BreadcrumbList.

The upgrade most sites miss is connecting them. Rather than each page emitting an isolated island of schema, give your core entities stable @id values and reference those ids from everywhere else:

{
  "@context": "https://schema.org",
  "@type": "BlogPosting",
  "headline": "Building Websites AI Crawlers Can Actually Read",
  "author":    { "@id": "https://jagatjeet.com/#person" },
  "publisher": { "@id": "https://jagatjeet.com/#business" }
}

Now every article, service page, and FAQ points at one canonical person and one canonical business, and the whole site reads as a single connected entity graph instead of six hundred unrelated documents. That's exactly the kind of corroboration that decides whether a model treats your business as a known thing or an unverified string.

Two practical notes from maintaining this on a large site. Centralise the builders — every piece of schema here comes out of one module, so the entity ids can't drift. And don't mint duplicate entities: one LocalBusiness for the site, referenced everywhere, not a fresh one per page. Duplicate entities are worse than missing ones, because they make the graph ambiguous.

3. Structure the site so depth is legible

Retrieval systems build a picture of what a site is about, and that picture comes from structure as much as prose.

Keep important pages within about three clicks of the homepage. Depth costs you: pages buried six levels down get crawled rarely and read as peripheral.

Use pillar-and-cluster. A hub page covering a topic broadly, linking down to specific pages and supporting articles, each linking back up and sideways to siblings. This isn't an SEO trick — it's a legible map of your expertise, and it's how a model works out that you cover a subject in depth rather than mentioning it once.

Give every page one canonical URL, absolute and self-referencing. Duplicates across www/non-www, trailing slashes, and query-parameter variants split whatever authority you have and produce contradictory copies of the same content in the retrieval index.

Keep URLs stable. Redirect chains and rewritten URL structures are how a site loses citations it already earned. If you must move something, 301 it and leave the redirect in place permanently.

4. Write paragraphs that survive being quoted

This is a content point with an architectural consequence, so it belongs here.

AI answers are assembled from extracted passages. A passage that depends on the three paragraphs above it for meaning can't be lifted cleanly. A self-contained one can:

A typical five-page small-business website in Kamloops runs $2,500–$5,000 and takes three to five weeks.

That sentence carries its own subject, scope, place, and numbers. It can be quoted, attributed, and reused without distortion. "It depends on your needs, but we're competitively priced" cannot.

Practically, that means: real headings in a sensible hierarchy (h1 once, then h2/h3 — not styled divs), an answer near the top of the page rather than after eight hundred words of preamble, plain conversational sentences over marketing abstraction, and specifics — numbers, places, timeframes — wherever you can honestly give them. Semantic HTML matters here too: <table> for tabular data, real lists for lists, <article>/<main> to mark where the content is. Crawlers use those boundaries to decide what's content and what's furniture.

5. Publish the cheap discovery infrastructure

Three files, each an afternoon at most, none of them magic:

llms.txt — a Markdown summary at your site root describing what the site is and pointing to its important resources. Through 2026 it's moved from experimental curiosity to a reasonable AEO signal, with Anthropic, Stripe, Vercel, Cloudflare and plenty of others publishing one. Honest framing: no major AI provider has committed to it as a ranking input, and it will not rescue a site whose content isn't server-rendered. It's cheap, it's a clean machine-readable statement of what you do, and being early on a convention costs nothing. Mine is at /llms.txt if you want the shape of it.

An RSS or Atom feed. Old technology, newly useful — feeds are still one of the most reliable ways for aggregators and crawlers to discover new content quickly. Link it from your <head> with rel="alternate".

A real robots.txt decision. AI crawler user-agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and friends) can be allowed or blocked. Either is defensible — a publisher selling subscriptions has a real argument for blocking. But make it a decision: for a local business trying to be recommended by assistants, blocking them is self-defeating, and I've seen it happen by accident via a copied config or an aggressive security plugin. Check yours.

6. Don't forget the basics that still decide it

Speed and mobile still matter, for the mundane reason that crawlers operate on timeouts and budgets. A slow server-rendered page can time out before it's read. And Bing deserves a specific mention: ChatGPT's browsing runs on Bing's index, so being absent from Bing Webmaster Tools is a self-inflicted blind spot. Verify there, submit your sitemap, and check for crawl errors — it takes ten minutes and almost nobody does it.

What this looks like when it's done

To make it concrete, this site is the worked example. Content renders on the server through React Server Components, so every page's text is in the HTML response. JSON-LD comes from one centralised module with stable entity ids for the business and the person, referenced from every page. Hubs link down to service pages and articles, which link back up and across. Every page carries an absolute self-canonical and sits in a hand-maintained sitemap with real modification dates. llms.txt and feed.xml are generated from live content. It's not exotic — it's just the boring version done consistently.

If you're weighing up whether your current stack can get there, when to rebuild vs refactor is the decision framework I'd use, and Next.js App Router patterns covers the server/client boundary in more depth. For the strategy layer above the code — which pages should exist and what each one is for — that's web architecture consulting, or digital architecture for Kamloops businesses if you're a local business wanting the whole system connected rather than a diagram.

Either way, start with the curl test. It takes ten seconds and tells you immediately whether you have a content problem or an architecture problem.