SuprAnalytics

AI and search crawlers

See which AI and search crawlers read your pages.

AI assistants and search engines fetch your pages without running JavaScript, so no tracking script sees them. Forward their requests from your server and the crawler card shows who came, how often and what they read.

Start free trial How it works

14-day free trial · No card required · From $9/month

The crawler card on a demo dashboard's AI answers tab: thirty days of requests from ChatGPT, Claude and Perplexity, a count for each, and tabs for indexing and training crawlers. Made-up data.

Three reasons a bot comes by.

Each request is sorted by its user agent.

AI answers
Fetched live because someone asked an assistant: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User and others.
Indexing
Search engines, including AI search: Googlebot, bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot and others.
Training
Collecting text to train models: GPTBot, ClaudeBot, Google-Extended, CCBot, Bytespider, Amazonbot and others.

What the crawler card shows.

On the dashboard, for the period you're looking at:

  • A total for each of the three groups, on its own tab
  • Each crawler's requests over the period, one line each
  • The pages each crawler requested, up to 12 per crawler
  • robots.txt, llms.txt and sitemaps count as reads; requests for scripts, styles, images and fonts are left out

Kept apart from your visitors.

A crawler request isn't a visit. Bots that do reach the tracking script are dropped, so crawler hits never show up as visitors, pageviews or sources. For people who click through from an assistant's answer, see AI traffic.

Not events
Crawler hits don't count towards your plan's monthly events.
Kept like analytics
They're kept for your plan's data history, like the rest of your analytics.
Yours to hide
Not interested? Hide the crawler card in Settings → General.

Set it up.

  1. Get an API key

    The website's API key is in Settings → API. Keep it on your server.

  2. Forward crawler requests

    In your server or middleware, send each bot request's user agent and path to POST /api/v1/crawl. The card's setup has snippets for Next.js, Express and curl.

  3. Watch the card

    Hits show on the dashboard's crawler card under AI answers, Indexing and Training.

How it works, in detail.

The docs page “Track AI and search crawlers” goes through it step by step.

How it works

Limits worth knowing

  • It needs a few lines on your server or in your middleware; without them the card stays empty
  • Crawlers are recognised by their user agent only: a bot that isn't on the list, or that claims to be a browser, isn't counted
  • A crawler fetching a page doesn't tell you whether the page then appeared in an answer
  • Up to 100 hits per request to the API

Questions, answered.

Why can't the tracking script see crawlers?

Crawlers fetch the page's HTML without running its JavaScript, so the script never loads. Your server sees every request, so it reports them instead.

Which crawlers are recognised?

For AI answers: ChatGPT, Claude, Perplexity, Gemini, Mistral, Meta AI and others. For indexing: Google, Bing, Apple, DuckDuckGo, OpenAI, Anthropic, Perplexity and others. For training: OpenAI, Anthropic, Google, Common Crawl, ByteDance, Amazon and others.

Do crawler hits count as visitors or events?

No. They're shown only on the crawler card, never as visitors, and don't count towards your plan's events.

Can I send every request, not just bots?

Yes. Unknown user agents and requests for static files are ignored, so you can forward everything; checking for bot-like user agents first just saves requests.

How is this different from AI traffic?

AI traffic is people who click a link in an assistant's answer and visit your website. Crawler hits are the bots themselves fetching your pages.

Is there an API reference?

Yes: the crawler hits API, with the fields, an example and the response. It takes one hit or up to 100 at a time.

More features

All features

See it on your own website.

Every feature is on every plan, with a free trial and no card.

Start free trial

14-day free trial · No card requiredPricingHow it worksAll features