Maya/Help Center

LLM Site Visibility — AI crawler analytics

See how AI crawlers discover and read your site — requests, pages, errors — and connect Cloudflare, Vercel, or log streaming.

Updated 2026-10-12

Before an AI assistant can mention or cite you, its crawlers have to fetch your pages. LLM Site Visibility is the analytics layer for that invisible traffic: which AI bots visit, what they read, where they fail. It requires a connected traffic source (Pro and above).

The dashboard

  • KPI cards: Total Requests, Pages Crawled, Error Rate, Active Bots — each with a trend chip.
  • Tabs: Overview, Report, AI Crawlers, Pages, Errors & Blocks, Config.
    • AI Crawlers — per-bot breakdown (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Bingbot and friends, grouped by company).
    • Pages — which URLs the bots actually read. Compare this against the pages you want read; crawl budget going to tag pages instead of product pages is a finding.
    • Errors & Blocks — 4xx/5xx responses and blocked requests served to AI bots. A 403 to GPTBot is self-inflicted invisibility; this tab is where bot-protection misconfigurations surface.
  • Sync Now forces a refresh; "No AI bot traffic found" appears until a source delivers data.

Connecting a traffic source (Config tab)

Any one of these feeds the same pipeline:

  • Cloudflare — paste an API Token and Zone ID, then Test Connection.
  • Vercel — connect a Log Drain from your Vercel project.
  • Log streaming — credentials for logs.withmaya.ai: point a log shipper (Vector, Fluent Bit) or a CDN push at it. AWS CloudFront standard logs (v2) delivered via Firehose use this path.

Setup guides per stack (Nginx, Apache, IIS, custom BFF) live in the developer docs.

  • Crawler Logs (Monitoring) — the raw request log, grouped by bot and company, when you need to inspect individual hits.
  • Agent Behavior — journey-level view: how AI agents move through the site and whether that reading converts into AI-referred visits (the Agentic Rate metric).
  • Visitor Analytics — the human side: GA4-measured traffic arriving from AI assistants.

What good looks like

  1. AI bots you care about appear in Active Bots, with near-zero Errors & Blocks.
  2. The Pages tab's top entries are your money pages, not pagination or parameter noise.
  3. Crawl activity on a page precedes citations of it — if a page you're pushing never gets crawled, fix discoverability before rewriting content.

Related articles