Skip to content
bucker

Web agents

Explore, Web agents shows what AI agents and crawlers do on your site: who they are, what they read, what they looked for and did not find, and where they waste effort. It is read from request logs, because agents run no JavaScript and so never run a script on your page.

Where the numbers come from

Request lines reach a project from a Vercel or Cloudflare log drain or from the http-log door. Each line is judged by the traffic classifier and stored with traffic.type, traffic.operator and, for a non-human client, a one-day pseudonymous agent_session_key. If no line reaches the project, the tab says so and names those two ways in.

The tab counts a line as an agent's when the classifier labelled it ai_agent (acts for a person) or ai_crawler (collects for training or an index). Search crawlers and monitors are a different question and are not here.

What it reads, and its limits

It reads the project's Logs within the Explore bounds: the last 30 days, and the newest 100,000 lines of each of two scans (agent lines, and AI-referral lines). When a scan reaches its limit the tab says so in a warning above the numbers: every count is then a lower bound and the oldest days may be missing. The sentence under "What this is computed from" says exactly which lines were read, how many, and whether the project samples its logs (counts are then of the lines kept). The same fields, scanTruncated and basis, are in the API answer.

The sections

Section What it answers
Operators Per operator: requests, distinct pages, llms.txt fetches, Markdown fetches, errors, last seen. A client the classifier could not name is "Unnamed operator".
Content gaps Pages agents asked for and got a 404. Each is content they wanted that is missing or moved. Malformed paths are left out (below).
Wasted fetches One page fetched in more than one format (HTML, Markdown, text, JSON) within 30 minutes by the same operator. Serving the format the agent prefers once saves the rest. A 404 in one format is a gap, not waste.
Malformed paths URLs an agent mangled: broken percent-encoding, spaces, markup or template characters (<>{}, [slug]), doubled slashes, dot segments, trailing punctuation. They are the agent's mistake, not missing pages.
AI referrals People who followed a link from an AI assistant (ChatGPT, Claude, Perplexity, Gemini, Copilot and a few others) to your site, by the referring host, with the pages they landed on.
Journeys One client's requests in order. A journey ends after 30 minutes without a request and shows up to 200 steps. Only clients with two or more requests are listed. The key is per UTC day, so a journey cannot span midnight UTC.

A path an agent chose is shown as text, never as a link.

What is stored about a client

No address and no full referring URL. The address is used to check the operator's published ranges, to look up a country (when a database is configured) and to compute the session key, then dropped. The key is a keyed hash that changes every day and cannot be reversed to an address.

API

GET /orgs/{org}/projects/{project}/web/agents needs the analytics:read scope on the project and returns the same data, including scanTruncated, basis, scanLimit and rowsScanned.