Web agents
Explore, Web agents shows what AI agents and crawlers do on your site: who they are, what they read, what they looked for and did not find, and where they waste effort. It is read from request logs, because agents run no JavaScript and so never run a script on your page.
Where the numbers come from
Request lines reach a project from a Vercel or Cloudflare log drain or from
the http-log door. Each line is judged by
the traffic classifier and stored with traffic.type, traffic.operator and, for a non-human
client, a one-day pseudonymous agent_session_key. If no line reaches the project, the tab says
so and names those two ways in.
The tab counts a line as an agent's when the classifier labelled it ai_agent (acts for a
person) or ai_crawler (collects for training or an index). Search crawlers and monitors are
a different question and are not here.
What it reads, and its limits
It reads the project's Logs within the Explore bounds: the last 30 days, and the
newest 100,000 lines of each of two scans (agent lines, and AI-referral lines). When a scan
reaches its limit the tab says so in a warning above the numbers: every count is then a lower
bound and the oldest days may be missing. The sentence under "What this is computed from" says
exactly which lines were read, how many, and whether the project samples its logs (counts are
then of the lines kept). The same fields, scanTruncated and basis, are in the API answer.
The sections
| Section | What it answers |
|---|---|
| Operators | Per operator: requests, distinct pages, llms.txt fetches, Markdown fetches, errors, last seen. A client the classifier could not name is "Unnamed operator". |
| Content gaps | Pages agents asked for and got a 404. Each is content they wanted that is missing or moved. Malformed paths are left out (below). |
| Wasted fetches | One page fetched in more than one format (HTML, Markdown, text, JSON) within 30 minutes by the same operator. Serving the format the agent prefers once saves the rest. A 404 in one format is a gap, not waste. |
| Malformed paths | URLs an agent mangled: broken percent-encoding, spaces, markup or template characters (<>{}, [slug]), doubled slashes, dot segments, trailing punctuation. They are the agent's mistake, not missing pages. |
| AI referrals | People who followed a link from an AI assistant (ChatGPT, Claude, Perplexity, Gemini, Copilot and a few others) to your site, by the referring host, with the pages they landed on. |
| Journeys | One client's requests in order. A journey ends after 30 minutes without a request and shows up to 200 steps. Only clients with two or more requests are listed. The key is per UTC day, so a journey cannot span midnight UTC. |
A path an agent chose is shown as text, never as a link.
What is stored about a client
No address and no full referring URL. The address is used to check the operator's published ranges, to look up a country (when a database is configured) and to compute the session key, then dropped. The key is a keyed hash that changes every day and cannot be reversed to an address.
API
GET /orgs/{org}/projects/{project}/web/agents needs the analytics:read scope on the project
and returns the same data, including scanTruncated, basis, scanLimit and rowsScanned.