# Web agents

**Explore, Web agents** shows what AI agents and crawlers do on your site: who they are, what
they read, what they looked for and did not find, and where they waste effort. It is read from
**request logs**, because agents run no JavaScript and so never run a script on your page.

## Where the numbers come from

Request lines reach a project from a [Vercel or Cloudflare log drain](log-drains.md) or from
the [http-log door](log-drains.md#request-lines-and-the-http-log-door). Each line is judged by
the traffic classifier and stored with `traffic.type`, `traffic.operator` and, for a non-human
client, a one-day pseudonymous `agent_session_key`. If no line reaches the project, the tab says
so and names those two ways in.

The tab counts a line as an agent's when the classifier labelled it **`ai_agent`** (acts for a
person) or **`ai_crawler`** (collects for training or an index). Search crawlers and monitors are
a different question and are not here.

## What it reads, and its limits

It reads the project's **Logs** within the Explore bounds: the **last 30 days**, and the
**newest 100,000 lines** of each of two scans (agent lines, and AI-referral lines). When a scan
reaches its limit the tab says so in a warning above the numbers: **every count is then a lower
bound** and the oldest days may be missing. The sentence under "What this is computed from" says
exactly which lines were read, how many, and whether the project samples its logs (counts are
then of the lines kept). The same fields, `scanTruncated` and `basis`, are in the API answer.

## The sections

| Section | What it answers |
| --- | --- |
| **Operators** | Per operator: requests, distinct pages, `llms.txt` fetches, Markdown fetches, errors, last seen. A client the classifier could not name is "Unnamed operator". |
| **Content gaps** | Pages agents asked for and got a **404**. Each is content they wanted that is missing or moved. Malformed paths are left out (below). |
| **Wasted fetches** | One page fetched in **more than one format** (HTML, Markdown, text, JSON) within **30 minutes** by the same operator. Serving the format the agent prefers once saves the rest. A 404 in one format is a gap, not waste. |
| **Malformed paths** | URLs an agent mangled: broken percent-encoding, spaces, markup or template characters (`<>{}`, `[slug]`), doubled slashes, dot segments, trailing punctuation. They are the agent's mistake, not missing pages. |
| **AI referrals** | People who followed a link from an AI assistant (ChatGPT, Claude, Perplexity, Gemini, Copilot and a few others) to your site, by the referring host, with the pages they landed on. |
| **Journeys** | One client's requests in order. A journey ends after **30 minutes** without a request and shows up to **200 steps**. Only clients with two or more requests are listed. The key is per UTC day, so a journey cannot span midnight UTC. |

A path an agent chose is shown as text, never as a link.

## What is stored about a client

No address and no full referring URL. The address is used to check the operator's published
ranges, to look up a country (when a database is configured) and to compute the session key, then
dropped. The key is a keyed hash that changes every day and cannot be reversed to an address.

## API

`GET /orgs/{org}/projects/{project}/web/agents` needs the `analytics:read` scope on the project
and returns the same data, including `scanTruncated`, `basis`, `scanLimit` and `rowsScanned`.
