# Logs: keys on every record, and how agents and people read them

A log record carries the keys it is joined and filtered on, and every door that reads logs
speaks the same search grammar. This page is the reference for what a stored record holds, how
`where` and `fields` work, what a search says about itself, and the three MCP tools that read
logs. The search box and its grammar are described under
[Searching logs](explore.md#searching-logs); how records arrive is in [OTLP ingest](otlp.md) and
[Log drains](log-drains.md).

## What a record carries

| Field | What it is |
| --- | --- |
| `service` | The emitting service: OTLP `service.name`, or the drain's service (one per Vercel project, Worker, Heroku app or Supabase service). Null when the sender named none, and on every record written before the column existed. |
| `resource` | At most 16 allow-listed resource facts, strings only, each cut to 200 characters: `service.version`, `host.name`, `k8s.pod.name`, `k8s.namespace.name`, `cloud.region`, `faas.name`, `vercel.deployment.id`, `vercel.request.id`. A Vercel drain carries the last two on each line, so a record's own value wins over the batch's. Anything else a sender puts in its resource is not stored on the row. |
| `entityId` | The infrastructure entity the record's resource resolved to, when entity resolution has run. |
| `subjectKey` | Reserved for the keyed pseudonym of the person a record is about. **Empty until F-04 ships**: nothing writes it today, so it is not offered as a join key (`session_id` is in the same position). Erasure already matches it, so rows written after F-04 are reached without a change. |
| `traceId`, `spanId`, `issueId`, `release`, `environment`, `sampleRate` | As before. |

`service` and `resource` go through the same stored-tier scrub as the body and the attributes
before they are written: a credential in a host name is redacted, and what is not sensitive is
kept. A drain still drops the client address, the referer and the query string; nothing here
changes that.

In the Explore logs dataset the keys are `service`, `entity_id` and the path `resource.<key>`:
`service:checkout`, `resource.host.name:ip-10-0-0-7`. They work in the log search, in Explore
aggregates, in saved queries and in the MCP tools alike.

## Searching: `where`, `fields` and the evidence

`GET /orgs/:org/projects/:project/logs` takes the grammar in `query` (the name the log view has
always used) and, under the name the MCP tool uses, in `where`; if both are sent they are ANDed.
It also takes `fields`, up to ten attribute keys (or `resource.<key>`) separated by commas, and
returns each row's present keys as `projected`, so a reader gets `http.route` without the whole
attribute bag. More than ten fields is a 400.

Every page now says what it rests on, as `evidence`:

| Field | Meaning |
| --- | --- |
| `compiled` | The parse of `where`: `filters`, `text` and, for an `OR` or a negated group, `expression`. Null when no `where` was sent. |
| `window` | `from`, `to` and `defaulted`. A search the caller bounded by neither time, ids, a trace, a span nor an issue is bounded to the last 24 hours by the search itself (`defaulted: true`); the REST list keeps its older behaviour of reading newest-first from the whole table when it is sent no window. |
| `scanned`, `truncated`, `scannedThrough` | With a scan cap (the MCP tool sets one): how many matching rows were counted, up to the newest 100,000, whether more matched, and how far back the scan reached (the window's start when it was covered whole). A page never reaches past a cut scan. Null and false when the search was not capped, which is the dashboard's case. |
| `tier`, `index` | The tier that served it (`hot`) and the index behind it. A search with a body term says `trigram` (the body index exists and every partition has it) or `none`; any other search names the access path its filters let the planner use: `primary_key`, `project_trace_time`, `issue_time`, `project_service_time` or `project_time`. The planner chooses; this is the shape's. |
| `rowsExamined` | The rows the read looked at. Without the index a body search reads the newest 100,000 rows of its window and says how many it read; with it, the matching rows counted (`scanned`). Null when the search was not capped. |
| `nextArrivalCursor` | Where "what arrived since" resumes; see [Reading what arrived](#reading-what-arrived-since-a-cursor-not-a-stream). |
| `floorNote` | Records at error severity or worse, records with an exception, records linked to an issue and records on an error trace are never sampled away. Anything else may have been, and says so in its `sampleRate`. |
| `sampleRates` | The distinct sample rates below 1 on this page. A count over a sampled record is an estimate of `1 / rate` per row. |

`GET /orgs/:org/projects/:project/logs/:recordId` returns one record with its joins (below). A
record of another project, or of another organization, is a 404 identical to an unknown id.

## Searching the body

Three body terms, all case-insensitive, all combinable with the grammar and with each other:

- **contains**: `q` (and a free-text term in `query` or `where`), `body ILIKE '%term%'`;
- **prefix**: `bodyPrefix`, `body ILIKE 'term%'`;
- **regex**: `bodyRegex`, a POSIX regular expression (`body ~* pattern`; note `\y` is the word boundary and `\b` is a backspace). At most **100 characters**, **no backreferences** (`\1`..`\9`), and cancelled after **10 seconds** with a 422 that says so. A pattern longer than that, with a backreference or that does not compile is a 400. `*:term` over every attribute value is not offered: a blanket index over attribute values is a write cost this deployment does not take on.

`LIKE` characters in a term (`%`, `_`) are literals.

**With the index.** `log_records_body_trgm_idx` is a `pg_trgm` GIN index on `body` (`fastupdate`
on) on every `log_records` partition, behind the `log_index` pillar. It is built by code and by an
operator script, never by a migration: a migration would build it on every partition inside one
transaction on the busiest insert table, whether or not the operator chose its write cost.

- New partitions are indexed as they are created once the pillar is on (`pillar.log_index.enabled`,
  switched by the operator after the insert-cost harness, H-30).
- Existing partitions are indexed by `pnpm -C code --filter @bucker/api backfill-log-trgm` (dry run
  by default, `-- --apply` to build, `-- --days N` to take the newest N days first): one partition
  at a time, `CREATE INDEX CONCURRENTLY`, then attached to the parent index, resumable, and refused
  while the pillar is off. The DEFAULT partition is indexed in a full run.
- The response says `index: trigram` only when the parent index exists and every partition, the
  DEFAULT included, carries a valid trigram index of its own. A half-finished backfill reads
  `none`. (This is read from the partitions, not from the parent's own valid flag: Postgres sets
  that flag only on an attach, so a parent left invalid by a day that retention has since dropped
  would otherwise read `none` for good. A backfill stopped between building a day's index and
  attaching it attaches it on the next run rather than skipping the day.)
- Turning the pillar off stops this code indexing new partitions; it does not drop the index, and
  once the parent index exists Postgres indexes every new partition itself. Removing it is an
  operator's `DROP INDEX log_records_body_trgm_idx`.

**Without it** (`index: none`). A body search does not run into the statement timeout. It reads the
newest 100,000 rows of its window that pass the other filters, newest first, applies the body term
to them, and says so: `rowsExamined`, `truncated: true` and `scannedThrough`, the timestamp of the
oldest row it read. When the page is not full and the window is not covered, `hasMore` is true and
`nextCursor` continues from the last row read, so the next call reads the next slice. A search
with no window is bounded to 24 hours, never more than the deployment's hot window.

## Reading what arrived since: a cursor, not a stream

A person's live tail is a server-sent stream. An agent gets no stream: `search_logs` takes
`afterArrivalCursor`, the `nextArrivalCursor` of an earlier answer, and returns only the records
**received** after it, oldest first, so "what came in since my last read" is one budgeted call.

- Every `search_logs` answer carries `search.nextArrivalCursor`. Start from the one on any ordinary
  read, or from a change event's time, and pass it back. An ordinary read that found nothing hands
  out its own instant, so a quiet project can be watched from "now".
- An arrival read looks at the last **5 minutes of record time**: `receivedAt` is not indexed, and
  this window keeps the read a range scan on the timestamp index. A record stamped by a clock more
  than five minutes behind its arrival is in a search, not in an arrival read.
- Nothing new returns an empty page and the same cursor. When a page was cut for the token budget
  the cursor resumes after the last record kept, so a record dropped for size is read next time.
- **At most 6 arrival reads a minute per principal per project**; the seventh is a 429 carrying the
  seconds to wait. An ordinary read is not counted. The limit is per api machine.

### The dashboard tail's bounds

`GET …/logs/stream` is bounded so one tenant cannot starve ingest of the primary's 8 connections:

| Bound | Value |
| --- | --- |
| Open tails | 3 per project and 10 per organization, per api machine; the next is a 429 |
| Poll interval | 2 seconds, 10 seconds after 30 seconds with nothing new, back to 2 on a record |
| One tick's query | a 2 second statement timeout; a tick that cannot finish is retried from the same cursor |
| A bad `bodyRegex` | a 400 before the stream opens (too long, a backreference, or a pattern Postgres cannot compile), taking no slot |
| High volume | above 500 matching records a second a tick keeps a trace-hash sample (a trace is kept or dropped whole) and sends `{ kind: 'sampled', rate, note: 'sampled at 0.333' }` |
| A capped tick | sends `{ kind: 'capped', shown, skipped }`: how many matching records it did not deliver |

## The MCP tools

| Tool | What it does | Needs |
| --- | --- | --- |
| `search_logs` | Rows. `where`, `fields`, a `cursor` and the evidence above; every hit carries its `id`, `service`, `sampleRate` and the short id of its issue. An empty answer says why (`not_ingested`, `no_data_in_window`, `filtered_out`). | `project:read` |
| `logs_query` | Numbers: counts and measures per bucket and group, through the query printer. | `event:read` |
| `log_record_get` | One record with its joins in 1,200 tokens. | `event:read` |

Everything these tools return that came from outside Bucker (bodies, attribute keys and values,
service names, group labels, titles) passes the agent-tier scrub and the injection scanner and is
named in `trust.untrustedFields`. Attributes, projected `fields` included, are scrubbed as a bag,
so the rules that match on a key apply: `user.id`, `user.email` and `user.ip_address` come back
hashed, and a credential-named key comes back filtered.

### `logs_query`

"ERROR lines by service every 5 minutes for 6 hours" is one call:

```json
{ "org": "acme", "project": "checkout", "where": "level:>=error",
  "groupBy": ["service"], "intervalMinutes": 5, "windowHours": 6 }
```

- `where` is the log search grammar, **AND only**: an `OR` or a negated group is refused with the
  parse echoed, rather than narrowed to a different question. A simple negation (`!env:staging`)
  is fine.
- `measures` (up to 4): `count` (the default), `count_unique`, `sum`, `avg`, `min`, `max`, `p50`,
  `p75`, `p95`, `p99`. All but `count` take a `field`.
- `groupBy` takes up to three keys. **Each key is its own breakdown** over the same filters and
  window: three marginal series, not the cross product. The printer groups by one key per run.
- A series is a dense array per measure: `v.<measure>[i]` is the bucket at `from + i × step`.
- Each breakdown says `countBasis`: `exact`, or `sampled` when any scanned row was stored
  sampled, in which case every count is a lower bound and the `sampling` block gives the number
  of sampled rows, the lowest rate and the corrected estimate.
- The scan is the newest 100,000 matching rows inside the window (at most 30 days) and says so
  when it was cut. It is charged to the QUERY_SCAN budget before it runs and logged on the `mcp`
  surface.
- With more than one key, a later page carries only the breakdowns that still have groups to
  read; each names its `groupBy`.
- When the answer would exceed its budget the smallest groups are dropped, `omitted` says how
  many, and no cursor is offered; ask again with a smaller `limit`.

### `log_record_get`

The record, plus:

| Join | Source | When it is empty |
| --- | --- | --- |
| `issue` | the issue the record became: short id, status, level, title | `no_key`, or `not_found` |
| `trace` | a summary of the record's trace: span count, duration, services, error spans, the root. At most 200 spans are read; `truncated` says when more exist. | `no_key`, or `not_kept` (logs and traces are sampled independently, so a log can name a trace that was not stored) |
| `release` | the release and its deploy change event: when, the summary, minutes between | `no_key`; the deploy is null when none was recorded |
| `flags` | the variants the request's root span carried (`flagsBasis: trace_root`), else the variants the issue's events carried over a span of time that covers the record (`issue_events`) | `no_key`, or `not_found` |
| `pattern` | a placeholder until log patterns ship | always `not_available` |
| `neighbours` | the five nearest records in time, within 15 minutes, on the trace, else on the service, else in the project | `not_found` |

The bundle is shrunk in a fixed order to fit: attribute values, then neighbour bodies, then
neighbours from the farthest, then the attribute bag. The head of the body, the severity, the
service and every join's key are never cut, and `omitted` counts what was.

## What this page does not cover

- **Exception records inside a logs export are not split out.** A log record that carries
  `exception.*` becomes an issue at ingest, and the export of a project's logs still lists it as
  one row among the others. Separating the exception records in an export is open.
- **`subjectKey` and `session_id`** are not populated; see above.
- **Patterns** (`pattern` on `log_record_get`) arrive with the pattern miner.
- **Insert cost of the index** is measured by the capacity harness before an operator switches the pillar on; this page states the mechanism, not the measurement.
