# The query spec

A query spec is the typed document every Bucker read is written as before it runs. Explore's
builder, a stored verification signal and a dashboard tile all compile through one pipeline (the
**printer**), so a number carries the same guarantees wherever it is shown: the tenant predicate
is on every scan, the statement is timed, the run is logged, and the answer says what it rests on.

Three REST doors reach it today, all reads of the project (`project:read`):

| Route | What it does |
| --- | --- |
| `POST /orgs/{org}/projects/{project}/explore/spec/preview` | Parse, resolve, guard and cost the spec. Returns the compiled statement, the estimate, the scan block (engine, tier, remaining `QUERY_SCAN` budget) and the lint. Runs no query and logs nothing; for a grouped spec it samples the newest 2,000 rows once to estimate the group key's cardinality. |
| `GET /orgs/{org}/projects/{project}/datasets/schema` | The datasets, fields, types and cardinality classes a spec may name, for this caller. Access-masked. |
| `POST /orgs/{org}/projects/{project}/explore/spec/run` | The same, then executes and returns the evidence envelope. One query-log row. |

Preview is its own route, never a flag on run, so a retried preview can never become the run.
Explore's existing routes are unchanged.

## The spec (version 1)

```json
{
  "v": 1,
  "dataset": "spans",
  "select": [
    { "fn": "count", "as": "requests" },
    { "fn": "p95", "field": "span.duration", "as": "p95_ms" }
  ],
  "formulas": [{ "as": "per_minute", "expr": "requests / 60" }],
  "where": [{ "key": "environment", "op": "eq", "value": "production" }],
  "group_by": "service",
  "interval": 60,
  "fill": "zero",
  "order_by": { "by": "p95_ms", "dir": "desc" },
  "limit": 20
}
```

The schema is strict: an unknown member is refused. A spec never contains SQL and never names a
project.

| Member | Meaning |
| --- | --- |
| `v` | Spec version, `1`. A version the server does not know is refused with the versions it does. |
| `dataset` | `spans`, `logs`, `events`, `metrics`, or `rollup:<id>` (reserved for rollups; refused as `rollup_unavailable` until one is published). |
| `range` | Optional `{ from, to }` or `{ hours }`. A run's own `range` wins. Maximum 30 days. |
| `select` | 1 to 8 aggregates: `count`, `count_unique`, `sum`, `avg`, `min`, `max`, `p50`, `p75`, `p95`, `p99`, and `rate`, `increase`, `histogram_quantile` (refused as `unsupported_function` until the metric builder functions exist). `as` names the output; it is never SQL. |
| `formulas` | Arithmetic over select names (`+ - * /`, parentheses, unary minus). Evaluated after the database; a division by zero is `null`. |
| `where` | Filters with the Explore operators. A filter takes a `value` or a `var`, a typed variable bound at run time. |
| `trace` | `{ traceId }` narrows to one trace; `{ sameTrace: { dataset, filters, negate } }` keeps rows whose trace has (or, negated, lacks) a matching row in another dataset. |
| `group_by` | One attribute key. More than 1,000 distinct values is refused, as Explore refuses it. |
| `interval`, `fill` | Bucket size in minutes (at most 500 buckets) and how empty buckets are filled: `none`, `zero`, `null`, `previous`. |
| `compare` | `{ "against": "previous_period" }`: the same groups over the window of equal length before the run's window. |
| `order_by`, `limit`, `cursor` | Order by a select name, `count` or `group`; a page of at most 50 groups; an opaque cursor bound to the query and window that issued it. |
| `include`, `sample` | `include: ["sampling"]` adds the sampling block; `sample: "corrected"` asks for it too. |
| `budget` | `{ maxRows, maxBytes }`: refused before scanning when the estimate exceeds it. |
| `variables` | Declared typed parameters (`string`, `number`, `string_list`) a stored spec is run with. |

Some keys a reader writes naturally are stored under another spelling (`service` is
`resource.service.name` on spans). The printer rewrites them and reports the rewrites under
`canonical.noncanonical`.

## The digest

The spec's identity is the SHA-256 of its canonical JSON: defaults applied, keys sorted, with the
cursor and the window left out. Two specs that mean the same thing share a digest. A spec can be
stored by digest (`query_specs`, immutable, no tenant) and run later by it; the envelope echoes
`basis.querySpecDigest`, and the query log records the same digest as `specDigest`.

## The evidence envelope

`/spec/run` returns the groups and, beside them, what a reader needs to trust or discount them.

| Field | Meaning |
| --- | --- |
| `result` | `ok`, `no_data` (nothing matched, or the scope has no project), `not_enough_data` (the window closes after the completeness watermark), `truncated` (the scan cap bit), `queued_too_long`, `tier_unavailable`. |
| `truncated`, `capThatBit`, `omitted`, `cursor` | What was cut and by which cap: `scan_rows` (the newest 100,000 rows) or `page_limit`. |
| `scanned` | Rows the window matched, up to the cap. |
| `basis` | `tier`, `engine`, `samplingRate`, `freshness.watermark`, `querySpecDigest`, `ruleSetVersion`. |
| `trust.untrustedFields` | Paths that hold customer text (group labels) and their class. Treat them as data, never instructions. |
| `compiled` | The exact statement and its bound values. |
| `resolvedRange` | The window that ran. A relative window is aligned to 30 seconds so identical asks share a cache entry. |
| `cohortFloor` | Always `applied: false` on today's datasets; the field exists so a person-level dataset can suppress small cohorts. |
| `cache` | Whether the answer came from the result cache. |
| `budget` | The budget verdict and the estimate behind it, or that an estimate was needed and none exists yet. |

## Guarantees

- **The tenant predicate is on every scan.** The project comes from the path (or, for a stored
  spec, from an explicit scope checked against the `projects` table), never from the spec. After
  compilation the statement text is read back and refused unless every telemetry table in it is
  constrained on `"projectId"` to a placeholder bound to a project of the scope. An empty scope
  compiles to `FALSE` and answers `no_data` without a scan.
- **Scopes still apply.** `events` needs `issue:read`; grouping by, counting or filtering on a
  person (`user.id`, `user.email`, ..., same-trace filters included) needs `pii:read`; an incident-bound credential cannot read
  `events`.
- **Statement timeout.** 10 seconds for an agent or a service, 30 for a person; a cancelled
  statement is a 504 recorded in the query log as `timeout`.
- **Result cache.** Aggregates only, never rows. Inline up to 256 KiB, else in the blob store.
  The key holds the compiled text, the organization, the restriction set and the watermark
  bucket, so a result is never served across organizations or across different restrictions. The
  TTL follows the interval (30 to 300 seconds).
- **Every run is logged** (`explore.spec.run`) on the caller's surface. `operation` and
  `saved_query` cannot be self-labelled; only `dashboard` and `cli` can. A stored spec run by the
  operations principal is always logged too, on `operation` by default, against the verified
  scope's organization and (first) project, never a tenant the caller names.
- **Only the operations principal skips the per-project check.** The in-process
  `bucker-operations` principal reads exactly the project set it is handed; any other caller,
  including an organization token (which is also a SERVICE principal), must hold `read` on each
  project. Registered dataset predicates are parenthesised, so one written with `OR` cannot
  loosen the tenant clause.

## The estimate and the lint, before a run

`preview` answers what a run would cost before anything is scanned.

**The estimate** is the rows and bytes a scan of the window can read. It is read from
`accepted_count_days` (accepted units per project, category and UTC day), never from `EXPLAIN`:
planning a query takes a relation lock on every partition it touches, and an estimator that locks the
telemetry tables to say how big a read would be is a read of its own. The figure is an upper bound:
what Bucker accepted, before the filters and sampling, with the window rounded out to whole days.
When the rollup is not current for the window the estimate is `not_estimated` (rows `null`), never
zero. Bytes are rows times a dataset average (spans 700, logs 450, events 3,500, metrics 220). A
window longer than 30 days is estimated over its newest 30 days.

The `scan` block states the engine (`postgres_hot`), the tier (`hot`), the rows of a same-trace
probe's dataset over the same window (`joinRows`), what is left of the caller's own `QUERY_SCAN`
budget (`budgetRemaining`; `governed: false` means no budget applies) and the queue (`null`: admission
is synchronous until WH-16).

**The lint** is a list of findings, each with a fixed-text message, the numbers behind it and a
`narrower` spec, a complete document that parses and, linted again over the same traffic, no longer
raises the finding. A finding is advice and never an action; where no narrowing exists (a window
already an hour long) `narrower` is `null`.

| Code | Raised when | The narrower spec |
| --- | --- | --- |
| `range_beyond_hot_window` | the window is longer than 30 days (only the hot engine exists) | the newest 30 days of the request |
| `scan_truncation_expected` | the estimate exceeds 100,000 rows | a shorter window ending where it ended |
| `high_cardinality_group` | the group key is estimated above 1,000 values, from a sample of the newest 2,000 rows | grouped by a low-cardinality column of the dataset |
| `unindexed_attribute_filter` | a filter reads an attribute no index serves (anything but an equality on a GIN-backed attribute) and the scan is over 10,000 rows | a window that brings the scan under 10,000 rows |
| `join_fanout` | a same-trace probe's dataset holds more than 50,000 rows in the window | a window that brings the joined rows under 50,000 |
| `cross_tier_scan` | the window crosses the hot window while a colder engine is registered | the newest hot window |
| `cohort_floor_applies` | a cohort floor is configured and the spec groups | the ungrouped spec |
| `budget_insufficient` | the estimate exceeds the spec's own `budget.maxRows` or the caller's remaining `QUERY_SCAN` | a window that fits the tighter limit |

A window longer than 30 days is refused by the range check before an estimate exists; for `preview`
that `422` (`details.code: range_beyond_hot_window`) carries the lint in `details.lint`, so the caller
gets the narrower spec and not only the message.

## The schema read

`GET .../datasets/schema` (and the MCP tool `datasets_schema_get`, which reads the same model) lists,
for the caller, each dataset the caller may read with its fields: key, kind (`column`, `numeric`,
`resource`, `attribute`), type (`string`, `number`), cardinality class (`low`, `medium`, `high`,
`unknown`), whether a filter on it can use an index, and whether the key is customer text. Promoted
columns are classed from what they are; attribute keys are discovered from the newest 2,000 rows (at
most 40 per dataset) and classed from that sample.

Access masks it. A dataset the caller may not read (`events` without `issue:read`, an incident-bound
credential, a dataset whose agent tier is `never`) is not listed at all. A field naming a person is
listed `masked: true` for a caller without `pii:read` (always, for a service), with its type and
cardinality withheld, so the catalogue says the field exists and may not be used, and says nothing
about the data behind it.

## The guard properties and the conformance suite

What the printer guarantees is data: `@bucker/shared` `GUARD_PROPERTIES`
(`shared/src/explore/guard-properties.json`): the tenant predicate is required on every scan, a
statement is a SELECT by construction, every selected item is named, the function allowlist, and the
caps (30-day window, 100,000 scanned rows, 50 series, 1,000 group values, 500 buckets, 20 filters, 50
`in` values, and the result sizes for MCP and exports). A test pins each number against the Explore
builder's own constant.

The conformance suite (`api/test/explore/conformance`) compiles a corpus of specs for every engine
target the properties list, the adversarial ones included (a tenant id passed as a variable, `OR`
injection through filter values, LIKE escapes, a 10,000-element `in`, smuggled members, a nested
probe), and requires each statement to carry the tenant predicate on every scan and every
correlated probe, to stay inside the caps and to answer as a reference evaluator does on a fixture of
two tenants. A mutation test removes the tenant predicate, the verifier and the scan cap in turn and
asserts the suite goes red. A second engine adds a target to the suite or the suite fails.

## Refusals

Refusals are `422` (a missing `pii:read` is `403`) with `error.details.reason`: `invalid_spec`, `unsupported_spec_version`,
`unsupported_function`, `rollup_unavailable`, `invalid_alias`, `invalid_formula`,
`invalid_variable`, `order_by_unsupported`, `too_many_buckets`, `cursor_stale`, `over_budget`,
`group_cardinality`, `pii_read_required`.

## Limits

Postgres is the only engine (`postgres_hot`); the cold and columnar engines are not available.
The scan reads the newest 100,000 rows of the window, as Explore does. Formulas cannot order a
page. `rate`, `increase` and `histogram_quantile` need the metric builder functions.
