Skip to content
bucker

The query spec

A query spec is the typed document every Bucker read is written as before it runs. Explore's builder, a stored verification signal and a dashboard tile all compile through one pipeline (the printer), so a number carries the same guarantees wherever it is shown: the tenant predicate is on every scan, the statement is timed, the run is logged, and the answer says what it rests on.

Three REST doors reach it today, all reads of the project (project:read):

Route What it does
POST /orgs/{org}/projects/{project}/explore/spec/preview Parse, resolve, guard and cost the spec. Returns the compiled statement, the estimate, the scan block (engine, tier, remaining QUERY_SCAN budget) and the lint. Runs no query and logs nothing; for a grouped spec it samples the newest 2,000 rows once to estimate the group key's cardinality.
GET /orgs/{org}/projects/{project}/datasets/schema The datasets, fields, types and cardinality classes a spec may name, for this caller. Access-masked.
POST /orgs/{org}/projects/{project}/explore/spec/run The same, then executes and returns the evidence envelope. One query-log row.

Preview is its own route, never a flag on run, so a retried preview can never become the run. Explore's existing routes are unchanged.

The spec (version 1)

{
  "v": 1,
  "dataset": "spans",
  "select": [
    { "fn": "count", "as": "requests" },
    { "fn": "p95", "field": "span.duration", "as": "p95_ms" }
  ],
  "formulas": [{ "as": "per_minute", "expr": "requests / 60" }],
  "where": [{ "key": "environment", "op": "eq", "value": "production" }],
  "group_by": "service",
  "interval": 60,
  "fill": "zero",
  "order_by": { "by": "p95_ms", "dir": "desc" },
  "limit": 20
}

The schema is strict: an unknown member is refused. A spec never contains SQL and never names a project.

Member Meaning
v Spec version, 1. A version the server does not know is refused with the versions it does.
dataset spans, logs, events, metrics, or rollup:<id> (reserved for rollups; refused as rollup_unavailable until one is published).
range Optional { from, to } or { hours }. A run's own range wins. Maximum 30 days.
select 1 to 8 aggregates: count, count_unique, sum, avg, min, max, p50, p75, p95, p99, and rate, increase, histogram_quantile (refused as unsupported_function until the metric builder functions exist). as names the output; it is never SQL.
formulas Arithmetic over select names (+ - * /, parentheses, unary minus). Evaluated after the database; a division by zero is null.
where Filters with the Explore operators. A filter takes a value or a var, a typed variable bound at run time.
trace { traceId } narrows to one trace; { sameTrace: { dataset, filters, negate } } keeps rows whose trace has (or, negated, lacks) a matching row in another dataset.
group_by One attribute key. More than 1,000 distinct values is refused, as Explore refuses it.
interval, fill Bucket size in minutes (at most 500 buckets) and how empty buckets are filled: none, zero, null, previous.
compare { "against": "previous_period" }: the same groups over the window of equal length before the run's window.
order_by, limit, cursor Order by a select name, count or group; a page of at most 50 groups; an opaque cursor bound to the query and window that issued it.
include, sample include: ["sampling"] adds the sampling block; sample: "corrected" asks for it too.
budget { maxRows, maxBytes }: refused before scanning when the estimate exceeds it.
variables Declared typed parameters (string, number, string_list) a stored spec is run with.

Some keys a reader writes naturally are stored under another spelling (service is resource.service.name on spans). The printer rewrites them and reports the rewrites under canonical.noncanonical.

The digest

The spec's identity is the SHA-256 of its canonical JSON: defaults applied, keys sorted, with the cursor and the window left out. Two specs that mean the same thing share a digest. A spec can be stored by digest (query_specs, immutable, no tenant) and run later by it; the envelope echoes basis.querySpecDigest, and the query log records the same digest as specDigest.

The evidence envelope

/spec/run returns the groups and, beside them, what a reader needs to trust or discount them.

Field Meaning
result ok, no_data (nothing matched, or the scope has no project), not_enough_data (the window closes after the completeness watermark), truncated (the scan cap bit), queued_too_long, tier_unavailable.
truncated, capThatBit, omitted, cursor What was cut and by which cap: scan_rows (the newest 100,000 rows) or page_limit.
scanned Rows the window matched, up to the cap.
basis tier, engine, samplingRate, freshness.watermark, querySpecDigest, ruleSetVersion.
trust.untrustedFields Paths that hold customer text (group labels) and their class. Treat them as data, never instructions.
compiled The exact statement and its bound values.
resolvedRange The window that ran. A relative window is aligned to 30 seconds so identical asks share a cache entry.
cohortFloor Always applied: false on today's datasets; the field exists so a person-level dataset can suppress small cohorts.
cache Whether the answer came from the result cache.
budget The budget verdict and the estimate behind it, or that an estimate was needed and none exists yet.

Guarantees

  • The tenant predicate is on every scan. The project comes from the path (or, for a stored spec, from an explicit scope checked against the projects table), never from the spec. After compilation the statement text is read back and refused unless every telemetry table in it is constrained on "projectId" to a placeholder bound to a project of the scope. An empty scope compiles to FALSE and answers no_data without a scan.
  • Scopes still apply. events needs issue:read; grouping by, counting or filtering on a person (user.id, user.email, ..., same-trace filters included) needs pii:read; an incident-bound credential cannot read events.
  • Statement timeout. 10 seconds for an agent or a service, 30 for a person; a cancelled statement is a 504 recorded in the query log as timeout.
  • Result cache. Aggregates only, never rows. Inline up to 256 KiB, else in the blob store. The key holds the compiled text, the organization, the restriction set and the watermark bucket, so a result is never served across organizations or across different restrictions. The TTL follows the interval (30 to 300 seconds).
  • Every run is logged (explore.spec.run) on the caller's surface. operation and saved_query cannot be self-labelled; only dashboard and cli can. A stored spec run by the operations principal is always logged too, on operation by default, against the verified scope's organization and (first) project, never a tenant the caller names.
  • Only the operations principal skips the per-project check. The in-process bucker-operations principal reads exactly the project set it is handed; any other caller, including an organization token (which is also a SERVICE principal), must hold read on each project. Registered dataset predicates are parenthesised, so one written with OR cannot loosen the tenant clause.

The estimate and the lint, before a run

preview answers what a run would cost before anything is scanned.

The estimate is the rows and bytes a scan of the window can read. It is read from accepted_count_days (accepted units per project, category and UTC day), never from EXPLAIN: planning a query takes a relation lock on every partition it touches, and an estimator that locks the telemetry tables to say how big a read would be is a read of its own. The figure is an upper bound: what Bucker accepted, before the filters and sampling, with the window rounded out to whole days. When the rollup is not current for the window the estimate is not_estimated (rows null), never zero. Bytes are rows times a dataset average (spans 700, logs 450, events 3,500, metrics 220). A window longer than 30 days is estimated over its newest 30 days.

The scan block states the engine (postgres_hot), the tier (hot), the rows of a same-trace probe's dataset over the same window (joinRows), what is left of the caller's own QUERY_SCAN budget (budgetRemaining; governed: false means no budget applies) and the queue (null: admission is synchronous until WH-16).

The lint is a list of findings, each with a fixed-text message, the numbers behind it and a narrower spec, a complete document that parses and, linted again over the same traffic, no longer raises the finding. A finding is advice and never an action; where no narrowing exists (a window already an hour long) narrower is null.

Code Raised when The narrower spec
range_beyond_hot_window the window is longer than 30 days (only the hot engine exists) the newest 30 days of the request
scan_truncation_expected the estimate exceeds 100,000 rows a shorter window ending where it ended
high_cardinality_group the group key is estimated above 1,000 values, from a sample of the newest 2,000 rows grouped by a low-cardinality column of the dataset
unindexed_attribute_filter a filter reads an attribute no index serves (anything but an equality on a GIN-backed attribute) and the scan is over 10,000 rows a window that brings the scan under 10,000 rows
join_fanout a same-trace probe's dataset holds more than 50,000 rows in the window a window that brings the joined rows under 50,000
cross_tier_scan the window crosses the hot window while a colder engine is registered the newest hot window
cohort_floor_applies a cohort floor is configured and the spec groups the ungrouped spec
budget_insufficient the estimate exceeds the spec's own budget.maxRows or the caller's remaining QUERY_SCAN a window that fits the tighter limit

A window longer than 30 days is refused by the range check before an estimate exists; for preview that 422 (details.code: range_beyond_hot_window) carries the lint in details.lint, so the caller gets the narrower spec and not only the message.

The schema read

GET .../datasets/schema (and the MCP tool datasets_schema_get, which reads the same model) lists, for the caller, each dataset the caller may read with its fields: key, kind (column, numeric, resource, attribute), type (string, number), cardinality class (low, medium, high, unknown), whether a filter on it can use an index, and whether the key is customer text. Promoted columns are classed from what they are; attribute keys are discovered from the newest 2,000 rows (at most 40 per dataset) and classed from that sample.

Access masks it. A dataset the caller may not read (events without issue:read, an incident-bound credential, a dataset whose agent tier is never) is not listed at all. A field naming a person is listed masked: true for a caller without pii:read (always, for a service), with its type and cardinality withheld, so the catalogue says the field exists and may not be used, and says nothing about the data behind it.

The guard properties and the conformance suite

What the printer guarantees is data: @bucker/shared GUARD_PROPERTIES (shared/src/explore/guard-properties.json): the tenant predicate is required on every scan, a statement is a SELECT by construction, every selected item is named, the function allowlist, and the caps (30-day window, 100,000 scanned rows, 50 series, 1,000 group values, 500 buckets, 20 filters, 50 in values, and the result sizes for MCP and exports). A test pins each number against the Explore builder's own constant.

The conformance suite (api/test/explore/conformance) compiles a corpus of specs for every engine target the properties list, the adversarial ones included (a tenant id passed as a variable, OR injection through filter values, LIKE escapes, a 10,000-element in, smuggled members, a nested probe), and requires each statement to carry the tenant predicate on every scan and every correlated probe, to stay inside the caps and to answer as a reference evaluator does on a fixture of two tenants. A mutation test removes the tenant predicate, the verifier and the scan cap in turn and asserts the suite goes red. A second engine adds a target to the suite or the suite fails.

Refusals

Refusals are 422 (a missing pii:read is 403) with error.details.reason: invalid_spec, unsupported_spec_version, unsupported_function, rollup_unavailable, invalid_alias, invalid_formula, invalid_variable, order_by_unsupported, too_many_buckets, cursor_stale, over_budget, group_cardinality, pii_read_required.

Limits

Postgres is the only engine (postgres_hot); the cold and columnar engines are not available. The scan reads the newest 100,000 rows of the window, as Explore does. Formulas cannot order a page. rate, increase and histogram_quantile need the metric builder functions.