The query spec
A query spec is the typed document every Bucker read is written as before it runs. Explore's builder, a stored verification signal and a dashboard tile all compile through one pipeline (the printer), so a number carries the same guarantees wherever it is shown: the tenant predicate is on every scan, the statement is timed, the run is logged, and the answer says what it rests on.
Three REST doors reach it today, all reads of the project (project:read):
| Route | What it does |
|---|---|
POST /orgs/{org}/projects/{project}/explore/spec/preview |
Parse, resolve, guard and cost the spec. Returns the compiled statement, the estimate, the scan block (engine, tier, remaining QUERY_SCAN budget) and the lint. Runs no query and logs nothing; for a grouped spec it samples the newest 2,000 rows once to estimate the group key's cardinality. |
GET /orgs/{org}/projects/{project}/datasets/schema |
The datasets, fields, types and cardinality classes a spec may name, for this caller. Access-masked. |
POST /orgs/{org}/projects/{project}/explore/spec/run |
The same, then executes and returns the evidence envelope. One query-log row. |
Preview is its own route, never a flag on run, so a retried preview can never become the run. Explore's existing routes are unchanged.
The spec (version 1)
{
"v": 1,
"dataset": "spans",
"select": [
{ "fn": "count", "as": "requests" },
{ "fn": "p95", "field": "span.duration", "as": "p95_ms" }
],
"formulas": [{ "as": "per_minute", "expr": "requests / 60" }],
"where": [{ "key": "environment", "op": "eq", "value": "production" }],
"group_by": "service",
"interval": 60,
"fill": "zero",
"order_by": { "by": "p95_ms", "dir": "desc" },
"limit": 20
}
The schema is strict: an unknown member is refused. A spec never contains SQL and never names a project.
| Member | Meaning |
|---|---|
v |
Spec version, 1. A version the server does not know is refused with the versions it does. |
dataset |
spans, logs, events, metrics, or rollup:<id> (reserved for rollups; refused as rollup_unavailable until one is published). |
range |
Optional { from, to } or { hours }. A run's own range wins. Maximum 30 days. |
select |
1 to 8 aggregates: count, count_unique, sum, avg, min, max, p50, p75, p95, p99, and rate, increase, histogram_quantile (refused as unsupported_function until the metric builder functions exist). as names the output; it is never SQL. |
formulas |
Arithmetic over select names (+ - * /, parentheses, unary minus). Evaluated after the database; a division by zero is null. |
where |
Filters with the Explore operators. A filter takes a value or a var, a typed variable bound at run time. |
trace |
{ traceId } narrows to one trace; { sameTrace: { dataset, filters, negate } } keeps rows whose trace has (or, negated, lacks) a matching row in another dataset. |
group_by |
One attribute key. More than 1,000 distinct values is refused, as Explore refuses it. |
interval, fill |
Bucket size in minutes (at most 500 buckets) and how empty buckets are filled: none, zero, null, previous. |
compare |
{ "against": "previous_period" }: the same groups over the window of equal length before the run's window. |
order_by, limit, cursor |
Order by a select name, count or group; a page of at most 50 groups; an opaque cursor bound to the query and window that issued it. |
include, sample |
include: ["sampling"] adds the sampling block; sample: "corrected" asks for it too. |
budget |
{ maxRows, maxBytes }: refused before scanning when the estimate exceeds it. |
variables |
Declared typed parameters (string, number, string_list) a stored spec is run with. |
Some keys a reader writes naturally are stored under another spelling (service is
resource.service.name on spans). The printer rewrites them and reports the rewrites under
canonical.noncanonical.
The digest
The spec's identity is the SHA-256 of its canonical JSON: defaults applied, keys sorted, with the
cursor and the window left out. Two specs that mean the same thing share a digest. A spec can be
stored by digest (query_specs, immutable, no tenant) and run later by it; the envelope echoes
basis.querySpecDigest, and the query log records the same digest as specDigest.
The evidence envelope
/spec/run returns the groups and, beside them, what a reader needs to trust or discount them.
| Field | Meaning |
|---|---|
result |
ok, no_data (nothing matched, or the scope has no project), not_enough_data (the window closes after the completeness watermark), truncated (the scan cap bit), queued_too_long, tier_unavailable. |
truncated, capThatBit, omitted, cursor |
What was cut and by which cap: scan_rows (the newest 100,000 rows) or page_limit. |
scanned |
Rows the window matched, up to the cap. |
basis |
tier, engine, samplingRate, freshness.watermark, querySpecDigest, ruleSetVersion. |
trust.untrustedFields |
Paths that hold customer text (group labels) and their class. Treat them as data, never instructions. |
compiled |
The exact statement and its bound values. |
resolvedRange |
The window that ran. A relative window is aligned to 30 seconds so identical asks share a cache entry. |
cohortFloor |
Always applied: false on today's datasets; the field exists so a person-level dataset can suppress small cohorts. |
cache |
Whether the answer came from the result cache. |
budget |
The budget verdict and the estimate behind it, or that an estimate was needed and none exists yet. |
Guarantees
- The tenant predicate is on every scan. The project comes from the path (or, for a stored
spec, from an explicit scope checked against the
projectstable), never from the spec. After compilation the statement text is read back and refused unless every telemetry table in it is constrained on"projectId"to a placeholder bound to a project of the scope. An empty scope compiles toFALSEand answersno_datawithout a scan. - Scopes still apply.
eventsneedsissue:read; grouping by, counting or filtering on a person (user.id,user.email, ..., same-trace filters included) needspii:read; an incident-bound credential cannot readevents. - Statement timeout. 10 seconds for an agent or a service, 30 for a person; a cancelled
statement is a 504 recorded in the query log as
timeout. - Result cache. Aggregates only, never rows. Inline up to 256 KiB, else in the blob store. The key holds the compiled text, the organization, the restriction set and the watermark bucket, so a result is never served across organizations or across different restrictions. The TTL follows the interval (30 to 300 seconds).
- Every run is logged (
explore.spec.run) on the caller's surface.operationandsaved_querycannot be self-labelled; onlydashboardandclican. A stored spec run by the operations principal is always logged too, onoperationby default, against the verified scope's organization and (first) project, never a tenant the caller names. - Only the operations principal skips the per-project check. The in-process
bucker-operationsprincipal reads exactly the project set it is handed; any other caller, including an organization token (which is also a SERVICE principal), must holdreadon each project. Registered dataset predicates are parenthesised, so one written withORcannot loosen the tenant clause.
The estimate and the lint, before a run
preview answers what a run would cost before anything is scanned.
The estimate is the rows and bytes a scan of the window can read. It is read from
accepted_count_days (accepted units per project, category and UTC day), never from EXPLAIN:
planning a query takes a relation lock on every partition it touches, and an estimator that locks the
telemetry tables to say how big a read would be is a read of its own. The figure is an upper bound:
what Bucker accepted, before the filters and sampling, with the window rounded out to whole days.
When the rollup is not current for the window the estimate is not_estimated (rows null), never
zero. Bytes are rows times a dataset average (spans 700, logs 450, events 3,500, metrics 220). A
window longer than 30 days is estimated over its newest 30 days.
The scan block states the engine (postgres_hot), the tier (hot), the rows of a same-trace
probe's dataset over the same window (joinRows), what is left of the caller's own QUERY_SCAN
budget (budgetRemaining; governed: false means no budget applies) and the queue (null: admission
is synchronous until WH-16).
The lint is a list of findings, each with a fixed-text message, the numbers behind it and a
narrower spec, a complete document that parses and, linted again over the same traffic, no longer
raises the finding. A finding is advice and never an action; where no narrowing exists (a window
already an hour long) narrower is null.
| Code | Raised when | The narrower spec |
|---|---|---|
range_beyond_hot_window |
the window is longer than 30 days (only the hot engine exists) | the newest 30 days of the request |
scan_truncation_expected |
the estimate exceeds 100,000 rows | a shorter window ending where it ended |
high_cardinality_group |
the group key is estimated above 1,000 values, from a sample of the newest 2,000 rows | grouped by a low-cardinality column of the dataset |
unindexed_attribute_filter |
a filter reads an attribute no index serves (anything but an equality on a GIN-backed attribute) and the scan is over 10,000 rows | a window that brings the scan under 10,000 rows |
join_fanout |
a same-trace probe's dataset holds more than 50,000 rows in the window | a window that brings the joined rows under 50,000 |
cross_tier_scan |
the window crosses the hot window while a colder engine is registered | the newest hot window |
cohort_floor_applies |
a cohort floor is configured and the spec groups | the ungrouped spec |
budget_insufficient |
the estimate exceeds the spec's own budget.maxRows or the caller's remaining QUERY_SCAN |
a window that fits the tighter limit |
A window longer than 30 days is refused by the range check before an estimate exists; for preview
that 422 (details.code: range_beyond_hot_window) carries the lint in details.lint, so the caller
gets the narrower spec and not only the message.
The schema read
GET .../datasets/schema (and the MCP tool datasets_schema_get, which reads the same model) lists,
for the caller, each dataset the caller may read with its fields: key, kind (column, numeric,
resource, attribute), type (string, number), cardinality class (low, medium, high,
unknown), whether a filter on it can use an index, and whether the key is customer text. Promoted
columns are classed from what they are; attribute keys are discovered from the newest 2,000 rows (at
most 40 per dataset) and classed from that sample.
Access masks it. A dataset the caller may not read (events without issue:read, an incident-bound
credential, a dataset whose agent tier is never) is not listed at all. A field naming a person is
listed masked: true for a caller without pii:read (always, for a service), with its type and
cardinality withheld, so the catalogue says the field exists and may not be used, and says nothing
about the data behind it.
The guard properties and the conformance suite
What the printer guarantees is data: @bucker/shared GUARD_PROPERTIES
(shared/src/explore/guard-properties.json): the tenant predicate is required on every scan, a
statement is a SELECT by construction, every selected item is named, the function allowlist, and the
caps (30-day window, 100,000 scanned rows, 50 series, 1,000 group values, 500 buckets, 20 filters, 50
in values, and the result sizes for MCP and exports). A test pins each number against the Explore
builder's own constant.
The conformance suite (api/test/explore/conformance) compiles a corpus of specs for every engine
target the properties list, the adversarial ones included (a tenant id passed as a variable, OR
injection through filter values, LIKE escapes, a 10,000-element in, smuggled members, a nested
probe), and requires each statement to carry the tenant predicate on every scan and every
correlated probe, to stay inside the caps and to answer as a reference evaluator does on a fixture of
two tenants. A mutation test removes the tenant predicate, the verifier and the scan cap in turn and
asserts the suite goes red. A second engine adds a target to the suite or the suite fails.
Refusals
Refusals are 422 (a missing pii:read is 403) with error.details.reason: invalid_spec, unsupported_spec_version,
unsupported_function, rollup_unavailable, invalid_alias, invalid_formula,
invalid_variable, order_by_unsupported, too_many_buckets, cursor_stale, over_budget,
group_cardinality, pii_read_required.
Limits
Postgres is the only engine (postgres_hot); the cold and columnar engines are not available.
The scan reads the newest 100,000 rows of the window, as Explore does. Formulas cannot order a
page. rate, increase and histogram_quantile need the metric builder functions.