# The insight query

An insight is a question about how people use your product: how many, how they convert, who comes
back, where they go. It is written as one **node**, a small JSON document from a closed set, and
answered from the project's **rollups**, never from raw events. The answer comes back with an
**envelope** that says which data cut produced it, so the same number can be re-derived tomorrow.

Everything here is behind the project's product-analytics gate (PA-01): a project that has not
enabled analytics, with its lawful basis recorded and the amended DPA accepted, refuses every door
below before the node is even read.

## The doors

Three REST doors, all reads of the project (`analytics:read`, relation `read_analytics`). They are
`POST` because the node is a body; none of them changes anything you can see.

| Route | What it does |
| --- | --- |
| `POST /orgs/{org}/projects/{project}/insights/query` | Run a node. Returns `{ envelope, data }`. |
| `POST /orgs/{org}/projects/{project}/insights/preview` | The plan the node would make, without running it: what it resolves to, the specs and tables it compiles to, why the rollups cannot answer it if they cannot, lint. Reads no rollup, stores nothing. |
| `POST /orgs/{org}/projects/{project}/insights/upgrade` | Migrate a node written in an older version to the current one. Rewrites JSON only. |

The body is `{ "node": { ... } }`. The project is the path and nothing else: a node cannot name a
project or an organization.

## The node grammar (version 1)

Every node is a strict object. A field this page does not name is refused with its path, not
ignored. Every node carries `v: 1`, a `kind`, a `range`, and the shared fields below.

| Shared field | Meaning |
| --- | --- |
| `range` | `{ "kind": "relative", "last": 30, "unit": "day" \| "week" \| "month" }` (ends today, UTC) or `{ "kind": "absolute", "from": "YYYY-MM-DD", "to": "YYYY-MM-DD" }`. At most 400 days. |
| `filters` | Up to 10, from the closed grammar: `event`, `action`, `property` (`eq`, `neq`, `in`, `not_in`, `exists`, `not_exists`), `cohort`, `account`. There is no expression language and no regular expression. |
| `exclude` | `{ "internal": boolean, "actors": [subject keys] }`. |
| `identity` | `as_of_event` (default) or `current`. |

| Kind | Fields | Limits |
| --- | --- | --- |
| `trends` | `series[]` (`event`, `math`: `total` `unique_subjects` `sessions` `accounts`, `filters`), `breakdowns[]`, `interval` (`hour` `day` `week` `month`), `compare` | 20 series; 2 breakdowns of at most 25 values, the rest folded into `other` |
| `funnel` | `steps[]` (`event` or `action`), `window` (`value`, `unit`: minute hour day), `order` (`ordered` `unordered` `strict`) | 2 to 20 steps; a window of 1 minute to 90 days |
| `retention` | `event`, `periods`, `period` (`day` `week`), `lookback_days` | 12 periods |
| `paths` | `start`, `steps`, `max_edges`, `exclude_events[]` | 5 steps, 50 edges |
| `stickiness` | `event` | DAU, WAU, MAU and the ratios per day |
| `lifecycle` | `event`, `period`, `lookback_days` | new, returning, resurrecting, dormant |
| `web` | `metric`, `interval`, `breakdown` | compiled by the web-analytics extension |

A version-0 node (`type`, a bare `days`, `events`, `window_days`) is migrated by `upgrade`.
A version this build does not know is refused, never guessed at.

### Events must be core events

A rollup holds an event only if it is `$any` (every event) or one of the project's declared core
events (up to 20). Asking for another event is a `422` with `reason: event_not_core` and the list
of core events, not an empty answer. A project still in its first-week observation has `$any` only.
A read never freezes the observation default; only the taxonomy update and the first fold do.

### The horizon

"New" depends on how far back you trust the data. Retention and lifecycle take `lookback_days`
(default 90, at most 365): a subject active on the cohort day and on any day from
`range.from - lookback_days` to the day before is not new. The envelope echoes the horizon.

## The envelope

```json
{
  "v": 1,
  "engine": "insight-compiler/1",
  "kind": "trends",
  "nodeHash": "…",
  "result": "ok",
  "compiled": { "specs": [ … ], "reads": [{ "table": "pa_core_event_daily", … }] },
  "resolved": { "range": { "from": "2026-11-01", "to": "2026-11-30", "days": 30, "relative": true }, "interval": "day", "coreEventsVersion": 2, "identity": "as_of_event", "horizon": null },
  "isApproximate": true,
  "approximation": ["subject_sketch_estimate"],
  "scanned": { "rows": 90, "resultBytes": 2113 },
  "tier": "postgres_rollup",
  "killed": false,
  "trustCheckDigest": null,
  "reconciliation": { "drops": 4, "sampled": 0, "quarantined": 0, "botsExcluded": null, "internalExcluded": null, "personless": null, "unavailable": ["botsExcluded", "internalExcluded", "personless"], "window": { … } },
  "rollup": { "id": "pa_core_event_daily", "version": 1, "state": "live", "watermark": "…", "reconciledWithin": 0.005, "lastDriftPct": 0.1 },
  "cohortFloor": { "applied": false, "k": 5, "floorVersion": 1, "suppressed": 0 },
  "warnings": [],
  "extensions": {},
  "cache": { "key": "…", "hit": false, "ageSeconds": 0, "ttlSeconds": 21600 }
}
```

- **`compiled`** is the node as F-05 query specs over `rollup:<table>` datasets, and the tables
  read. Every table is a sufficient-statistics rollup (`pa_core_event_daily`, `pa_subject_activity`,
  `pa_subject_event_first`, `pa_funnel_daily`, `pa_path_edges_daily`); a plan that names any other
  table is refused. The specs parse with the printer's grammar.
- **`resolved`** is what the node meant: a relative range as the absolute days it covered.
- **`isApproximate`** with its reasons. `subject_sketch_estimate` (a daily distinct count is a
  sketch, about 1% off), `sessions_summed_across_days` and its siblings (a week's sessions are the
  sum of days, an over-count), `funnel_first_of_day_rule` (a subject counts by its first sighting of
  a step each day, never above the strict funnel), `edges_folded_into_other`. An exact answer says
  `isApproximate: false`.
- **`reconciliation`** is what never reached the rollups: drops, sampled and quarantined counts from
  the outcome ledger and the filters. Facts the ledger does not keep today are `null` and listed in
  `unavailable`; a `null` is not a zero.
- **`rollup`** is the audited rollup's watermark and how closely the last audit matched the ledger
  (half a percent). A range that reaches past the watermark carries a warning.
- **`cache`**: see below.
- **`result`** is `ok`, `no_data` (no rollup row fell in the range) or `tier_unavailable`.

### What the rollups cannot answer

Where the grammar allows more than the rollups hold, the answer is `result: tier_unavailable` with
a reason, status 200, no data and no approximation. The columnar tier answers these later.

| Reason | Node |
| --- | --- |
| `interval_hour_unavailable` | an hourly trend (the rollups are daily) |
| `multiple_breakdowns_unavailable`, `multiple_dimensions_unavailable` | two breakdowns, or a filter and a breakdown on different properties (a rollup row carries one key) |
| `breakdown_not_promoted` | a breakdown on a property the project has not verified and promoted |
| `property_filter_unavailable` | a property filter that is not `eq` or `in` on a promoted property, or on any node but trends |
| `event_filter_unavailable` | an `event` filter (the series or step already names the event) |
| `funnel_order_unavailable` | an `unordered` or `strict` funnel (first-of-day rows keep no order inside a day) |
| `funnel_steps_unavailable`, `funnel_window_unavailable`, `step_filters_unavailable` | more than 10 steps, a window past 30 days, a filter on a step |
| `actor_exclusion_unavailable`, `identity_current_unavailable` | `exclude.actors`, `identity: current` |
| `subject_filter_unavailable` | a cohort or account restriction on a trends or paths node |
| `web_compiler_unavailable` | the web node with no web-analytics extension registered |

A `cohort`, `account` or `action` filter with no extension that resolves it is a `422`
(`extension_unavailable`), not an empty answer.

### The results cache

A computed answer is kept in `insight_results`, keyed by everything it depends on and nothing it
does not: the node's hash, the resolved absolute range, the rollup watermark, the core-event
version, the floor version and k, the viewer's floor class and the engine version. A changed cut is
a new key, so nothing is ever invalidated; an entry just ages out, by interval: **hour 1 hour,
day 6 hours, week and month 24 hours**. The gate is checked before the cache is consulted, so a
project whose gate has since shut is never served a cached answer. A stored answer carries no
timestamp, which is what lets a nightly re-run at the echoed range and watermark reproduce it
exactly.

## The floor, and what it is not

Every answer states a `cohortFloor` block: `applied`, `k`, `floorVersion`, how many cells were
withheld (`suppressed`) and the exemption when there is one. `k` is the project's
`minCohortSize` (default 5, 3 to 50). Once `installCohortFloor()` has run at api boot, any cell
(a trends data point, a breakdown value, a funnel step, a retention cell, a lifecycle bucket) with
fewer than `k` distinct subjects is folded into `other` or shown as `"<k"`, and every ratio, total
or histogram that would hand the small count back is withheld with it: a trends total beside any
withheld point, a funnel entry-day column with one withheld day, an `other` that is itself under
`k`. A `paths` node is refused to a floored caller (`floor_refused`, rule
`paths_count_transitions`): an edge counts transitions, and one person looping between two pages
makes an edge of any size, so nothing in the rollup can show an edge holds `k` people. A filter
that pins one subject (a property that names a person, or a promoted value held by fewer than `k`
subjects) is refused with `floor_refused`. The floor applies to every caller except a signed-in
human holding the human-only `pii:read` and admitted by the organization's person-level gate; an
agent, an OAuth client, a delegated call or an organization token is floored even holding the
scope, and the review-mode step cannot lift it. An exempt read is ledgered (one `data.read` row,
no value on it) and cached under its own viewer class, never served to a floored caller.

`k` moves through the `analytics_floor` family: `analytics_floor.narrow` (raise, governed, ALLOW)
and `analytics_floor.widen` (lower, human-only `org:admin` on the person's own session), at
`GET/PUT .../analytics/floor` and `POST .../analytics/floor/preview`.

What the floor is not: differential privacy. The difference of two SHOWN counts (a funnel step's
entered minus converted) or of two answers can still isolate a small group; query budgets (PA-11)
and the read ledger bound that, the floor does not. And person-level reads (a subject's properties, an event
timeline, an actor list) are not insight nodes and never will be. They are dashboard reads for a
human holding `pii:read`, each writing an audit row.

## Extensions

Cohorts, accounts, actions, the web node, revenue and metric definitions plug in through
`registerNodeExtension`, from their own files. An extension can add a filter resolver, a breakdown
key, an `x_<name>` aggregation, a compiler for a node kind, an envelope block and the rollup tables
it reads. It cannot drop the tenant predicate (the tenant is the read's scope, applied by the
readers, not a clause), name a tenant key in a clause, mutate the context it is handed, widen a
subject set (an include set can only narrow, and two intersect), or read a table outside the rollups.
A failing or oversized envelope block is dropped with a warning; it never fails the read.
