# Exports

An export sends one stream of a project's data to a destination you control, on a schedule, with a cursor
that follows the order Bucker RECEIVED rows in. A row that is stored late (a slow ingest job, a client whose
clock is behind) is never skipped. Sending history, and sending something again, are separate acts that a
person approves after reading a row estimate.

Exports ride on [destinations](destinations.md): an export names an existing destination, and so inherits
its host, its credential and its delivery ledger. Nothing here adds a place data can go.

## Streams

| Stream | What a row is | Order the cursor follows |
| --- | --- | --- |
| `error_essences` | an issue with its essence | the issue's `updatedAt` |
| `issue_rollups` | an issue's counts and state | the issue's `updatedAt` |
| `release_health_rollups` | an hourly release-health rollup | the rollup's `updatedAt` |
| `ingest_outcomes` | one outcome row (accepted, sampled, dropped) | the row's `createdAt` |
| `spans` | one stored span | the time Bucker received it |
| `logs` | one stored log record | the time Bucker received it |
| `product_rollups` | one `pa_core_event_daily` row | closed days only |
| `metric_rollups` | one 5-minute metric rollup | closed buckets only |
| `saved_query` | the result rows of one PINNED saved-query version | the end of each window |

Every stream is read for ONE project. Raw product events are folded before they are stored, so
`product_rollups` is the only product stream: no raw event leaves. It is behind the per-project end-user
analytics gate (PA-01), which is checked when the export is proposed, when it goes live and on every run.
A breakdown cohort smaller than the project's floor leaves as `other`, and the distinct-subject sketch is
not a field at all.

A field is named before it leaves. An export's `fieldAllowList` is a field of the stream's catalogue or a
dotted path under one (`attributes.http.method`); anything else is refused when the export is proposed.
Each stream's catalogue marks the fields that can identify a person (free text, attribute bags, a session or
subject key).

## The cursor

The cursor is `{ at, id }`, strictly after it. It moves only after the batch it ends was acknowledged.

**Spans and logs** are read in `(receivedAt, id)` order from `ingest/durable-log.ts`. At HEAD there is no
long-lived, arrival-ordered log of raw telemetry: `ingest_jobs` holds a request body only until its job
completes and is pruned within about an hour, and its bodies are the request before sampling, scrubbing and
the egress cut, so an export never opens one. It uses what is durable and arrival-stamped:

1. The stored row. `spans.receivedAt` and `log_records.receivedAt` are the job's accept time. They are not
   indexed, and exports add no index to either table: a read bounds the INDEXED client-time column (one day
   before the arrival window), which keeps the scan on the right partitions, and uses `receivedAt` for order
   and filter only.
2. The job log, as the high-water mark. A row is stamped with the time its request was accepted but exists
   only once its job is processed, so the newest arrival safe to read is the oldest job still pending or
   processing for that project. Without it a cursor could advance past a slow job and skip the row the job
   writes later. Jobs of every lane hold it, up to 12 hours; a job held longer stops holding and the pass
   reports `capped`.

A row whose client time is more than a day older than its arrival (a historical replay) is outside the
indexed bound and is not reached by the forward cursor. The nightly parity check finds those days; a
backfill sends them.

**Rollup streams** export CLOSED periods only (a day after it and a grace period have passed; a 5-minute
bucket after ten minutes). A rollup table has no `updatedAt`, so a restated closed period is not re-sent by
the forward cursor; a backfill sends it again.

**A saved-query export** runs the pinned version once per window, bound to the window's end and a lookback of
whole hours, so it runs hourly, every six hours or daily. The manifest names the time basis.

### No overlap duplicates

Every batch's key derives from the cursor range it covers, and the range is recorded in its ledger row. A
crash between the send and the commit re-sends the SAME bytes under the SAME key (the receiver's
idempotency key, `webhook-id`); it never cuts a second, different batch for the same rows. A batch is one
arrival day's rows, at most 500, shrunk until the body fits.

## What leaves

Per batch, in this order: suppressed subjects are dropped; the export's ordered rules run (a rule that
cannot be evaluated holds the batch and pauses the export); the cut to the allow-list; the scrub at the
STRICTER of the export's tier and the destination's; the body is signed (Standard Webhooks). Each body
carries a manifest: the export, the stream, the schema version, the cursor range, the scrub tier and
`subjectBearingFields`, with `controller: "customer"`. An exported copy is the customer's, outside Bucker's
erasure (F-08); the manifest says which fields a subject-rights request would have to reach.

A destination of kind `object_store` is accepted by the model and governance but is not delivered yet (its
transport is the customer-bucket sink); such an export reports `unsupported_kind` and sends nothing.

## Governance

| Act | Class | How |
| --- | --- | --- |
| `pipeline.export.create` | SENSITIVE, ask floor | `export_propose`; fills a draft shell; STAYS `draft` |
| `pipeline.export.resume` | SENSITIVE, ASK | going live, or resuming; its own ask |
| `pipeline.export.update` | SENSITIVE, ask floor | fewer fields, the agent tier, another schedule |
| `pipeline.export.pause` | ELEVATED, ALLOW | the cursor holds, so nothing is skipped |
| `pipeline.export.backfill` | SENSITIVE, ask floor; CRITICAL when derived | `export_backfill_propose`; always pending |

The act is derived from the difference between the export and the state asked for; nothing in an input
names an action. A change that ADDS a field or LOWERS the tier lets more leave: that is
`pipeline.destination.widen`, a person's act with `egress:admin`, refused before anyone is asked, for an
agent and for a person. The destination, the stream and a pinned saved-query version have no input field:
changing one is a different export. No plan reads the clock: the preview digest and the backfill seal carry
a slot, a plan verifies the keyed MAC alone, and the door refuses one older than 15 minutes once, when it
is filed. An approval that takes a day still applies; a destination changed while the yes was pending
spends it.

## Backfill

A backfill sends a range of past rows again, at the destination's paced rate. It is priced first:

- **The row estimate** comes from `accepted_count_days`; no scan of `spans` or `log_records`. It is an
  upper bound (sampled units count). When the daily counts are not current, or the stream has no category
  (a saved query), the estimate is NOT zero: it is stated as not made and priced as the worst case.
- **The destination digest** and **whether the destination changed since rows first left**. Nothing records
  the destination's digest at the first delivery, so this fails closed: it changed unless it has not been
  written since the export's first acknowledged batch.
- **The class is derived**: SENSITIVE with the ask floor for an unchanged destination, a range of at most 30
  days and at most 10 million rows; CRITICAL (denied by default, dual control) otherwise. The caller states
  no number: the estimate and the digest are sealed to this export, this backfill row and this range.

A backfill ends no later than the live cursor, covers at most 366 days, and only one waits or runs per
export. After the yes the workers' pass sends chunks through the export's own cut, rules, suppression and
scrub. It YIELDS to live ingest (no work at all while the live lanes hold more than 1,000 pending jobs, and
between chunks) and RESUMES where the ledger says it stopped. Rows already sent are not recalled. Cancelling
a backfill is audited (`pipeline.export.backfill.cancel`).

## The nightly parity check

For each UTC day behind an export's cursor, the check compares the rows that ARRIVED that day (counted
through a client-time bound of seven days, wider than the forward read's one day) with the rows the export's
batches READ. Rows looked at and deliberately not sent (a suppressed subject, a rule, a cut that left
nothing) are counted as read. `missed > 0` is a finding, with its export, stream and day; the day's
`accepted_count_days` units are shown beside it for context and never judged against. The pass writes a
`scheduler_runs` row (`exports` / `parity`), and the check is listed in the DATA-05 registry as
`export_parity`.

## Tools and routes

| Tool | Route | Notes |
| --- | --- | --- |
| `exports_list` | `GET …/exports`, `GET …/exports/:name` | `project:read`; `include: [rules]`, `[backfills]`, `[preview]` |
| (preview) | `POST …/exports/preview` | rows and bytes a day, fields, tier, egress entry, digest; no row, no network |
| `export_propose` | `POST …/exports` | always asked |
| `exports_update` | `POST …/exports/:name/update` | the engine derives update, pause or resume |
| (estimate) | `POST …/exports/:name/backfills/estimate` | a read: creates nothing |
| `export_backfill_propose` | `POST …/exports/:name/backfills` | always asked |
| (cancel) | `POST …/exports/:name/backfills/:id/cancel` | audited |

All four tools are held (SG-059): `configure_pipeline` (`pipeline:write`), not mintable by a production
token. Names, hosts and field paths are customer text, fenced and declared untrusted.
