Skip to content
bucker

Exports

An export sends one stream of a project's data to a destination you control, on a schedule, with a cursor that follows the order Bucker RECEIVED rows in. A row that is stored late (a slow ingest job, a client whose clock is behind) is never skipped. Sending history, and sending something again, are separate acts that a person approves after reading a row estimate.

Exports ride on destinations: an export names an existing destination, and so inherits its host, its credential and its delivery ledger. Nothing here adds a place data can go.

Streams

Stream What a row is Order the cursor follows
error_essences an issue with its essence the issue's updatedAt
issue_rollups an issue's counts and state the issue's updatedAt
release_health_rollups an hourly release-health rollup the rollup's updatedAt
ingest_outcomes one outcome row (accepted, sampled, dropped) the row's createdAt
spans one stored span the time Bucker received it
logs one stored log record the time Bucker received it
product_rollups one pa_core_event_daily row closed days only
metric_rollups one 5-minute metric rollup closed buckets only
saved_query the result rows of one PINNED saved-query version the end of each window

Every stream is read for ONE project. Raw product events are folded before they are stored, so product_rollups is the only product stream: no raw event leaves. It is behind the per-project end-user analytics gate (PA-01), which is checked when the export is proposed, when it goes live and on every run. A breakdown cohort smaller than the project's floor leaves as other, and the distinct-subject sketch is not a field at all.

A field is named before it leaves. An export's fieldAllowList is a field of the stream's catalogue or a dotted path under one (attributes.http.method); anything else is refused when the export is proposed. Each stream's catalogue marks the fields that can identify a person (free text, attribute bags, a session or subject key).

The cursor

The cursor is { at, id }, strictly after it. It moves only after the batch it ends was acknowledged.

Spans and logs are read in (receivedAt, id) order from ingest/durable-log.ts. At HEAD there is no long-lived, arrival-ordered log of raw telemetry: ingest_jobs holds a request body only until its job completes and is pruned within about an hour, and its bodies are the request before sampling, scrubbing and the egress cut, so an export never opens one. It uses what is durable and arrival-stamped:

  1. The stored row. spans.receivedAt and log_records.receivedAt are the job's accept time. They are not indexed, and exports add no index to either table: a read bounds the INDEXED client-time column (one day before the arrival window), which keeps the scan on the right partitions, and uses receivedAt for order and filter only.
  2. The job log, as the high-water mark. A row is stamped with the time its request was accepted but exists only once its job is processed, so the newest arrival safe to read is the oldest job still pending or processing for that project. Without it a cursor could advance past a slow job and skip the row the job writes later. Jobs of every lane hold it, up to 12 hours; a job held longer stops holding and the pass reports capped.

A row whose client time is more than a day older than its arrival (a historical replay) is outside the indexed bound and is not reached by the forward cursor. The nightly parity check finds those days; a backfill sends them.

Rollup streams export CLOSED periods only (a day after it and a grace period have passed; a 5-minute bucket after ten minutes). A rollup table has no updatedAt, so a restated closed period is not re-sent by the forward cursor; a backfill sends it again.

A saved-query export runs the pinned version once per window, bound to the window's end and a lookback of whole hours, so it runs hourly, every six hours or daily. The manifest names the time basis.

No overlap duplicates

Every batch's key derives from the cursor range it covers, and the range is recorded in its ledger row. A crash between the send and the commit re-sends the SAME bytes under the SAME key (the receiver's idempotency key, webhook-id); it never cuts a second, different batch for the same rows. A batch is one arrival day's rows, at most 500, shrunk until the body fits.

What leaves

Per batch, in this order: suppressed subjects are dropped; the export's ordered rules run (a rule that cannot be evaluated holds the batch and pauses the export); the cut to the allow-list; the scrub at the STRICTER of the export's tier and the destination's; the body is signed (Standard Webhooks). Each body carries a manifest: the export, the stream, the schema version, the cursor range, the scrub tier and subjectBearingFields, with controller: "customer". An exported copy is the customer's, outside Bucker's erasure (F-08); the manifest says which fields a subject-rights request would have to reach.

A destination of kind object_store is accepted by the model and governance but is not delivered yet (its transport is the customer-bucket sink); such an export reports unsupported_kind and sends nothing.

Governance

Act Class How
pipeline.export.create SENSITIVE, ask floor export_propose; fills a draft shell; STAYS draft
pipeline.export.resume SENSITIVE, ASK going live, or resuming; its own ask
pipeline.export.update SENSITIVE, ask floor fewer fields, the agent tier, another schedule
pipeline.export.pause ELEVATED, ALLOW the cursor holds, so nothing is skipped
pipeline.export.backfill SENSITIVE, ask floor; CRITICAL when derived export_backfill_propose; always pending

The act is derived from the difference between the export and the state asked for; nothing in an input names an action. A change that ADDS a field or LOWERS the tier lets more leave: that is pipeline.destination.widen, a person's act with egress:admin, refused before anyone is asked, for an agent and for a person. The destination, the stream and a pinned saved-query version have no input field: changing one is a different export. No plan reads the clock: the preview digest and the backfill seal carry a slot, a plan verifies the keyed MAC alone, and the door refuses one older than 15 minutes once, when it is filed. An approval that takes a day still applies; a destination changed while the yes was pending spends it.

Backfill

A backfill sends a range of past rows again, at the destination's paced rate. It is priced first:

  • The row estimate comes from accepted_count_days; no scan of spans or log_records. It is an upper bound (sampled units count). When the daily counts are not current, or the stream has no category (a saved query), the estimate is NOT zero: it is stated as not made and priced as the worst case.
  • The destination digest and whether the destination changed since rows first left. Nothing records the destination's digest at the first delivery, so this fails closed: it changed unless it has not been written since the export's first acknowledged batch.
  • The class is derived: SENSITIVE with the ask floor for an unchanged destination, a range of at most 30 days and at most 10 million rows; CRITICAL (denied by default, dual control) otherwise. The caller states no number: the estimate and the digest are sealed to this export, this backfill row and this range.

A backfill ends no later than the live cursor, covers at most 366 days, and only one waits or runs per export. After the yes the workers' pass sends chunks through the export's own cut, rules, suppression and scrub. It YIELDS to live ingest (no work at all while the live lanes hold more than 1,000 pending jobs, and between chunks) and RESUMES where the ledger says it stopped. Rows already sent are not recalled. Cancelling a backfill is audited (pipeline.export.backfill.cancel).

The nightly parity check

For each UTC day behind an export's cursor, the check compares the rows that ARRIVED that day (counted through a client-time bound of seven days, wider than the forward read's one day) with the rows the export's batches READ. Rows looked at and deliberately not sent (a suppressed subject, a rule, a cut that left nothing) are counted as read. missed > 0 is a finding, with its export, stream and day; the day's accepted_count_days units are shown beside it for context and never judged against. The pass writes a scheduler_runs row (exports / parity), and the check is listed in the DATA-05 registry as export_parity.

Tools and routes

Tool Route Notes
exports_list GET …/exports, GET …/exports/:name project:read; include: [rules], [backfills], [preview]
(preview) POST …/exports/preview rows and bytes a day, fields, tier, egress entry, digest; no row, no network
export_propose POST …/exports always asked
exports_update POST …/exports/:name/update the engine derives update, pause or resume
(estimate) POST …/exports/:name/backfills/estimate a read: creates nothing
export_backfill_propose POST …/exports/:name/backfills always asked
(cancel) POST …/exports/:name/backfills/:id/cancel audited

All four tools are held (SG-059): configure_pipeline (pipeline:write), not mintable by a production token. Names, hosts and field paths are customer text, fenced and declared untrusted.