Skip to content
bucker

Sources: business data as evidence

A source brings business data (invoices, accounts, orders) into Bucker as typed tables, so a question about an error or an experiment can be answered against the revenue it touched. It lands on the same seam the error-tracker connectors use, is metered in the same units, and is governed like every other configuration change: proposed by a person or an agent, approved by a person, inert until a second approval turns it on.

Everything on this page is opt-in and inert by default. A deployment with no source configured runs no sync, opens no connection and stores no business row.

What a source is

Part What it holds
Manifest A versioned, reviewed description of a kind of source, shipped in the repository. A source records the manifest type and version it was created on and keeps reading that version.
Source A name, a mode, a state (draft, enabled, paused, failing), a schedule, a monthly unit cap and the month's meter.
Tables The tables it lands: primary key, the columns kept, an optional cursor column, and (behind the product-analytics gate) a subject or account column.
Runs One row per sync with a receipt: rows read, inserted, updated, unchanged, retired, rejected by reason, units, and a digest.
Rows The landed rows, scrubbed at the stored tier before anything is hashed or written.

Built-in manifests

Type Label Modes Auth Notes
upload_csv beta upload your session A CSV file of at most 50 MB through the blob store.
upload_jsonl beta upload your session JSON Lines, at most 50 MB.
upload_parquet alpha upload your session Declared and priced; the reader is not built yet, so an upload is refused with that reason rather than half-read.
object_path alpha object_path, full_refresh, append a sealed connection An S3-compatible prefix. The credential is sealed in a connection (see connections); the source names the connection by id and nothing else.
inbound_webhook beta webhook a per-source signing secret A signed endpoint with the same replay dedupe the other webhook doors use.

A manifest cannot carry a host, a URL or a secret; registration refuses one that does. A later package adds its own manifest with registerManifest (Stripe's is its own file). There is no path from a request to a new manifest.

Setting one up

  1. Preview. POST .../sources/preview (or sources_list with include: [preview] and a candidate) returns the schema it would land (column names, types and null fractions), the row forecast with the basis it came from (the provider's count, one sample page, a stated file size, declared columns, or unknown; a number is never invented), the units, the cap that would pause it, and for a subject or account column an estimated join-key match against this project's subjects. It returns no rows: business rows are person-level data. It also returns a previewToken, good for 15 minutes.
  2. Propose. POST .../sources or the source_propose tool, with that token. The token proves the forecast was shown for exactly this candidate in this project; a missing, expired, foreign or changed one is refused before anyone is asked. The act is pipeline.source.create, which always asks, whatever the organization's policy says. The source is then filled in, and it is still a draft.
  3. Go live. state: enabled is its own act, pipeline.source.resume, and asks again.
  4. Feed it. Upload a file (POST .../sources/:name/upload?table=...), deliver to the webhook, or let the schedule pull.

Changing one

POST .../sources/:name/update and the sources_update tool take exactly one change. The act is derived from the difference between the source as it is and the one you ask for, never from an argument.

Change Act Risk Applies
Pause pipeline.source.pause ELEVATED Under an ALLOW policy; SENSITIVE and asked if a join reads this source
Fewer tables or columns, or a slower schedule pipeline.source.narrow ELEVATED Under an ALLOW policy; SENSITIVE and asked if a join reads what is removed (a slower schedule starves every join over the source)
More tables or columns, or a faster schedule pipeline.source.update SENSITIVE After a person's yes
Resume, or go live pipeline.source.resume SENSITIVE After a person's yes
A lower monthly cap pipeline.source_cap.update SENSITIVE Asked: a cap under the month's meter pauses the source, so lowering it is not a free tightening
A higher monthly cap pipeline.source_cap.update SENSITIVE After a person's yes
An earlier historyFrom pipeline.source.backfill SENSITIVE After a person's yes
Delete the source and its rows pipeline.source.purge irreversible Human-only, org:admin

The purge is DELETE .../sources/:name for an organization owner behind a person's session. No agent, OAuth client or delegated credential reaches it, and the MCP tools have no field for it. Rotating a webhook source's signing secret is likewise a person's act.

Units, caps and what happens at the cap

A landed row costs 0.01 units when it changes: a row that arrives again unchanged costs nothing, so a nightly snapshot of a quiet table is nearly free. Every source has a monthly cap in units (10,000 by default, which is a million changed rows); usage_get reports each source's unitsMonth.

A source that reaches its cap pauses, with a reason a person can read and a notice (source.cap_paused in the audit log). It does not sample: the over-cap sampling used for telemetry is never applied to business rows, because a revenue table with every tenth row missing is a wrong answer, not a cheaper one. Rows already landed are kept, and nothing more lands until a person raises the cap, or resumes the source once the month has turned over.

Landing

  • Scrub on write. Every value is scrubbed at the stored tier before it is hashed or stored.
  • Identity. A row is identified by its declared primary key. A table with none is identified by its content, which means a changed row is a new row; the preview says so.
  • Upload semantics. An upload is by default the whole table: rows it no longer holds are retired (soft-deleted). replace=false appends.
  • Months. A row lives in the monthly partition it first landed in and is updated there.
  • Subjects and accounts. A subject or account column is removed from the stored data and kept as a keyed pseudonym beside it, only while the project's analytics gate is open. With the gate shut the link is dropped, never kept raw. Those pseudonyms are what an erasure matches on.
  • Webhooks. The body is read raw and verified against an HMAC of the exact bytes (x-bucker-signature) under a secret derived per source. An unknown source, a wrong signature, a draft and another source's secret all answer the same 401. The event id is the dedupe key, so a replay writes nothing; a stale signed event is dropped. The project is the source's, never the body's.
  • Pulls. Scheduled by a worker pass (sources_sync); it honors the upstream's rate limit with the upstream's own wait, pages by a stored position, and marks a source failing on an error and enabled again on the next success.

For agents

Tool Does Relation
sources_list Lists sources; include: [preview] with a candidate forecasts one project:read; the preview block needs analytics:read
source_propose Creates, always pending approval pipeline:write
sources_update One change to one source pipeline:write

All three are held behind the SG-059 security review: a token a production path can mint does not list them. Source, table and column names come from your data, so they are sanitized and declared untrusted in every answer. No credential appears in any request or response.

Erasure and export

Rows linked to a subject or an account by a keyed pseudonym are reached by an erasure or an access request for that pseudonym, in every month and only in the project that holds them. A row whose table named no subject or account column holds no key and is not reached.