# Sources: business data as evidence

A source brings **business data** (invoices, accounts, orders) into Bucker as typed tables, so a
question about an error or an experiment can be answered against the revenue it touched. It lands
on the same seam the error-tracker connectors use, is metered in the same units, and is governed
like every other configuration change: proposed by a person or an agent, approved by a person,
inert until a second approval turns it on.

Everything on this page is **opt-in and inert by default**. A deployment with no source configured
runs no sync, opens no connection and stores no business row.

## What a source is

| Part | What it holds |
|---|---|
| Manifest | A versioned, reviewed description of a kind of source, shipped in the repository. A source records the manifest type and version it was created on and keeps reading that version. |
| Source | A name, a mode, a state (`draft`, `enabled`, `paused`, `failing`), a schedule, a monthly unit cap and the month's meter. |
| Tables | The tables it lands: primary key, the columns kept, an optional cursor column, and (behind the product-analytics gate) a subject or account column. |
| Runs | One row per sync with a receipt: rows read, inserted, updated, unchanged, retired, rejected by reason, units, and a digest. |
| Rows | The landed rows, scrubbed at the stored tier before anything is hashed or written. |

### Built-in manifests

| Type | Label | Modes | Auth | Notes |
|---|---|---|---|---|
| `upload_csv` | beta | `upload` | your session | A CSV file of at most 50 MB through the blob store. |
| `upload_jsonl` | beta | `upload` | your session | JSON Lines, at most 50 MB. |
| `upload_parquet` | alpha | `upload` | your session | Declared and priced; the reader is not built yet, so an upload is refused with that reason rather than half-read. |
| `object_path` | alpha | `object_path`, `full_refresh`, `append` | a sealed connection | An S3-compatible prefix. The credential is sealed in a connection (see [connections](connections.md)); the source names the connection by id and nothing else. |
| `inbound_webhook` | beta | `webhook` | a per-source signing secret | A signed endpoint with the same replay dedupe the other webhook doors use. |

A manifest cannot carry a host, a URL or a secret; registration refuses one that does. A later
package adds its own manifest with `registerManifest` (Stripe's is its own file). There is no path
from a request to a new manifest.

## Setting one up

1. **Preview.** `POST .../sources/preview` (or `sources_list` with `include: [preview]` and a
   `candidate`) returns the schema it would land (column names, types and null fractions), the row
   forecast with the **basis** it came from (the provider's count, one sample page, a stated file
   size, declared columns, or `unknown`; a number is never invented), the units, the cap that would
   pause it, and for a subject or account column an estimated join-key match against this
   project's subjects. It returns **no rows**: business rows are person-level data. It also returns
   a `previewToken`, good for 15 minutes.
2. **Propose.** `POST .../sources` or the `source_propose` tool, with that token. The token proves
   the forecast was shown for exactly this candidate in this project; a missing, expired, foreign or
   changed one is refused before anyone is asked. The act is `pipeline.source.create`, which
   **always asks**, whatever the organization's policy says. The source is then filled in, and it
   is **still a draft**.
3. **Go live.** `state: enabled` is its own act, `pipeline.source.resume`, and asks again.
4. **Feed it.** Upload a file (`POST .../sources/:name/upload?table=...`), deliver to the webhook,
   or let the schedule pull.

## Changing one

`POST .../sources/:name/update` and the `sources_update` tool take exactly **one** change. The act is
derived from the difference between the source as it is and the one you ask for, never from an
argument.

| Change | Act | Risk | Applies |
|---|---|---|---|
| Pause | `pipeline.source.pause` | ELEVATED | Under an ALLOW policy; SENSITIVE and asked if a join reads this source |
| Fewer tables or columns, or a slower schedule | `pipeline.source.narrow` | ELEVATED | Under an ALLOW policy; SENSITIVE and asked if a join reads what is removed (a slower schedule starves every join over the source) |
| More tables or columns, or a faster schedule | `pipeline.source.update` | SENSITIVE | After a person's yes |
| Resume, or go live | `pipeline.source.resume` | SENSITIVE | After a person's yes |
| A lower monthly cap | `pipeline.source_cap.update` | SENSITIVE | Asked: a cap under the month's meter pauses the source, so lowering it is not a free tightening |
| A higher monthly cap | `pipeline.source_cap.update` | SENSITIVE | After a person's yes |
| An earlier `historyFrom` | `pipeline.source.backfill` | SENSITIVE | After a person's yes |
| Delete the source and its rows | `pipeline.source.purge` | irreversible | **Human-only**, `org:admin` |

The purge is `DELETE .../sources/:name` for an organization owner behind a person's session. No
agent, OAuth client or delegated credential reaches it, and the MCP tools have no field for it.
Rotating a webhook source's signing secret is likewise a person's act.

## Units, caps and what happens at the cap

A landed row costs 0.01 units when it **changes**: a row that arrives again unchanged costs nothing,
so a nightly snapshot of a quiet table is nearly free. Every source has a monthly cap in units
(10,000 by default, which is a million changed rows); `usage_get` reports each source's
`unitsMonth`.

A source that reaches its cap **pauses**, with a reason a person can read and a notice
(`source.cap_paused` in the audit log). It does not sample: the over-cap sampling used for
telemetry is never applied to business rows, because a revenue table with every tenth row missing
is a wrong answer, not a cheaper one. Rows already landed are kept, and nothing more lands until a
person raises the cap, or resumes the source once the month has turned over.

## Landing

- **Scrub on write.** Every value is scrubbed at the stored tier before it is hashed or stored.
- **Identity.** A row is identified by its declared primary key. A table with none is identified by
  its content, which means a changed row is a new row; the preview says so.
- **Upload semantics.** An upload is by default the whole table: rows it no longer holds are retired
  (soft-deleted). `replace=false` appends.
- **Months.** A row lives in the monthly partition it first landed in and is updated there.
- **Subjects and accounts.** A subject or account column is removed from the stored data and kept
  as a keyed pseudonym beside it, only while the project's analytics gate is open. With the gate
  shut the link is dropped, never kept raw. Those pseudonyms are what an erasure matches on.
- **Webhooks.** The body is read raw and verified against an HMAC of the exact bytes
  (`x-bucker-signature`) under a secret derived per source. An unknown source, a wrong signature, a
  draft and another source's secret all answer the same 401. The event id is the dedupe key, so a
  replay writes nothing; a stale signed event is dropped. The project is the source's, never the
  body's.
- **Pulls.** Scheduled by a worker pass (`sources_sync`); it honors the upstream's rate limit with
  the upstream's own wait, pages by a stored position, and marks a source `failing` on an error and
  `enabled` again on the next success.

## For agents

| Tool | Does | Relation |
|---|---|---|
| `sources_list` | Lists sources; `include: [preview]` with a `candidate` forecasts one | `project:read`; the preview block needs `analytics:read` |
| `source_propose` | Creates, always pending approval | `pipeline:write` |
| `sources_update` | One change to one source | `pipeline:write` |

All three are **held** behind the SG-059 security review: a token a production path can mint does
not list them. Source, table and column names come from your data, so they are sanitized and
declared untrusted in every answer. No credential appears in any request or response.

## Erasure and export

Rows linked to a subject or an account by a keyed pseudonym are reached by an erasure or an access
request for that pseudonym, in every month and only in the project that holds them. A row whose
table named no subject or account column holds no key and is not reached.
