Sources: business data as evidence
A source brings business data (invoices, accounts, orders) into Bucker as typed tables, so a question about an error or an experiment can be answered against the revenue it touched. It lands on the same seam the error-tracker connectors use, is metered in the same units, and is governed like every other configuration change: proposed by a person or an agent, approved by a person, inert until a second approval turns it on.
Everything on this page is opt-in and inert by default. A deployment with no source configured runs no sync, opens no connection and stores no business row.
What a source is
| Part | What it holds |
|---|---|
| Manifest | A versioned, reviewed description of a kind of source, shipped in the repository. A source records the manifest type and version it was created on and keeps reading that version. |
| Source | A name, a mode, a state (draft, enabled, paused, failing), a schedule, a monthly unit cap and the month's meter. |
| Tables | The tables it lands: primary key, the columns kept, an optional cursor column, and (behind the product-analytics gate) a subject or account column. |
| Runs | One row per sync with a receipt: rows read, inserted, updated, unchanged, retired, rejected by reason, units, and a digest. |
| Rows | The landed rows, scrubbed at the stored tier before anything is hashed or written. |
Built-in manifests
| Type | Label | Modes | Auth | Notes |
|---|---|---|---|---|
upload_csv |
beta | upload |
your session | A CSV file of at most 50 MB through the blob store. |
upload_jsonl |
beta | upload |
your session | JSON Lines, at most 50 MB. |
upload_parquet |
alpha | upload |
your session | Declared and priced; the reader is not built yet, so an upload is refused with that reason rather than half-read. |
object_path |
alpha | object_path, full_refresh, append |
a sealed connection | An S3-compatible prefix. The credential is sealed in a connection (see connections); the source names the connection by id and nothing else. |
inbound_webhook |
beta | webhook |
a per-source signing secret | A signed endpoint with the same replay dedupe the other webhook doors use. |
A manifest cannot carry a host, a URL or a secret; registration refuses one that does. A later
package adds its own manifest with registerManifest (Stripe's is its own file). There is no path
from a request to a new manifest.
Setting one up
- Preview.
POST .../sources/preview(orsources_listwithinclude: [preview]and acandidate) returns the schema it would land (column names, types and null fractions), the row forecast with the basis it came from (the provider's count, one sample page, a stated file size, declared columns, orunknown; a number is never invented), the units, the cap that would pause it, and for a subject or account column an estimated join-key match against this project's subjects. It returns no rows: business rows are person-level data. It also returns apreviewToken, good for 15 minutes. - Propose.
POST .../sourcesor thesource_proposetool, with that token. The token proves the forecast was shown for exactly this candidate in this project; a missing, expired, foreign or changed one is refused before anyone is asked. The act ispipeline.source.create, which always asks, whatever the organization's policy says. The source is then filled in, and it is still a draft. - Go live.
state: enabledis its own act,pipeline.source.resume, and asks again. - Feed it. Upload a file (
POST .../sources/:name/upload?table=...), deliver to the webhook, or let the schedule pull.
Changing one
POST .../sources/:name/update and the sources_update tool take exactly one change. The act is
derived from the difference between the source as it is and the one you ask for, never from an
argument.
| Change | Act | Risk | Applies |
|---|---|---|---|
| Pause | pipeline.source.pause |
ELEVATED | Under an ALLOW policy; SENSITIVE and asked if a join reads this source |
| Fewer tables or columns, or a slower schedule | pipeline.source.narrow |
ELEVATED | Under an ALLOW policy; SENSITIVE and asked if a join reads what is removed (a slower schedule starves every join over the source) |
| More tables or columns, or a faster schedule | pipeline.source.update |
SENSITIVE | After a person's yes |
| Resume, or go live | pipeline.source.resume |
SENSITIVE | After a person's yes |
| A lower monthly cap | pipeline.source_cap.update |
SENSITIVE | Asked: a cap under the month's meter pauses the source, so lowering it is not a free tightening |
| A higher monthly cap | pipeline.source_cap.update |
SENSITIVE | After a person's yes |
An earlier historyFrom |
pipeline.source.backfill |
SENSITIVE | After a person's yes |
| Delete the source and its rows | pipeline.source.purge |
irreversible | Human-only, org:admin |
The purge is DELETE .../sources/:name for an organization owner behind a person's session. No
agent, OAuth client or delegated credential reaches it, and the MCP tools have no field for it.
Rotating a webhook source's signing secret is likewise a person's act.
Units, caps and what happens at the cap
A landed row costs 0.01 units when it changes: a row that arrives again unchanged costs nothing,
so a nightly snapshot of a quiet table is nearly free. Every source has a monthly cap in units
(10,000 by default, which is a million changed rows); usage_get reports each source's
unitsMonth.
A source that reaches its cap pauses, with a reason a person can read and a notice
(source.cap_paused in the audit log). It does not sample: the over-cap sampling used for
telemetry is never applied to business rows, because a revenue table with every tenth row missing
is a wrong answer, not a cheaper one. Rows already landed are kept, and nothing more lands until a
person raises the cap, or resumes the source once the month has turned over.
Landing
- Scrub on write. Every value is scrubbed at the stored tier before it is hashed or stored.
- Identity. A row is identified by its declared primary key. A table with none is identified by its content, which means a changed row is a new row; the preview says so.
- Upload semantics. An upload is by default the whole table: rows it no longer holds are retired
(soft-deleted).
replace=falseappends. - Months. A row lives in the monthly partition it first landed in and is updated there.
- Subjects and accounts. A subject or account column is removed from the stored data and kept as a keyed pseudonym beside it, only while the project's analytics gate is open. With the gate shut the link is dropped, never kept raw. Those pseudonyms are what an erasure matches on.
- Webhooks. The body is read raw and verified against an HMAC of the exact bytes
(
x-bucker-signature) under a secret derived per source. An unknown source, a wrong signature, a draft and another source's secret all answer the same 401. The event id is the dedupe key, so a replay writes nothing; a stale signed event is dropped. The project is the source's, never the body's. - Pulls. Scheduled by a worker pass (
sources_sync); it honors the upstream's rate limit with the upstream's own wait, pages by a stored position, and marks a sourcefailingon an error andenabledagain on the next success.
For agents
| Tool | Does | Relation |
|---|---|---|
sources_list |
Lists sources; include: [preview] with a candidate forecasts one |
project:read; the preview block needs analytics:read |
source_propose |
Creates, always pending approval | pipeline:write |
sources_update |
One change to one source | pipeline:write |
All three are held behind the SG-059 security review: a token a production path can mint does not list them. Source, table and column names come from your data, so they are sanitized and declared untrusted in every answer. No credential appears in any request or response.
Erasure and export
Rows linked to a subject or an account by a keyed pseudonym are reached by an erasure or an access request for that pseudonym, in every month and only in the project that holds them. A row whose table named no subject or account column holds no key and is not reached.