Skip to content
bucker

Data quality over your exports

A warehouse sink ships your telemetry to a place you own. A stream that quietly stops, thins out, changes shape or writes a row twice is worse than one that fails loudly, so Bucker watches its own export streams with five standing checks. They cover Bucker's own export streams only: error essences, issue rollups, release-health rollups and ingest outcomes. Monitors on your own database tables are a separate feature (they need a connected database, which this release does not have), and the tools below say so rather than guess.

The five checks

Each check belongs to one sink and one stream. Lower thresholds are always stricter.

Check What it judges Threshold (default) States
freshness How old the oldest row still waiting behind the export cursor is, against the export interval (five minutes). Intervals of backlog before it is stale (2). Failing at three times that (6). healthy, warning (stale), failing, not_run before the first export attempt.
volume Rows exported in the last complete hour against the learned median and spread for that hour of the week. Robust z (6). Failing at twice that. not_run until four weeks of history exist: never a verdict before.
schema_version Every record matches the stream's declared, versioned record schema. none healthy or failing.
uniqueness (recordId, watermark) appears once per stream, every batch starts strictly after the cursor it was read from, and the cursor never moves back. none healthy or failing.
completeness Rows exported against rows eligible between the two cursors of a batch (one count query per batch). Share of missing rows (0.01). healthy or failing.

A check that could not run is erroring (and says basis: evaluation_failed); a check never reads healthy because it failed to look.

The schema tag

Every object an export writes carries bucker-schema: <STREAM>@<n> on every JSON line, for example ISSUE_ROLLUPS@1. The version is raised whenever the shape of a record changes, in the same change that changes it. A record with a field the declared schema does not name fails schema_version: that is how a silent change of contract is caught before a loader downstream breaks on it.

Turning checks on

Checks are off until two things are true:

  1. The sink names the project that receives its findings (alertProjectId, set with PATCH /orgs/{org}/warehouse/sinks/{sinkId} by an organization admin; null turns every check on that sink off). A sink belongs to the organization and an issue to a project, so the findings need a project to open in. The project must be your own.
  2. A check is turned on with a governed change (below).

A sink with no alert project cannot have a check changed, and is refused with that remedy. Sinks without checks cost the exporter nothing.

What a failing check does

It opens an ordinary issue in the sink's alert project, with exception.type DataQualityFailed, fingerprinted on the sink, the stream and the check, so the one alert engine routes it like any other. The issue opens when a check enters warning or failing, or climbs from warning to failing, never on every run: a stalled export is an issue within two export intervals, and the same issue gets another occurrence when it becomes failing. A stale check opens at level warning; a failing one opens at the check's severity (error by default). The text of the issue is Bucker's own words over a closed set: the sink's name and your data are never put in it.

Runs are stored for 30 days: when a state changes, and at most once an hour otherwise.

Reading the state

  • Dashboard: Warehouse, under the sink list.
  • REST: GET /orgs/{org}/data-quality (optionally ?sinkId=).
  • MCP: data_quality_status_get (project:read, the data toolset). By default only the checks that are on are listed; pass all to list the others. include of tables, check_sql or monitor_backtest answers not_available and needs db:read on top.

A sink's alert project, and the issue a check opened there, are shown only to a caller who can read that project. The sink's name is text a member typed: the API and the tool label it untrusted, and the tool fences it before an agent reads it.

Changing a check

POST /orgs/{org}/data-quality/checks and the MCP tool data_quality_checks_update change one check: turn it on or off (enabled), change its threshold, or change the level of its issue (severity). It is a governed write (data_quality_check.update, risk class SENSITIVE, asked for by default):

  • The act is read from the difference, never from an argument. Turning a check on, a lower threshold or a higher severity is a tightening. Turning a check off is silencing; a higher threshold or a lower severity is a loosening.
  • Silencing and loosening changes carry a 30-day TTL: when it ends, the stored inverse puts the previous setting back (through the sweeper), unless somebody has changed the check since.
  • Every plan shows a 30-day replay: what the stored runs of the last thirty days would have raised under the new setting, against what they did raise. The window ends at the start of the current UTC day, so the figure an approver reads is the figure the retry re-derives.
  • A first call answers 202 input_required with the approval request; a person with the project's configure_data approver rights decides it, and the retry (approvalRequestId) applies once, against the revision that was asked on. A check someone edited in between is refused as TARGET_CHANGED and the yes is not spent.
  • An organization's own approval-policy row can let a change apply without a person. It applies to the action, so scope it with a condition if you want a tightening to apply while a loosening still asks. Review mode never lifts these changes: a check on an export stream states no environment, and an unstated environment is production.
  • Every apply writes a signed receipt, and operation_revert can undo it through the stored inverse.

The export gate (data_gate.enable) and monitors on your own tables (data_monitor.disable) are not reachable here: they are other features with their own approval, and the request is refused by name.

What this does not do

  • It does not block an export. A failing check reports; the one thing the exporter itself enforces, with checks on or off, is that the cursor only moves forward: a batch that ends at or before the cursor it started from is refused, and the checkpoint stays where it was.
  • It does not use a model. Nothing in this feature phrases anything with one.
  • It does not read your warehouse. Every judgement is over what Bucker exported and what Bucker holds.