Skip to content
bucker

Instrumentation health

"Is capture working?" is one read. Bucker checks every producer that talks to a project (Bucker SDKs, Sentry SDKs and OpenTelemetry exporters) against data it already holds: the hourly SDK inventory, the drop ledger and the symbolication totals. A release that breaks capture is a finding within minutes.

The checks are code, versioned as instrumentation_checks_v1, each with a published threshold. No model is in this path: a finding is a count over a stated window, the producer concerned, and one named fix.

The inventory

GET /orgs/{org}/projects/{project}/sdks and the MCP tool sdk_inventory_get list every producer of the last 7 days with its version, family (bucker, sentry, otel), what it sent by item family, what the accept edge dropped of it, when it was first and last seen, and the hints from one rule table.

An OpenTelemetry exporter is listed as otlp:<name>, from telemetry.sdk.name, .language and .version (or telemetry.distro.name and .version when the exporter states a distribution). Each resource of an export is one producer, so a Collector forwarding several services shows each of them. Counts are approximate: they are collected without slowing ingest down. OTLP metrics exports are not inventoried yet; an exporter that sends only metrics appears when it also sends traces or logs.

No SDK reports its effective configuration yet and the hourly rollup keeps no capture warnings, so sdk_inventory_get says so under unavailable instead of showing an invented value.

The checks

Check Fails or warns when Fix
silence A producer that sent in the last 7 days has sent nothing for 30 minutes while its own history for this hour (median and MAD of the same hour of the week, or of the day when under three weeks exist) says to expect at least 5 envelopes in that time. A producer that is usually quiet is never silent. Fewer than three hours of history answer unknown. Check that the service behind the producer is running and still configured with this project's DSN or OTLP endpoint.
below_floor A version below the deployment's declared SDK floor sent. Warns until the floor's sunset date, then fails. Upgrade to the floor's minimum version.
outdated A producer that sent in the last 24 hours runs a version two minors (or a major) or 120 days behind the newest version in sdk-versions.json. Warns. Upgrade to the newest published version.
drops_rising In the last hour, items dropped for one reason (a client discard or a server refusal) exceed 2 percent of all items, with at least 100 items. Internal SDK errors and OTLP refusals are their own checks. Read the drop reasons; change the capture rule or the SDK option that causes them.
internal_sdk_error Any client report in the last hour carries the reason internal_sdk_error. There is no minimum. Update the SDK; report a defect if it persists.
otlp_refused An OTLP door refused a body (too large once decompressed, or undecodable): fails. Metric series were folded or dropped past the cardinality caps: warns. Lower the exporter's batch size; drop high-cardinality metric attributes.
sourcemaps_unresolved More than 50 percent of stack frames in the last hour could not be symbolicated (warns above 10 percent), with at least 20 frames. Upload the build's source maps with the bundler plugin.
config_drift Instances of one service report different effective-configuration digests. Compare the reported configuration and roll the odd instance back.
consent_pending_overflow, tunnel_bypass, csp_blocked Not measured yet. Each answers unknown until its card ships.

A state is pass, warn, fail or unknown. unknown is never a pass: it carries a reason, not_enough_data (too little history or volume) or source_not_built (nothing reports it yet). A finding that rests on client reports alone (internal_sdk_error, a drops_rising made only of client discards) carries basis: client_reported, so a proposal built on it runs at the input-provenance rung cap.

The standing finding for each (project, check, producer) is rewritten on every evaluation; it is not a history. A producer's key is the SDK name for silence and name@version for the version-scoped checks.

The issue

A check that has been failing for 30 minutes opens one instrumentation_gap issue (silence is failing for 30 minutes by definition, so it opens the moment it fails). The issue resolves itself the first evaluation the check passes again. A person who resolves it while the check is still failing is not overruled: it reopens only when the check recovers and fails again. The issue title is Bucker's own sentence; the producer name rides in a tag and is never a title.

Each check is a registered detector (instrumentation_<check>), and like every new detector it starts in shadow: findings are recorded and readable at once, and the issue opens only once a version is live (see Detectors).

Muting a check

A mute makes a check fire less, so it is governed. POST /orgs/{org}/projects/{project}/instrumentation/checks and the MCP tool instrumentation_checks_update set the desired mute state of one check for one producer (or every producer), with an end.

  • instrumentation_check.mute: SENSITIVE, asked for (an approval bound to the plan digest, decided by a person), at most 30 days (7 days is the registry's default), never for internal_sdk_error on a producer that sent in the last 24 hours.
  • instrumentation_check.unmute: ELEVATED, allowed. It ends a silence, so checks fire more, never less.

The act is derived from the change, not from an argument. A mute applies only to the mute set it was approved against: if any door changes that set between the approval and the retry, the request closes as TARGET_CHANGED and the yes is not spent. A mute ends by itself; an expired mute is not in force even if nothing has swept it. In production the tool is held until the release review (capture:write).

Reading it

Door What it returns
GET …/instrumentation/health and instrumentation_health_get The standing findings, worst first, with counts, window, producer, fix, and each muted check with its end, who muted it and why. Filters check and producer.
GET …/instrumentation/checks The published check set with thresholds and fixes, and the mutes in force.
sdk_inventory_get The producers.

SDK names, versions and mute reasons are client- or member-authored text: tools return them capped, scrubbed and listed under trust.untrustedFields.

Releasing SDK versions

sdk-versions.json (beside instrumentation-checks.ts) lists the versions each SDK has published. A release adds its line in the same change that publishes it; a version missing from the list is judged by its number alone. The file also mirrors the SDK floor register, and a test holds the two equal.