# Instrumentation health

"Is capture working?" is one read. Bucker checks every producer that talks to a project (Bucker SDKs,
Sentry SDKs and OpenTelemetry exporters) against data it already holds: the hourly SDK inventory, the drop
ledger and the symbolication totals. A release that breaks capture is a finding within minutes.

The checks are code, versioned as `instrumentation_checks_v1`, each with a published threshold. No model is in
this path: a finding is a count over a stated window, the producer concerned, and **one** named fix.

## The inventory

`GET /orgs/{org}/projects/{project}/sdks` and the MCP tool `sdk_inventory_get` list every producer of the last
7 days with its version, family (`bucker`, `sentry`, `otel`), what it sent by item family, what the accept edge
dropped of it, when it was first and last seen, and the hints from one rule table.

An OpenTelemetry exporter is listed as `otlp:<name>`, from `telemetry.sdk.name`, `.language` and `.version`
(or `telemetry.distro.name` and `.version` when the exporter states a distribution). Each resource of an export
is one producer, so a Collector forwarding several services shows each of them. Counts are approximate: they are
collected without slowing ingest down. OTLP **metrics** exports are not inventoried yet; an exporter that sends
only metrics appears when it also sends traces or logs.

No SDK reports its effective configuration yet and the hourly rollup keeps no capture warnings, so
`sdk_inventory_get` says so under `unavailable` instead of showing an invented value.

## The checks

| Check | Fails or warns when | Fix |
| --- | --- | --- |
| `silence` | A producer that sent in the last 7 days has sent nothing for **30 minutes** while its own history for this hour (median and MAD of the same hour of the week, or of the day when under three weeks exist) says to expect at least **5** envelopes in that time. A producer that is usually quiet is never silent. Fewer than three hours of history answer `unknown`. | Check that the service behind the producer is running and still configured with this project's DSN or OTLP endpoint. |
| `below_floor` | A version below the deployment's declared SDK floor sent. Warns until the floor's sunset date, then fails. | Upgrade to the floor's minimum version. |
| `outdated` | A producer that sent in the last 24 hours runs a version **two minors** (or a major) or **120 days** behind the newest version in `sdk-versions.json`. Warns. | Upgrade to the newest published version. |
| `drops_rising` | In the last hour, items dropped for one reason (a client discard or a server refusal) exceed **2 percent** of all items, with at least 100 items. Internal SDK errors and OTLP refusals are their own checks. | Read the drop reasons; change the capture rule or the SDK option that causes them. |
| `internal_sdk_error` | Any client report in the last hour carries the reason `internal_sdk_error`. There is no minimum. | Update the SDK; report a defect if it persists. |
| `otlp_refused` | An OTLP door refused a body (too large once decompressed, or undecodable): fails. Metric series were folded or dropped past the cardinality caps: warns. | Lower the exporter's batch size; drop high-cardinality metric attributes. |
| `sourcemaps_unresolved` | More than **50 percent** of stack frames in the last hour could not be symbolicated (warns above 10 percent), with at least 20 frames. | Upload the build's source maps with the bundler plugin. |
| `config_drift` | Instances of one service report different effective-configuration digests. | Compare the reported configuration and roll the odd instance back. |
| `consent_pending_overflow`, `tunnel_bypass`, `csp_blocked` | Not measured yet. | Each answers `unknown` until its card ships. |

A state is `pass`, `warn`, `fail` or `unknown`. `unknown` is never a pass: it carries a reason,
`not_enough_data` (too little history or volume) or `source_not_built` (nothing reports it yet). A finding that
rests on client reports alone (`internal_sdk_error`, a `drops_rising` made only of client discards) carries
`basis: client_reported`, so a proposal built on it runs at the input-provenance rung cap.

The standing finding for each (project, check, producer) is rewritten on every evaluation; it is not a history.
A producer's key is the SDK name for `silence` and `name@version` for the version-scoped checks.

## The issue

A check that has been **failing for 30 minutes** opens one `instrumentation_gap` issue (`silence` is failing for
30 minutes by definition, so it opens the moment it fails). The issue resolves itself the first evaluation the
check passes again. A person who resolves it while the check is still failing is not overruled: it reopens only
when the check recovers and fails again. The issue title is Bucker's own sentence; the producer name rides in a
tag and is never a title.

Each check is a registered detector (`instrumentation_<check>`), and like every new detector it starts in
**shadow**: findings are recorded and readable at once, and the issue opens only once a version is live (see
[Detectors](detectors.md)).

## Muting a check

A mute makes a check fire less, so it is governed. `POST /orgs/{org}/projects/{project}/instrumentation/checks`
and the MCP tool `instrumentation_checks_update` set the **desired** mute state of one check for one producer
(or every producer), with an end.

- `instrumentation_check.mute`: SENSITIVE, asked for (an approval bound to the plan digest, decided by a
  person), at most 30 days (7 days is the registry's default), never for `internal_sdk_error` on a producer that
  sent in the last 24 hours.
- `instrumentation_check.unmute`: ELEVATED, allowed. It ends a silence, so checks fire more, never less.

The act is derived from the change, not from an argument. A mute applies only to the mute set it was approved
against: if any door changes that set between the approval and the retry, the request closes as
`TARGET_CHANGED` and the yes is not spent. A mute ends by itself; an expired mute is not in force even if
nothing has swept it. In production the tool is held until the release review (`capture:write`).

## Reading it

| Door | What it returns |
| --- | --- |
| `GET …/instrumentation/health` and `instrumentation_health_get` | The standing findings, worst first, with counts, window, producer, fix, and each muted check with its end, who muted it and why. Filters `check` and `producer`. |
| `GET …/instrumentation/checks` | The published check set with thresholds and fixes, and the mutes in force. |
| `sdk_inventory_get` | The producers. |

SDK names, versions and mute reasons are client- or member-authored text: tools return them capped, scrubbed and
listed under `trust.untrustedFields`.

## Releasing SDK versions

`sdk-versions.json` (beside `instrumentation-checks.ts`) lists the versions each SDK has published. A release
adds its line in the same change that publishes it; a version missing from the list is judged by its number
alone. The file also mirrors the SDK floor register, and a test holds the two equal.
