Skip to content
bucker

Sampling rules

Choose what share of a project's spans Bucker stores, on the server, without changing any SDK. This page covers span sampling rules: how they behave, how a person or an agent changes them, and what holds whoever asks.

Log pipeline rules share the same table and the same preview engine and are described in the logs documentation.

What a rule is

A rule says "of the spans matching these filters, keep this share". The filters are the Explore filters (resource.service.name, span.name, any attribute), ANDed, and the rate runs from 0 (drop every match) to 1 (keep every match). The first rule that matches a span decides; a span no rule matches is kept.

A rule is in one of three states:

State What it does
shadow Evaluated beside the live rules and counts what it would have kept and dropped (shadowKept, shadowDropped). It drops nothing.
active Applies: matching traces are kept at the rate.
disabled Not evaluated.

A shadow rule is stored switched off for an older image (enabled: false), so a server that predates the status column can never apply it by mistake.

What is never sampled

The sampler keeps these whole, whatever a rule says, and a rule that matches only these is refused as a no-op rather than approved and then ignored:

  • error spans, spans carrying an exception.* attribute, and every other span of their traces;
  • traces whose root returned a 5xx;
  • traces linked to an open issue's occurrence;
  • APM-19's sampling-budget floor (reserved: nothing registers it yet).

The promise is that the cross-batch gap is reduced, not closed. The floor decides per request from what the request and a bounded lookback of recent error traces say. A trace whose error arrives in a later request than its other spans can have been decided before the error was known. The preview reports that as crossBatchExposure.

Previewing a rule

POST /orgs/{org}/projects/{project}/sampling-rules/preview (and the MCP tool sampling_rules_preview) replays a rule through the sampler over the last 24 hours of the project's stored spans (at most 168). It answers:

  • the rows the rule matches and the rows it would drop and keep (counted, not extrapolated);
  • the share of the project's volume, the meter units and dollars saved, and the days of the monthly allowance gained;
  • the alert rules, saved views and dashboards whose match sets change, named, with the number of evaluations that would have changed;
  • the never-drop floor: a rule that would drop an error span is refused with a count and no digest, and says how to narrow it (span.status neq error).

An accepted preview returns a digest, valid for 30 minutes and bound to the project. A proposal cites it.

Previewing needs event:read on the project, because it reads stored spans.

Changing rules

In settings

A person with project write changes rules in Settings, Projects, Sampling rules, or over POST|PATCH|DELETE .../sampling-rules. A rule made there is active (as it always was) unless the request says status: "shadow", may carry expiresInDays, and every change to a governed field bumps the rule's version and stores the revision it replaced.

Deleting a rule is a human act in settings. No tool can ask for it.

As an agent

Two governed tools, both held until the SG-059 review (they need apm:write):

  • sampling_rules_create proposes a new rule. It lands in shadow, so the act is derived ELEVATED: it changes no data.
  • sampling_rules_update changes a rule's rate, filters, position or renewal, or moves a shadow rule to active. The engine derives the act from the diff, never from an argument. A change to a shadow rule is ELEVATED. A change to a live rule, and the move to active, are SENSITIVE and ask a person. A rate that rises keeps more data and costs more, so it asks too.

Each write must cite the previewDigest of a preview of the rule as it would be, and repeat the preview's arguments (the reason included): they are the plan. The figures the plan carries are loaded from the stored preview, never from the caller.

A previous revision is restored by operation_revert and by the sweeper at expiry, from the revision the rule itself stored. It is not a tool.

The caps

Computed on the server and refused before any approval opens, each naming the cap:

Cap Limit
Services changed by one proposal at most 3 (a filter set that names none changes every service)
Share of the project's span volume changed by one proposal at most 25 %
A service's keep rate within 24 hours falls at most 4x
Approved changes to a service's live sampling in a day at most 2
Shadow rules in a project at most 5
Shadow before a move to active at least 72 hours of tallies
Tallies against the preview the move cites within 25 %, or preview the rule again

They are constants, not organisation settings: a cap an agent's own organisation could widen would be a cap in name only.

Expiry

Every rule an agent makes carries expiresAt: 7 days, renewable (renew: true) up to 30 in all. Expiry is judged on read, so a stopped worker never leaves a temporary rule dropping data. The operations sweeper's end-of-TTL restore takes the rule out of force, or back to the revision it replaced, and tells the person who approved it (window_ended). A person adopts a rule in settings (clear its expiry) or lets it end.

Verification

After a live change, once its 60-minute window has closed, the verification compares what the rule dropped of the spans it matched since the apply (its own live tallies, measured from the base the apply recorded) with the dropped share the cited preview stated. Within 10 % of the preview, or within the sampling noise of the spans seen, passes; 10 to 25 % is recorded as drift and asks nobody; above 25 % opens the ask "keep or revert?" and sends the receipt_failed notice. A loss of the protected classes (error spans and error traces, the hour after the apply against the hour before) does the same whatever the effect did, when the fall is larger than both 5 % and the hour-to-hour noise of the counts (at least 20 before, and more than three standard deviations). Fewer than 100 spans matched since the apply is not_enough_data; a rule another change owns since (a renewal, a person's edit in settings) is superseded and nobody is asked. The check never reverts by itself.

Tables and fields

sampling_rules carries status, kind, expiresAt, proposedByPrincipalId, approvalRequestId, previousRevision, previewDigest, shadowKept, shadowDropped, operationId and version. previousRevision holds the definition the last governed change replaced (null for a create) and the start of the rule's shadow clock; it is written by every door and never recomputed, because it is the inverse. sampling_rule_previews holds the digests.