Sampling rules
Choose what share of a project's spans Bucker stores, on the server, without changing any SDK. This page covers span sampling rules: how they behave, how a person or an agent changes them, and what holds whoever asks.
Log pipeline rules share the same table and the same preview engine and are described in the logs documentation.
What a rule is
A rule says "of the spans matching these filters, keep this share". The filters are the
Explore filters (resource.service.name, span.name, any attribute), ANDed, and the rate
runs from 0 (drop every match) to 1 (keep every match). The first rule that matches a
span decides; a span no rule matches is kept.
A rule is in one of three states:
| State | What it does |
|---|---|
| shadow | Evaluated beside the live rules and counts what it would have kept and dropped (shadowKept, shadowDropped). It drops nothing. |
| active | Applies: matching traces are kept at the rate. |
| disabled | Not evaluated. |
A shadow rule is stored switched off for an older image (enabled: false), so a server that
predates the status column can never apply it by mistake.
What is never sampled
The sampler keeps these whole, whatever a rule says, and a rule that matches only these is refused as a no-op rather than approved and then ignored:
- error spans, spans carrying an
exception.*attribute, and every other span of their traces; - traces whose root returned a 5xx;
- traces linked to an open issue's occurrence;
- APM-19's sampling-budget floor (reserved: nothing registers it yet).
The promise is that the cross-batch gap is reduced, not closed. The floor decides per
request from what the request and a bounded lookback of recent error traces say. A trace
whose error arrives in a later request than its other spans can have been decided before
the error was known. The preview reports that as crossBatchExposure.
Previewing a rule
POST /orgs/{org}/projects/{project}/sampling-rules/preview (and the MCP tool
sampling_rules_preview) replays a rule through the sampler over the last 24 hours of the
project's stored spans (at most 168). It answers:
- the rows the rule matches and the rows it would drop and keep (counted, not extrapolated);
- the share of the project's volume, the meter units and dollars saved, and the days of the monthly allowance gained;
- the alert rules, saved views and dashboards whose match sets change, named, with the number of evaluations that would have changed;
- the never-drop floor: a rule that would drop an error span is refused with a count and no
digest, and says how to narrow it (
span.status neq error).
An accepted preview returns a digest, valid for 30 minutes and bound to the project. A proposal cites it.
Previewing needs event:read on the project, because it reads stored spans.
Changing rules
In settings
A person with project write changes rules in Settings, Projects, Sampling rules, or over
POST|PATCH|DELETE .../sampling-rules. A rule made there is active (as it always was) unless
the request says status: "shadow", may carry expiresInDays, and every change to a governed
field bumps the rule's version and stores the revision it replaced.
Deleting a rule is a human act in settings. No tool can ask for it.
As an agent
Two governed tools, both held until the SG-059 review (they need apm:write):
sampling_rules_createproposes a new rule. It lands in shadow, so the act is derived ELEVATED: it changes no data.sampling_rules_updatechanges a rule's rate, filters, position or renewal, or moves a shadow rule to active. The engine derives the act from the diff, never from an argument. A change to a shadow rule is ELEVATED. A change to a live rule, and the move to active, are SENSITIVE and ask a person. A rate that rises keeps more data and costs more, so it asks too.
Each write must cite the previewDigest of a preview of the rule as it would be, and repeat
the preview's arguments (the reason included): they are the plan. The figures the plan carries
are loaded from the stored preview, never from the caller.
A previous revision is restored by operation_revert and by the sweeper at expiry, from the
revision the rule itself stored. It is not a tool.
The caps
Computed on the server and refused before any approval opens, each naming the cap:
| Cap | Limit |
|---|---|
| Services changed by one proposal | at most 3 (a filter set that names none changes every service) |
| Share of the project's span volume changed by one proposal | at most 25 % |
| A service's keep rate within 24 hours | falls at most 4x |
| Approved changes to a service's live sampling in a day | at most 2 |
| Shadow rules in a project | at most 5 |
| Shadow before a move to active | at least 72 hours of tallies |
| Tallies against the preview the move cites | within 25 %, or preview the rule again |
They are constants, not organisation settings: a cap an agent's own organisation could widen would be a cap in name only.
Expiry
Every rule an agent makes carries expiresAt: 7 days, renewable (renew: true) up to 30 in
all. Expiry is judged on read, so a stopped worker never leaves a temporary rule dropping
data. The operations sweeper's end-of-TTL restore takes the rule out of force, or back to the
revision it replaced, and tells the person who approved it (window_ended). A person adopts a
rule in settings (clear its expiry) or lets it end.
Verification
After a live change, once its 60-minute window has closed, the verification compares what the
rule dropped of the spans it matched since the apply (its own live tallies, measured from the
base the apply recorded) with the dropped share the cited preview stated. Within 10 % of the
preview, or within the sampling noise of the spans seen, passes; 10 to 25 % is recorded as drift
and asks nobody; above 25 % opens the ask "keep or revert?" and sends the receipt_failed
notice. A loss of the protected classes (error spans and error traces, the hour after the apply
against the hour before) does the same whatever the effect did, when the fall is larger than both
5 % and the hour-to-hour noise of the counts (at least 20 before, and more than three standard
deviations). Fewer than 100 spans matched since the apply is not_enough_data; a rule another
change owns since (a renewal, a person's edit in settings) is superseded and nobody is asked.
The check never reverts by itself.
Tables and fields
sampling_rules carries status, kind, expiresAt, proposedByPrincipalId,
approvalRequestId, previousRevision, previewDigest, shadowKept, shadowDropped,
operationId and version. previousRevision holds the definition the last governed change
replaced (null for a create) and the start of the rule's shadow clock; it is written by every
door and never recomputed, because it is the inverse. sampling_rule_previews holds the
digests.