Log pipeline rules
Choose what share of a project's log records Bucker stores, on the server, without changing any SDK. This page covers the keep, drop and sample rules of the log pipeline: how they behave, how a person or an agent changes them, and what holds whoever asks. Span rules share the same table and the same preview engine and are described in Sampling rules.
What a rule is
A rule says "of the log records matching these filters, keep this share". The filters are the
Explore filters (severity_number, severity, message, environment, any attribute), ANDed, and
the rate runs from 0 to 1. The first enabled rule by position that matches a record decides; a
record no rule matches is kept. The rate says what the rule is:
| Kind | Rate | What it does |
|---|---|---|
| keep | 1 |
Keeps what it matches, and so protects those records from any later rule. |
| drop | 0 |
Drops what it matches. |
| sample | between | Keeps that share of what it matches. |
A rule is in one of three states:
| State | What it does |
|---|---|
| shadow | Evaluated beside the live rules and counts what it would have kept and dropped (shadowKept, shadowDropped). It drops nothing. |
| active | Applies. |
| disabled | Not evaluated. |
A shadow rule is stored switched off for an older image (enabled: false), so a server that predates
the status column can never apply it by mistake. Every rule an agent creates starts in shadow and
lapses in 7 days unless it is renewed (30 in all).
Log metrics, quotas, retention routes and conversions are other cards' (LOG-13, -10, -18, -14). Asking for one is refused, naming the card. The transform kinds are described next.
Transform rules (processing at ingest)
A transform rule rewrites a record instead of choosing whether to keep it. Seven kinds exist, stored
in the same table (kind, with typed parameters in config), each with a filter in the Explore
grammar. There is no expression language, no template, no script and no Grok dialect: a rule can only
do what one of these does, to the fields it names.
| Kind | What it does | Bounds |
|---|---|---|
parse_json |
Reads the body as JSON into attributes. A key the sender already sent wins. | At most 50 keys, values truncated at 1,000 characters |
parse_kv |
Reads key=value pairs from the body. Sender keys win. |
Same |
parse_regex |
One pattern with named groups; each group becomes an attribute. | Pattern at most 200 characters, passes the same ReDoS check an org scrub rule does, runs over at most 8,192 characters |
rename |
Moves an attribute to another key; never onto a key that exists. | Floor keys refused |
remove_fields |
Removes attributes (exact keys or a prefix*). |
At most 20 keys; floor keys refused |
set_severity |
Sets severity from a field or the body's leading token, through a map. | Never lowers a record the floor protects, nor one that arrived at ERROR or worse below ERROR |
route_retention |
Chooses the retention class a record is stored under (LOG-18's choice). | Classes 3 to 365 days; the stage runs here, the governed write is LOG-18's, so the tools refuse it by name |
Order is fixed, and transforms run first: parse_json, parse_kv, parse_regex, rename,
remove_fields, set_severity, route_retention, then the keep, drop and sample rules, then the
quota gate, then the stored-tier scrub. This departs deliberately from OTEP 4738, which keeps and
then transforms: here a sampling rule, a quota or a metric sees the fields a parse rule extracted,
which a Datadog pipeline filter cannot. Within one kind, rules run by position. A stage registry
(registerLogStage(order, kind, fn)) lets the annotate, quota and metric stages attach from their
own modules; two stages may not share an order or a kind.
What the stages promise:
- Inert by default. Behind the
log_stagespillar. With the pillar off, or no transform rule, the batch is the same array and no byte of any row changes. A row withoutconfigis a sampler. - A transform never drops a record and never touches the floor. A rule naming
exception.*, a severity, a trace id or a span id as something to remove or rename is refused at save, and at run time a change that would end a record's floor protection, or re-level a record that arrived at ERROR or worse below ERROR, is discarded for that record and countedfailed. - Fails open. A record a rule cannot process is stored unparsed and counted
failed. A stage that throws or returns the wrong number of records is skipped for that batch. - Counted. Each rule keeps
matched,applied,failedandcpuMicros. A rule whose mean cost of processing the records it SELECTED (not of judging its filter on the rest) exceeds 200 microseconds over 1,000 such records is skipped in that process for five minutes. This is an in-process breaker, not yet the card's persistent move to shadow with a finding. - Marked. A record a rule changed carries
bucker.pipeline, the ids of the rules that did. When the stages run, a sender's ownbucker.pipelineis overwritten on a changed record and removed from every other one. - Shadow. A shadow transform counts what it would do and changes nothing.
Governance: creating or changing a transform is SENSITIVE and asks; no pipeline_tuning rung reaches
it. A change that removes less or matches less is narrower. A regex over its budget, a pattern without
a named group or one that fails the ReDoS check is refused at save, with no request created. A
project may hold at most 50 transform rules.
What is never dropped
The never-drop floor keeps these whatever a rule says, and a rule that matches only them (a filter that forces severity 17 or above) is refused as a no-op:
- records of severity ERROR or worse;
- records carrying an
exception.*attribute; - records linked to an open issue;
- records on a trace that has an error.
The preview refuses a plan that would drop one, with a count, no digest and a narrower plan (for
example severity_number lt 17). Nothing the floor protects is ever discounted by a rate.
Previewing a rule
POST /orgs/{org}/projects/{project}/log-pipeline/preview (and the MCP tool log_pipeline_preview)
replays a rule through the sampler over the last 24 hours of the project's stored log records (at
most 168). The replay reads a bounded sample, newest first, and scales every count to the matching
set, saying so in approximations. It answers:
- the records the rule matches, drops and keeps, and its share of the project's log volume;
- the meter units and dollars saved a day and a month, and the days of the monthly allowance gained;
- the alert rules, saved views, dashboard widgets and saved queries whose match sets change, named, with the number of evaluations that would have changed ("would have changed N evaluations");
- the floor.
An accepted preview returns a digest, valid for 60 minutes and bound to the project. A proposal cites it, and the figures the proposal carries are loaded from that stored preview, never from the caller.
Previewing needs event:read on the project, because it reads stored log records.
Listing the pipeline
GET /orgs/{org}/projects/{project}/log-pipeline/rules (and log_pipeline_rules_list) returns every
stage in order: each rule with its kind, state, rate, filters, expiry, shadow tallies and the hours
they have run, its lifetime live tallies with the records the floor kept, who proposed it and the
approval that let it in. Beside them: the project's log tallies for the last 24 hours and 7 days, the
units the drops saved a day, the consumers of the logs dataset, and the stages other cards own, each
not_available with its owner. Rules keep no per-window counters, so the windows are the project's,
from the outcome ledger.
How a change is governed
A person edits a rule in settings (PATCH /sampling-rules/{id}), which writes the row directly,
bumps its version, keeps the revision it replaced and audits the change as log_pipeline_rule.update
with the filters before and after. An agent goes through the log_pipeline_rule operation family and
nothing else. The act is derived from the before-and-after diff, never from an argument:
| Act | Derived when | Class |
|---|---|---|
log_pipeline_rule.create |
A new rule. It lands in shadow. | ELEVATED |
log_pipeline_rule.update |
The rule removes more (a lower rate, a shadow rule moved live) or matches differently (filters that are not a narrowing of the old ones). | SENSITIVE, always asks |
log_pipeline_rule.narrow |
The rule only does less (a higher rate, more conditions on top of the old filters, any change to a rule still in shadow, a rule taken out of force). | ELEVATED; SENSITIVE when the records it adds exceed 10 % of the monthly allowance |
log_pipeline_rule.delete |
Never by an agent. A person, in settings. | human-only |
A plan carries the version of the project's log rules it was made on (base_version). If the rules
move between the approval and the retry, from any door, the apply is TARGET_CHANGED and spends
nothing. The write must cite a preview digest of the exact rule (at most 60 minutes old, or the
approved plan's own figures on a retry after an approval).
Refused before anyone is asked
Each of these names its cap and leaves no request, no operation and no rule:
- a plan that would drop a floor record;
- a plan that drops more than 50 % of the project's log records;
- a rule that matches only error severities;
- a sixth shadow rule, a twenty-first rule, a name already in force;
- a kind that is another card's, a kind that disagrees with its rate, or a transform whose configuration fails its kind's check (an unbounded or unsafe pattern, a floor key).
Silencing-class
A live drop or sample rule whose preview named an alert rule, saved view, widget or saved query is silencing-class. It lapses seven days after the move that made it live (judged on read, so a stopped worker never leaves it dropping) and the operation states a seven-day TTL. A person confirms it by adopting the rule in settings, which clears the expiry and supersedes the operation.
Verification
At 60 minutes after an apply, and again at 7 days, the share of matched records the rule actually dropped (its own tallies since the apply) is compared with the preview. Within 20 % passes; 20 to 50 % is recorded as drift and asks nobody; above 50 % ends the watch not verified and opens the "keep or revert?" question. An alert rule that read the affected records and reported no data after the apply does the same, whatever the share did. The pass never reverts anything itself.
Reverting
operation_revert runs the stored inverse: a new rule is narrowed to disabled, a change restores the
previous revision the rule itself stored. A restore never re-arms an expiry, and a rule changed by a
person since is left alone.
Configuration as code
bucker.yml has no log pipeline resource. A file apply would be a second door to the same table with
no preview digest and no approval; the resource ships only when its apply runs through the same plan
digest.
What no tool can do
Delete a rule (log_pipeline_rule.delete, a person's act, project write). Raise a scope: the tools
are held behind logs:write until the OAuth 2.1 security review (SG-059).