Skip to content
bucker

Log pipeline rules

Choose what share of a project's log records Bucker stores, on the server, without changing any SDK. This page covers the keep, drop and sample rules of the log pipeline: how they behave, how a person or an agent changes them, and what holds whoever asks. Span rules share the same table and the same preview engine and are described in Sampling rules.

What a rule is

A rule says "of the log records matching these filters, keep this share". The filters are the Explore filters (severity_number, severity, message, environment, any attribute), ANDed, and the rate runs from 0 to 1. The first enabled rule by position that matches a record decides; a record no rule matches is kept. The rate says what the rule is:

Kind Rate What it does
keep 1 Keeps what it matches, and so protects those records from any later rule.
drop 0 Drops what it matches.
sample between Keeps that share of what it matches.

A rule is in one of three states:

State What it does
shadow Evaluated beside the live rules and counts what it would have kept and dropped (shadowKept, shadowDropped). It drops nothing.
active Applies.
disabled Not evaluated.

A shadow rule is stored switched off for an older image (enabled: false), so a server that predates the status column can never apply it by mistake. Every rule an agent creates starts in shadow and lapses in 7 days unless it is renewed (30 in all).

Log metrics, quotas, retention routes and conversions are other cards' (LOG-13, -10, -18, -14). Asking for one is refused, naming the card. The transform kinds are described next.

Transform rules (processing at ingest)

A transform rule rewrites a record instead of choosing whether to keep it. Seven kinds exist, stored in the same table (kind, with typed parameters in config), each with a filter in the Explore grammar. There is no expression language, no template, no script and no Grok dialect: a rule can only do what one of these does, to the fields it names.

Kind What it does Bounds
parse_json Reads the body as JSON into attributes. A key the sender already sent wins. At most 50 keys, values truncated at 1,000 characters
parse_kv Reads key=value pairs from the body. Sender keys win. Same
parse_regex One pattern with named groups; each group becomes an attribute. Pattern at most 200 characters, passes the same ReDoS check an org scrub rule does, runs over at most 8,192 characters
rename Moves an attribute to another key; never onto a key that exists. Floor keys refused
remove_fields Removes attributes (exact keys or a prefix*). At most 20 keys; floor keys refused
set_severity Sets severity from a field or the body's leading token, through a map. Never lowers a record the floor protects, nor one that arrived at ERROR or worse below ERROR
route_retention Chooses the retention class a record is stored under (LOG-18's choice). Classes 3 to 365 days; the stage runs here, the governed write is LOG-18's, so the tools refuse it by name

Order is fixed, and transforms run first: parse_json, parse_kv, parse_regex, rename, remove_fields, set_severity, route_retention, then the keep, drop and sample rules, then the quota gate, then the stored-tier scrub. This departs deliberately from OTEP 4738, which keeps and then transforms: here a sampling rule, a quota or a metric sees the fields a parse rule extracted, which a Datadog pipeline filter cannot. Within one kind, rules run by position. A stage registry (registerLogStage(order, kind, fn)) lets the annotate, quota and metric stages attach from their own modules; two stages may not share an order or a kind.

What the stages promise:

  • Inert by default. Behind the log_stages pillar. With the pillar off, or no transform rule, the batch is the same array and no byte of any row changes. A row without config is a sampler.
  • A transform never drops a record and never touches the floor. A rule naming exception.*, a severity, a trace id or a span id as something to remove or rename is refused at save, and at run time a change that would end a record's floor protection, or re-level a record that arrived at ERROR or worse below ERROR, is discarded for that record and counted failed.
  • Fails open. A record a rule cannot process is stored unparsed and counted failed. A stage that throws or returns the wrong number of records is skipped for that batch.
  • Counted. Each rule keeps matched, applied, failed and cpuMicros. A rule whose mean cost of processing the records it SELECTED (not of judging its filter on the rest) exceeds 200 microseconds over 1,000 such records is skipped in that process for five minutes. This is an in-process breaker, not yet the card's persistent move to shadow with a finding.
  • Marked. A record a rule changed carries bucker.pipeline, the ids of the rules that did. When the stages run, a sender's own bucker.pipeline is overwritten on a changed record and removed from every other one.
  • Shadow. A shadow transform counts what it would do and changes nothing.

Governance: creating or changing a transform is SENSITIVE and asks; no pipeline_tuning rung reaches it. A change that removes less or matches less is narrower. A regex over its budget, a pattern without a named group or one that fails the ReDoS check is refused at save, with no request created. A project may hold at most 50 transform rules.

What is never dropped

The never-drop floor keeps these whatever a rule says, and a rule that matches only them (a filter that forces severity 17 or above) is refused as a no-op:

  • records of severity ERROR or worse;
  • records carrying an exception.* attribute;
  • records linked to an open issue;
  • records on a trace that has an error.

The preview refuses a plan that would drop one, with a count, no digest and a narrower plan (for example severity_number lt 17). Nothing the floor protects is ever discounted by a rate.

Previewing a rule

POST /orgs/{org}/projects/{project}/log-pipeline/preview (and the MCP tool log_pipeline_preview) replays a rule through the sampler over the last 24 hours of the project's stored log records (at most 168). The replay reads a bounded sample, newest first, and scales every count to the matching set, saying so in approximations. It answers:

  • the records the rule matches, drops and keeps, and its share of the project's log volume;
  • the meter units and dollars saved a day and a month, and the days of the monthly allowance gained;
  • the alert rules, saved views, dashboard widgets and saved queries whose match sets change, named, with the number of evaluations that would have changed ("would have changed N evaluations");
  • the floor.

An accepted preview returns a digest, valid for 60 minutes and bound to the project. A proposal cites it, and the figures the proposal carries are loaded from that stored preview, never from the caller.

Previewing needs event:read on the project, because it reads stored log records.

Listing the pipeline

GET /orgs/{org}/projects/{project}/log-pipeline/rules (and log_pipeline_rules_list) returns every stage in order: each rule with its kind, state, rate, filters, expiry, shadow tallies and the hours they have run, its lifetime live tallies with the records the floor kept, who proposed it and the approval that let it in. Beside them: the project's log tallies for the last 24 hours and 7 days, the units the drops saved a day, the consumers of the logs dataset, and the stages other cards own, each not_available with its owner. Rules keep no per-window counters, so the windows are the project's, from the outcome ledger.

How a change is governed

A person edits a rule in settings (PATCH /sampling-rules/{id}), which writes the row directly, bumps its version, keeps the revision it replaced and audits the change as log_pipeline_rule.update with the filters before and after. An agent goes through the log_pipeline_rule operation family and nothing else. The act is derived from the before-and-after diff, never from an argument:

Act Derived when Class
log_pipeline_rule.create A new rule. It lands in shadow. ELEVATED
log_pipeline_rule.update The rule removes more (a lower rate, a shadow rule moved live) or matches differently (filters that are not a narrowing of the old ones). SENSITIVE, always asks
log_pipeline_rule.narrow The rule only does less (a higher rate, more conditions on top of the old filters, any change to a rule still in shadow, a rule taken out of force). ELEVATED; SENSITIVE when the records it adds exceed 10 % of the monthly allowance
log_pipeline_rule.delete Never by an agent. A person, in settings. human-only

A plan carries the version of the project's log rules it was made on (base_version). If the rules move between the approval and the retry, from any door, the apply is TARGET_CHANGED and spends nothing. The write must cite a preview digest of the exact rule (at most 60 minutes old, or the approved plan's own figures on a retry after an approval).

Refused before anyone is asked

Each of these names its cap and leaves no request, no operation and no rule:

  • a plan that would drop a floor record;
  • a plan that drops more than 50 % of the project's log records;
  • a rule that matches only error severities;
  • a sixth shadow rule, a twenty-first rule, a name already in force;
  • a kind that is another card's, a kind that disagrees with its rate, or a transform whose configuration fails its kind's check (an unbounded or unsafe pattern, a floor key).

Silencing-class

A live drop or sample rule whose preview named an alert rule, saved view, widget or saved query is silencing-class. It lapses seven days after the move that made it live (judged on read, so a stopped worker never leaves it dropping) and the operation states a seven-day TTL. A person confirms it by adopting the rule in settings, which clears the expiry and supersedes the operation.

Verification

At 60 minutes after an apply, and again at 7 days, the share of matched records the rule actually dropped (its own tallies since the apply) is compared with the preview. Within 20 % passes; 20 to 50 % is recorded as drift and asks nobody; above 50 % ends the watch not verified and opens the "keep or revert?" question. An alert rule that read the affected records and reported no data after the apply does the same, whatever the share did. The pass never reverts anything itself.

Reverting

operation_revert runs the stored inverse: a new rule is narrowed to disabled, a change restores the previous revision the rule itself stored. A restore never re-arms an expiry, and a rule changed by a person since is left alone.

Configuration as code

bucker.yml has no log pipeline resource. A file apply would be a second door to the same table with no preview digest and no approval; the resource ships only when its apply runs through the same plan digest.

What no tool can do

Delete a rule (log_pipeline_rule.delete, a person's act, project write). Raise a scope: the tools are held behind logs:write until the OAuth 2.1 security review (SG-059).