# Log pipeline rules

Choose what share of a project's log records Bucker stores, on the server, without changing any
SDK. This page covers the keep, drop and sample rules of the log pipeline: how they behave, how a
person or an agent changes them, and what holds whoever asks. Span rules share the same table and
the same preview engine and are described in [Sampling rules](sampling.md).

## What a rule is

A rule says "of the log records matching these filters, keep this share". The filters are the
Explore filters (`severity_number`, `severity`, `message`, `environment`, any attribute), ANDed, and
the rate runs from `0` to `1`. The first enabled rule by position that matches a record decides; a
record no rule matches is kept. The rate says what the rule is:

| Kind | Rate | What it does |
| --- | --- | --- |
| **keep** | `1` | Keeps what it matches, and so protects those records from any later rule. |
| **drop** | `0` | Drops what it matches. |
| **sample** | between | Keeps that share of what it matches. |

A rule is in one of three states:

| State | What it does |
| --- | --- |
| **shadow** | Evaluated beside the live rules and counts what it _would_ have kept and dropped (`shadowKept`, `shadowDropped`). It **drops nothing**. |
| **active** | Applies. |
| **disabled** | Not evaluated. |

A shadow rule is stored switched off for an older image (`enabled: false`), so a server that predates
the status column can never apply it by mistake. Every rule an agent creates starts in shadow and
lapses in 7 days unless it is renewed (30 in all).

Log metrics, quotas, retention routes and conversions are other cards' (LOG-13, -10, -18, -14).
Asking for one is refused, naming the card. The transform kinds are described next.

## Transform rules (processing at ingest)

A transform rule rewrites a record instead of choosing whether to keep it. Seven kinds exist, stored
in the same table (`kind`, with typed parameters in `config`), each with a filter in the Explore
grammar. There is no expression language, no template, no script and no Grok dialect: a rule can only
do what one of these does, to the fields it names.

| Kind | What it does | Bounds |
| --- | --- | --- |
| `parse_json` | Reads the body as JSON into attributes. A key the sender already sent wins. | At most 50 keys, values truncated at 1,000 characters |
| `parse_kv` | Reads `key=value` pairs from the body. Sender keys win. | Same |
| `parse_regex` | One pattern with named groups; each group becomes an attribute. | Pattern at most 200 characters, passes the same ReDoS check an org scrub rule does, runs over at most 8,192 characters |
| `rename` | Moves an attribute to another key; never onto a key that exists. | Floor keys refused |
| `remove_fields` | Removes attributes (exact keys or a `prefix*`). | At most 20 keys; floor keys refused |
| `set_severity` | Sets severity from a field or the body's leading token, through a map. | Never lowers a record the floor protects, nor one that arrived at ERROR or worse below ERROR |
| `route_retention` | Chooses the retention class a record is stored under (LOG-18's choice). | Classes 3 to 365 days; the stage runs here, the governed write is LOG-18's, so the tools refuse it by name |

**Order is fixed, and transforms run first**: `parse_json`, `parse_kv`, `parse_regex`, `rename`,
`remove_fields`, `set_severity`, `route_retention`, then the keep, drop and sample rules, then the
quota gate, then the stored-tier scrub. This departs deliberately from OTEP 4738, which keeps and
then transforms: here a sampling rule, a quota or a metric sees the fields a parse rule extracted,
which a Datadog pipeline filter cannot. Within one kind, rules run by position. A stage registry
(`registerLogStage(order, kind, fn)`) lets the annotate, quota and metric stages attach from their
own modules; two stages may not share an order or a kind.

What the stages promise:

- **Inert by default.** Behind the `log_stages` pillar. With the pillar off, or no transform rule, the
  batch is the same array and no byte of any row changes. A row without `config` is a sampler.
- **A transform never drops a record and never touches the floor.** A rule naming `exception.*`, a
  severity, a trace id or a span id as something to remove or rename is refused at save, and at run
  time a change that would end a record's floor protection, or re-level a record that arrived at
  ERROR or worse below ERROR, is discarded for that record and counted `failed`.
- **Fails open.** A record a rule cannot process is stored unparsed and counted `failed`. A stage that
  throws or returns the wrong number of records is skipped for that batch.
- **Counted.** Each rule keeps `matched`, `applied`, `failed` and `cpuMicros`. A rule whose mean cost
  of processing the records it SELECTED (not of judging its filter on the rest) exceeds 200
  microseconds over 1,000 such records is skipped in that process for five minutes. This is an
  in-process breaker, not yet the card's persistent move to shadow with a finding.
- **Marked.** A record a rule changed carries `bucker.pipeline`, the ids of the rules that did. When
  the stages run, a sender's own `bucker.pipeline` is overwritten on a changed record and removed
  from every other one.
- **Shadow.** A shadow transform counts what it would do and changes nothing.

Governance: creating or changing a transform is SENSITIVE and asks; no `pipeline_tuning` rung reaches
it. A change that removes less or matches less is narrower. A regex over its budget, a pattern without
a named group or one that fails the ReDoS check is refused at save, with no request created. A
project may hold at most 50 transform rules.

## What is never dropped

The never-drop floor keeps these whatever a rule says, and a rule that **matches only** them (a
filter that forces severity 17 or above) is refused as a no-op:

- records of severity ERROR or worse;
- records carrying an `exception.*` attribute;
- records linked to an open issue;
- records on a trace that has an error.

The preview refuses a plan that would drop one, with a count, no digest and a narrower plan (for
example `severity_number lt 17`). Nothing the floor protects is ever discounted by a rate.

## Previewing a rule

`POST /orgs/{org}/projects/{project}/log-pipeline/preview` (and the MCP tool `log_pipeline_preview`)
replays a rule through the sampler over the last 24 hours of the project's stored log records (at
most 168). The replay reads a bounded sample, newest first, and scales every count to the matching
set, saying so in `approximations`. It answers:

- the records the rule matches, drops and keeps, and its share of the project's log volume;
- the meter units and dollars saved a day and a month, and the days of the monthly allowance gained;
- the alert rules, saved views, dashboard widgets and saved queries whose match sets change, named,
  with the number of evaluations that would have changed ("would have changed N evaluations");
- the floor.

An accepted preview returns a **digest**, valid for 60 minutes and bound to the project. A proposal
cites it, and the figures the proposal carries are loaded from that stored preview, never from the
caller.

Previewing needs `event:read` on the project, because it reads stored log records.

## Listing the pipeline

`GET /orgs/{org}/projects/{project}/log-pipeline/rules` (and `log_pipeline_rules_list`) returns every
stage in order: each rule with its kind, state, rate, filters, expiry, shadow tallies and the hours
they have run, its lifetime live tallies with the records the floor kept, who proposed it and the
approval that let it in. Beside them: the project's log tallies for the last 24 hours and 7 days, the
units the drops saved a day, the consumers of the logs dataset, and the stages other cards own, each
`not_available` with its owner. Rules keep no per-window counters, so the windows are the project's,
from the outcome ledger.

## How a change is governed

A person edits a rule in settings (`PATCH /sampling-rules/{id}`), which writes the row directly,
bumps its version, keeps the revision it replaced and audits the change as `log_pipeline_rule.update`
with the filters before and after. An agent goes through the `log_pipeline_rule` operation family and
nothing else. The act is derived from the before-and-after diff, never from an argument:

| Act | Derived when | Class |
| --- | --- | --- |
| `log_pipeline_rule.create` | A new rule. It lands in shadow. | ELEVATED |
| `log_pipeline_rule.update` | The rule removes more (a lower rate, a shadow rule moved live) or matches differently (filters that are not a narrowing of the old ones). | SENSITIVE, always asks |
| `log_pipeline_rule.narrow` | The rule only does less (a higher rate, more conditions on top of the old filters, any change to a rule still in shadow, a rule taken out of force). | ELEVATED; SENSITIVE when the records it adds exceed 10 % of the monthly allowance |
| `log_pipeline_rule.delete` | Never by an agent. A person, in settings. | human-only |

A plan carries the version of the project's log rules it was made on (`base_version`). If the rules
move between the approval and the retry, from any door, the apply is `TARGET_CHANGED` and spends
nothing. The write must cite a preview digest of the exact rule (at most 60 minutes old, or the
approved plan's own figures on a retry after an approval).

### Refused before anyone is asked

Each of these names its cap and leaves no request, no operation and no rule:

- a plan that would drop a floor record;
- a plan that drops more than 50 % of the project's log records;
- a rule that matches only error severities;
- a sixth shadow rule, a twenty-first rule, a name already in force;
- a kind that is another card's, a kind that disagrees with its rate, or a transform whose
  configuration fails its kind's check (an unbounded or unsafe pattern, a floor key).

### Silencing-class

A live drop or sample rule whose preview named an alert rule, saved view, widget or saved query is
**silencing-class**. It lapses seven days after the move that made it live (judged on read, so a
stopped worker never leaves it dropping) and the operation states a seven-day TTL. A person confirms
it by adopting the rule in settings, which clears the expiry and supersedes the operation.

### Verification

At 60 minutes after an apply, and again at 7 days, the share of matched records the rule actually
dropped (its own tallies since the apply) is compared with the preview. Within 20 % passes; 20 to 50 %
is recorded as drift and asks nobody; above 50 % ends the watch not verified and opens the "keep or
revert?" question. An alert rule that read the affected records and reported no data after the apply
does the same, whatever the share did. The pass never reverts anything itself.

### Reverting

`operation_revert` runs the stored inverse: a new rule is narrowed to disabled, a change restores the
previous revision the rule itself stored. A restore never re-arms an expiry, and a rule changed by a
person since is left alone.

## Configuration as code

`bucker.yml` has no log pipeline resource. A file apply would be a second door to the same table with
no preview digest and no approval; the resource ships only when its apply runs through the same plan
digest.

## What no tool can do

Delete a rule (`log_pipeline_rule.delete`, a person's act, project `write`). Raise a scope: the tools
are held behind `logs:write` until the OAuth 2.1 security review (SG-059).
