# Pipeline rules: what leaves for a destination or an export, as typed data

A destination or an export has an ordered list of rules (at most 20 per target) that decide which
rows and which fields leave Bucker for it. A rule is typed data a person can read and diff: a kind,
a predicate, the kind's arguments and, only where a computed condition or value is needed, a CEL
expression. There is no customer JavaScript and no Bucker VM.

The grammar lives in one place, `@bucker/shared/pipeline`, and is imported by the egress rules
described here, the log-ingest rules (LOG-11), the product-event rules (PA-18) and the capture rules
(SDK-14). Each keeps its own table and stage order; none of them has its own dialect.

## The kinds

| Kind | What it does to a row that matches |
| --- | --- |
| `drop` | discards it |
| `sample` | keeps a deterministic share (`keepRate`, 0 up to but not including 1), keyed by `keyBy` so a subject's rows stay together |
| `keep_fields` | keeps only the named fields |
| `remove_fields` | removes the named fields |
| `mask` | replaces the named fields' values with a fixed marker |
| `hash` | replaces the named fields' values with a salted SHA-256 |
| `rename` | moves a field to another name |
| `set` | writes a constant, or a CEL expression's value, into a field |
| `truncate` | cuts the named string fields to a length |
| `route` | labels the row with a named route |

The predicate is Explore's filter grammar (twelve operators, the same field-path rule; a path that
reaches the prototype chain is refused). A rule with no predicate matches every row. Rules run in
`position` order. A rule is in one of three states: `shadow` (records what it would do, changes
nothing), `active` and `disabled`.

## The never-drop floor

No applied rule drops an error event, whatever its predicate, kind or expression. The floor is
consulted at apply, and the tally of every rule says how many matching rows it kept. A shadow rule
removes nothing at all.

## CEL, bounded

`celExpr` is a condition (a boolean, ANDed with the predicate) on every kind except `set`, where it
is the value (a scalar, at most 512 characters). It is evaluated by `@bufbuild/cel` behind the
`expression` provider seam, never trusted to enforce its own limits:

- **No I/O.** Function names are read from the text and checked against a closed allow-list
  (`size`, `has`, `contains`, `startsWith`, `endsWith`, `lowerAscii`, `upperAscii`, `string`, `int`,
  `double`, `bool`, `all`, `exists`, `exists_one`). A name on the I/O list (`http`, `exec`, `read`,
  `now`, `random` and the like) is refused by name; a name on neither list is refused as not allowed.
- **A cost limit per event** (1,000 units) at compile time and again at run time.
- **Outputs checked.** A condition must be a boolean and a value must be a bounded scalar; anything
  else is a wrong-type result.
- A text over 1,024 characters or a syntax error is refused.

A refused expression never earns a preview digest. An expression that returns the wrong type on real
rows of the preview window is refused the same way.

### When the expression provider is unusable

Each caller declares what "selected but unusable" means, and the two differ on purpose:

| Caller | Policy | Why |
| --- | --- | --- |
| egress (destinations, exports) | fail **closed**: the target holds its rows and is paused with `expression_unavailable` | an unapplied mask on egress is a leak |
| ingest | fail **open**: the row is stored untransformed and every one is counted | dropping telemetry over a missing library is the worse outcome |

A CEL rule cannot be proposed while the provider is unusable, and a pipeline that carries one cannot
be previewed.

## Previewing a change

`POST /orgs/:org/projects/:project/pipeline/preview` (and the digest every write must cite) replays
the target's whole ordered pipeline, with the proposal composed in, over up to 10,000 stored events
and log records of the last 24 hours, scrubbed at the agent tier. It reports:

- rows in and out per rule, and the rows the floor kept;
- the fields changed, and three example diffs rendered as data;
- rows a day saved or added (scaled when the window holds more than the sample);
- the dependents: lineage says `not_available` until a package declares an edge, and the consumers
  (alert rules, saved views, widgets, saved queries) of the log signal are named;
- the alert evaluations a drop would have changed.

A rule in shadow is priced as what it would do once live. An accepted preview stores a digest bound
to the project, the target and the composed pipeline, valid 30 minutes. A refused one (a widening, an
expression the seam would refuse) stores nothing.

## Changing a pipeline

The write is the `pipeline_rule` operation family, an upsert through the governed engine
(`transformations_update` over MCP, `POST …/pipeline/rules` over REST). The act is derived from the
diff, never from an argument:

| Change | Act | Class |
| --- | --- | --- |
| no `ruleId` | `pipeline.transformation.create` (the rule is born in `shadow`) | SENSITIVE, ASK |
| `state: disabled` | `pipeline.transformation.disable` | SENSITIVE, ASK |
| anything else: an edit, a reorder, going live, a re-enable | `pipeline.transformation.update` | SENSITIVE, ASK |
| delete | `pipeline.transformation.delete` | a person's act, never a tool's |

A rule born in shadow goes live as its own act. Only the earned `pipeline_tuning` rung may ALLOW a
create or update, and only for a `drop` or `sample` rule without CEL; a CEL rule and every field rule
always ask. A rollback (`operation_revert`) restores the rule, not the rows a drop rule discarded
while it was live.

Refused before anyone is asked: a change with no preview digest, with another change's, another
target's or an expired one; an invalid or identical rule; a 21st rule; an expression the seam
refuses; and any change that would let a held-back field leave (disabling or weakening a `mask`,
`hash`, `remove_fields`, `truncate` or `keep_fields` rule). That last one is
`pipeline.destination.widen`, a person's act with `egress:admin`. The same holds for a delete:
a person deleting an active field-protecting rule whose removal would let a field leave is refused
with `widens_egress` (delete a shadow or disabled rule, or widen through `pipeline.destination.widen`).

## Who can do what

- Reading rules and previewing: the project's readers with telemetry access.
- Proposing: `pipeline:write` (`configure_pipeline`). The MCP tool is held until the SG-059 review:
  no production-mintable token carries the scope, so the dashboard and REST are the live surfaces.
- Approving: a different person with the approver relation, bound to the plan digest.
- Deleting: a person on their own session. An agent, a service and an OAuth client are refused.

Another organization's target, rules, previews and digests are one "not found".

## Text a rule carries is data

A rule's predicate values, arguments and expression are written by a member or an agent. Every
response that returns them labels them untrusted (`trust.untrustedFields`), and the preview's
example rows, which are customer telemetry, are labelled the same way.
