# Ingestion filters and schema rules: keeping noise out of product analytics, in steps you can undo

Some traffic should not count: a health check that posts `bot_ping` every minute, a staging build
that leaked a production key, an event that arrives without the property every funnel needs. An
**ingestion filter** matches events as they arrive; a **schema rule** requires an event to carry
named properties. Both run at the product-event door (stage 55 of the accept stages), after the
consent basis is resolved and before the event is queued. Both are stored per project, behind the
product-analytics gate (PA-01): a project that has not opened the gate runs nothing and every read
and write here refuses with `analytics_not_enabled`.

## The ladder

A filter is always in exactly one mode, and the modes only move one way without a person's eyes:

| Mode | What it does with a matching event | How a filter gets there |
| --- | --- | --- |
| `dry_run` | Counts it as "would quarantine" and leaves it alone | Every new filter starts here, whatever was asked for |
| `quarantine` | Stores it apart for **14 days**, out of every insight; it can be released | A dry-run filter whose test cases pass; a person approves |
| `reject` | Refuses it: never stored, cannot be recovered | A filter that has sat **7 days** in quarantine, tests passing, no release request open; a person approves under an ask floor |

`disabled` and `archived` are inert states. Disabling remembers the mode it left.

The rule is one function (`judgeFilterMove` in `filters.ts`) read by the planner, the preconditions
and the structural test, so there is a single statement of it. No policy, trust rung or approver
can lift a refusal: a refusal is a precondition of the plan, not a decision about it, and the
engine refuses the plan before anyone is asked. **Nothing an agent proposes drops an event before
that event has spent seven days in quarantine.**

## Writing a filter

An expression is a tree of at most **20 conditions** and **5 levels**: `{ and: [...] }`,
`{ or: [...] }`, `{ not: ... }`, over leaves that compare one of three things:

* `event_name`, the event's name as sent (normalised the way the taxonomy does);
* `distinct_id_hash`, the **keyed hash** of the distinct id. A filter never reads a raw identifier;
* `property`, one top-level property, by `name`.

Operators are `equals`, `contains` and `regex`. A regex is a closed grammar run by a linear-time
engine (a Thompson NFA, the seam P3-121's RE2 binding replaces), so no pattern can backtrack.
Backreferences, lookaround, named groups, flags, lazy quantifiers and `\b` are refused with the
reason; a pattern is at most 200 characters and matching is unanchored unless it says `^` or `$`.

```json
{
  "name": "bot-pings",
  "expr": { "field": "event_name", "op": "equals", "value": "bot_ping" },
  "tests": {
    "mustMatch":    [{ "event": "bot_ping" }],
    "mustNotMatch": [{ "event": "signup" }]
  }
}
```

A project holds at most 50 filters and 50 schema rules. A filter name is letters, digits, space and
`_ . : -`.

### Test cases

`tests.mustMatch` and `tests.mustNotMatch` hold up to 20 events each (an event name, an optional
distinct-id hash, optional properties). A filter **cannot move to quarantine until both lists have
an entry and every case passes now**. Test cases are editable only while the filter is in dry run,
and not in the same call that moves it, so a move is judged on tests that were already there.

### A filter is never edited in place

An expression is immutable once written. A different expression is a different filter: it starts in
dry run, and the old one is archived (dry run) or disabled (live). This is the same copy-on-write
rule insights follow, and it is what makes "the 7 days it spent in quarantine" mean something.

## Schema rules

A schema rule lives on an event definition (`enforcement`) and says which properties every such
event must carry, with an optional type (`string`, `number`, `boolean`). It follows the same ladder:
dry run counts the events that would fail, quarantine holds them, reject refuses them. The event
must already have been seen. Writing a rule does not bump the definition's revision, so it never
invalidates a pending taxonomy approval.

## What the counts mean

Each evaluator keeps hourly counters (`ProductEventWarning` rows keyed by the filter's id):
would-quarantine (dry run), held, rejected, and `filter_error`. The numbers an approver reads on
the approval card are the **7-day** counts. They are read when the plan is made and are **not part
of the plan's digest**: live traffic cannot invalidate an approval.

When several evaluators match one event the strongest wins (reject, then quarantine) and every
dry-run filter that matches is still counted.

A stored filter that cannot be evaluated (its regex no longer compiles) counts a `filter_error` and
**does not match**: the stage fails open, and never fails the request. A quarantine that cannot be
stored leaves the event live and counts the same error.

## Quarantine

A held event is stored as one blob and one `ProductEventQuarantine` row per (envelope, evaluator),
not per event, with a **14-day** expiry. The ingest receipt answers `accepted` with
`ingestion_filter_quarantined`; a rejected event is answered `invalid` with
`ingestion_filter_blocked` (a closed vocabulary, billed against: the filter id is in the per-filter
counters and the quarantine row, not on the wire). Quarantined events are excluded from every insight.

**Release** puts the held items back on the analytics lane without running the accept stages again
(they would be quarantined again) and without a second outcome row (they were already counted as
accepted). Claiming a row is a compare-and-swap on its expiry, so two releases enqueue it once.
Release is part of disable: stopping a filter and releasing what it held happen in one transaction.

### Reading quarantined events as an agent

Agents never read rows. `quarantineSamples` returns scrubbed samples (agent-tier scrub), scans each
for injection (failing closed), withholds and **counts** what trips either, and labels everything
`untrusted`: a quarantined event is by construction traffic somebody chose to send.

## Who may do what

| Act | Action | Class | Door |
| --- | --- | --- | --- |
| Create a filter (dry run), edit its tests, un-archive | `ingestion_filter.create` | ELEVATED, allow | dashboard, REST, `ingestion_filters_update` |
| Dry run to quarantine | `ingestion_filter.quarantine` | SENSITIVE, ask | same |
| Quarantine to reject | `ingestion_filter.block` | SENSITIVE, ask floor | same |
| Archive your own dry-run filter | `ingestion_filter.archive` | ELEVATED, allow | same |
| Disable, move down, release | `ingestion_filter.disable` | SENSITIVE, ask | **dashboard and REST only** |

Disabling, releasing and moving a filter down each re-admit events into every insight at once, so
the MCP tool has no field for them and refuses a diff that amounts to one. An agent may archive only
a dry-run filter **it authored itself**.

Every write is one governed operation (`governedOperation`, family `ingestion_filter`): a plan, a
digest, the engine's decision, a human's yes bound to that digest, one apply against the version
the plan read, a signed receipt and a stored inverse. A preview is its own route
(`POST .../ingestion-filters/preview`), never a flag. `operation_revert` restores the previous mode.

A pattern copied from a quarantined sample is a pattern driven by payload text: every condition
value and test string is in the plan's free text, the operation is marked `untrusted_text` and the
approver is told.

## Interfaces

REST, under `/orgs/:orgSlug/projects/:projectSlug/ingestion-filters`:

* `GET` the overview: each filter and rule with mode, time in mode, 7-day counts, test report,
  what is held, and the moves open to it with the reason any is closed (`read_analytics`);
* `GET /:filterId/quarantine` samples, scrubbed and labelled untrusted (`read_analytics`);
* `POST /preview` what a change would be, the act it would derive and what refuses it; writes
  nothing (`configure_analytics`);
* `POST` the governed write: 200 applied, 202 asked, retried with `approvalRequestId`
  (`configure_analytics`).

MCP: `ingestion_filters_update` (held, `configure_analytics`, toolset `analytics`) and the
`filters` block of `taxonomy_get`, which counts rows by mode. Both are held until the SG-059
review: no token a production path can mint sees them.

## Caching across machines

The set of filters is cached per process for a short TTL under `<orgId>:<projectId>`. The machine
that applied a change evicts its copy at once; another api machine keeps its own until the TTL ends.
A new filter therefore takes up to that long to start counting, and a **disabled** one up to that
long to stop. What a machine still holding the old set quarantined after the disable is not lost:
`{ name, lifecycle: "disable", release: true }` on an already-disabled filter is a release on its
own (`ingestion_filter.disable`, asked, the mode unmoved). A filter disabled from reject may drop
events on such a machine for the same window; those are not recoverable.
