Ingestion filters and schema rules: keeping noise out of product analytics, in steps you can undo
Some traffic should not count: a health check that posts bot_ping every minute, a staging build
that leaked a production key, an event that arrives without the property every funnel needs. An
ingestion filter matches events as they arrive; a schema rule requires an event to carry
named properties. Both run at the product-event door (stage 55 of the accept stages), after the
consent basis is resolved and before the event is queued. Both are stored per project, behind the
product-analytics gate (PA-01): a project that has not opened the gate runs nothing and every read
and write here refuses with analytics_not_enabled.
The ladder
A filter is always in exactly one mode, and the modes only move one way without a person's eyes:
| Mode | What it does with a matching event | How a filter gets there |
|---|---|---|
dry_run |
Counts it as "would quarantine" and leaves it alone | Every new filter starts here, whatever was asked for |
quarantine |
Stores it apart for 14 days, out of every insight; it can be released | A dry-run filter whose test cases pass; a person approves |
reject |
Refuses it: never stored, cannot be recovered | A filter that has sat 7 days in quarantine, tests passing, no release request open; a person approves under an ask floor |
disabled and archived are inert states. Disabling remembers the mode it left.
The rule is one function (judgeFilterMove in filters.ts) read by the planner, the preconditions
and the structural test, so there is a single statement of it. No policy, trust rung or approver
can lift a refusal: a refusal is a precondition of the plan, not a decision about it, and the
engine refuses the plan before anyone is asked. Nothing an agent proposes drops an event before
that event has spent seven days in quarantine.
Writing a filter
An expression is a tree of at most 20 conditions and 5 levels: { and: [...] },
{ or: [...] }, { not: ... }, over leaves that compare one of three things:
event_name, the event's name as sent (normalised the way the taxonomy does);distinct_id_hash, the keyed hash of the distinct id. A filter never reads a raw identifier;property, one top-level property, byname.
Operators are equals, contains and regex. A regex is a closed grammar run by a linear-time
engine (a Thompson NFA, the seam P3-121's RE2 binding replaces), so no pattern can backtrack.
Backreferences, lookaround, named groups, flags, lazy quantifiers and \b are refused with the
reason; a pattern is at most 200 characters and matching is unanchored unless it says ^ or $.
{
"name": "bot-pings",
"expr": { "field": "event_name", "op": "equals", "value": "bot_ping" },
"tests": {
"mustMatch": [{ "event": "bot_ping" }],
"mustNotMatch": [{ "event": "signup" }]
}
}
A project holds at most 50 filters and 50 schema rules. A filter name is letters, digits, space and
_ . : -.
Test cases
tests.mustMatch and tests.mustNotMatch hold up to 20 events each (an event name, an optional
distinct-id hash, optional properties). A filter cannot move to quarantine until both lists have
an entry and every case passes now. Test cases are editable only while the filter is in dry run,
and not in the same call that moves it, so a move is judged on tests that were already there.
A filter is never edited in place
An expression is immutable once written. A different expression is a different filter: it starts in dry run, and the old one is archived (dry run) or disabled (live). This is the same copy-on-write rule insights follow, and it is what makes "the 7 days it spent in quarantine" mean something.
Schema rules
A schema rule lives on an event definition (enforcement) and says which properties every such
event must carry, with an optional type (string, number, boolean). It follows the same ladder:
dry run counts the events that would fail, quarantine holds them, reject refuses them. The event
must already have been seen. Writing a rule does not bump the definition's revision, so it never
invalidates a pending taxonomy approval.
What the counts mean
Each evaluator keeps hourly counters (ProductEventWarning rows keyed by the filter's id):
would-quarantine (dry run), held, rejected, and filter_error. The numbers an approver reads on
the approval card are the 7-day counts. They are read when the plan is made and are not part
of the plan's digest: live traffic cannot invalidate an approval.
When several evaluators match one event the strongest wins (reject, then quarantine) and every dry-run filter that matches is still counted.
A stored filter that cannot be evaluated (its regex no longer compiles) counts a filter_error and
does not match: the stage fails open, and never fails the request. A quarantine that cannot be
stored leaves the event live and counts the same error.
Quarantine
A held event is stored as one blob and one ProductEventQuarantine row per (envelope, evaluator),
not per event, with a 14-day expiry. The ingest receipt answers accepted with
ingestion_filter_quarantined; a rejected event is answered invalid with
ingestion_filter_blocked (a closed vocabulary, billed against: the filter id is in the per-filter
counters and the quarantine row, not on the wire). Quarantined events are excluded from every insight.
Release puts the held items back on the analytics lane without running the accept stages again (they would be quarantined again) and without a second outcome row (they were already counted as accepted). Claiming a row is a compare-and-swap on its expiry, so two releases enqueue it once. Release is part of disable: stopping a filter and releasing what it held happen in one transaction.
Reading quarantined events as an agent
Agents never read rows. quarantineSamples returns scrubbed samples (agent-tier scrub), scans each
for injection (failing closed), withholds and counts what trips either, and labels everything
untrusted: a quarantined event is by construction traffic somebody chose to send.
Who may do what
| Act | Action | Class | Door |
|---|---|---|---|
| Create a filter (dry run), edit its tests, un-archive | ingestion_filter.create |
ELEVATED, allow | dashboard, REST, ingestion_filters_update |
| Dry run to quarantine | ingestion_filter.quarantine |
SENSITIVE, ask | same |
| Quarantine to reject | ingestion_filter.block |
SENSITIVE, ask floor | same |
| Archive your own dry-run filter | ingestion_filter.archive |
ELEVATED, allow | same |
| Disable, move down, release | ingestion_filter.disable |
SENSITIVE, ask | dashboard and REST only |
Disabling, releasing and moving a filter down each re-admit events into every insight at once, so the MCP tool has no field for them and refuses a diff that amounts to one. An agent may archive only a dry-run filter it authored itself.
Every write is one governed operation (governedOperation, family ingestion_filter): a plan, a
digest, the engine's decision, a human's yes bound to that digest, one apply against the version
the plan read, a signed receipt and a stored inverse. A preview is its own route
(POST .../ingestion-filters/preview), never a flag. operation_revert restores the previous mode.
A pattern copied from a quarantined sample is a pattern driven by payload text: every condition
value and test string is in the plan's free text, the operation is marked untrusted_text and the
approver is told.
Interfaces
REST, under /orgs/:orgSlug/projects/:projectSlug/ingestion-filters:
GETthe overview: each filter and rule with mode, time in mode, 7-day counts, test report, what is held, and the moves open to it with the reason any is closed (read_analytics);GET /:filterId/quarantinesamples, scrubbed and labelled untrusted (read_analytics);POST /previewwhat a change would be, the act it would derive and what refuses it; writes nothing (configure_analytics);POSTthe governed write: 200 applied, 202 asked, retried withapprovalRequestId(configure_analytics).
MCP: ingestion_filters_update (held, configure_analytics, toolset analytics) and the
filters block of taxonomy_get, which counts rows by mode. Both are held until the SG-059
review: no token a production path can mint sees them.
Caching across machines
The set of filters is cached per process for a short TTL under <orgId>:<projectId>. The machine
that applied a change evicts its copy at once; another api machine keeps its own until the TTL ends.
A new filter therefore takes up to that long to start counting, and a disabled one up to that
long to stop. What a machine still holding the old set quarantined after the disable is not lost:
{ name, lifecycle: "disable", release: true } on an already-disabled filter is a release on its
own (ingestion_filter.disable, asked, the mode unmoved). A filter disabled from reject may drop
events on such a machine for the same window; those are not recoverable.