Skip to content
bucker

Ingestion filters and schema rules: keeping noise out of product analytics, in steps you can undo

Some traffic should not count: a health check that posts bot_ping every minute, a staging build that leaked a production key, an event that arrives without the property every funnel needs. An ingestion filter matches events as they arrive; a schema rule requires an event to carry named properties. Both run at the product-event door (stage 55 of the accept stages), after the consent basis is resolved and before the event is queued. Both are stored per project, behind the product-analytics gate (PA-01): a project that has not opened the gate runs nothing and every read and write here refuses with analytics_not_enabled.

The ladder

A filter is always in exactly one mode, and the modes only move one way without a person's eyes:

Mode What it does with a matching event How a filter gets there
dry_run Counts it as "would quarantine" and leaves it alone Every new filter starts here, whatever was asked for
quarantine Stores it apart for 14 days, out of every insight; it can be released A dry-run filter whose test cases pass; a person approves
reject Refuses it: never stored, cannot be recovered A filter that has sat 7 days in quarantine, tests passing, no release request open; a person approves under an ask floor

disabled and archived are inert states. Disabling remembers the mode it left.

The rule is one function (judgeFilterMove in filters.ts) read by the planner, the preconditions and the structural test, so there is a single statement of it. No policy, trust rung or approver can lift a refusal: a refusal is a precondition of the plan, not a decision about it, and the engine refuses the plan before anyone is asked. Nothing an agent proposes drops an event before that event has spent seven days in quarantine.

Writing a filter

An expression is a tree of at most 20 conditions and 5 levels: { and: [...] }, { or: [...] }, { not: ... }, over leaves that compare one of three things:

  • event_name, the event's name as sent (normalised the way the taxonomy does);
  • distinct_id_hash, the keyed hash of the distinct id. A filter never reads a raw identifier;
  • property, one top-level property, by name.

Operators are equals, contains and regex. A regex is a closed grammar run by a linear-time engine (a Thompson NFA, the seam P3-121's RE2 binding replaces), so no pattern can backtrack. Backreferences, lookaround, named groups, flags, lazy quantifiers and \b are refused with the reason; a pattern is at most 200 characters and matching is unanchored unless it says ^ or $.

{
  "name": "bot-pings",
  "expr": { "field": "event_name", "op": "equals", "value": "bot_ping" },
  "tests": {
    "mustMatch":    [{ "event": "bot_ping" }],
    "mustNotMatch": [{ "event": "signup" }]
  }
}

A project holds at most 50 filters and 50 schema rules. A filter name is letters, digits, space and _ . : -.

Test cases

tests.mustMatch and tests.mustNotMatch hold up to 20 events each (an event name, an optional distinct-id hash, optional properties). A filter cannot move to quarantine until both lists have an entry and every case passes now. Test cases are editable only while the filter is in dry run, and not in the same call that moves it, so a move is judged on tests that were already there.

A filter is never edited in place

An expression is immutable once written. A different expression is a different filter: it starts in dry run, and the old one is archived (dry run) or disabled (live). This is the same copy-on-write rule insights follow, and it is what makes "the 7 days it spent in quarantine" mean something.

Schema rules

A schema rule lives on an event definition (enforcement) and says which properties every such event must carry, with an optional type (string, number, boolean). It follows the same ladder: dry run counts the events that would fail, quarantine holds them, reject refuses them. The event must already have been seen. Writing a rule does not bump the definition's revision, so it never invalidates a pending taxonomy approval.

What the counts mean

Each evaluator keeps hourly counters (ProductEventWarning rows keyed by the filter's id): would-quarantine (dry run), held, rejected, and filter_error. The numbers an approver reads on the approval card are the 7-day counts. They are read when the plan is made and are not part of the plan's digest: live traffic cannot invalidate an approval.

When several evaluators match one event the strongest wins (reject, then quarantine) and every dry-run filter that matches is still counted.

A stored filter that cannot be evaluated (its regex no longer compiles) counts a filter_error and does not match: the stage fails open, and never fails the request. A quarantine that cannot be stored leaves the event live and counts the same error.

Quarantine

A held event is stored as one blob and one ProductEventQuarantine row per (envelope, evaluator), not per event, with a 14-day expiry. The ingest receipt answers accepted with ingestion_filter_quarantined; a rejected event is answered invalid with ingestion_filter_blocked (a closed vocabulary, billed against: the filter id is in the per-filter counters and the quarantine row, not on the wire). Quarantined events are excluded from every insight.

Release puts the held items back on the analytics lane without running the accept stages again (they would be quarantined again) and without a second outcome row (they were already counted as accepted). Claiming a row is a compare-and-swap on its expiry, so two releases enqueue it once. Release is part of disable: stopping a filter and releasing what it held happen in one transaction.

Reading quarantined events as an agent

Agents never read rows. quarantineSamples returns scrubbed samples (agent-tier scrub), scans each for injection (failing closed), withholds and counts what trips either, and labels everything untrusted: a quarantined event is by construction traffic somebody chose to send.

Who may do what

Act Action Class Door
Create a filter (dry run), edit its tests, un-archive ingestion_filter.create ELEVATED, allow dashboard, REST, ingestion_filters_update
Dry run to quarantine ingestion_filter.quarantine SENSITIVE, ask same
Quarantine to reject ingestion_filter.block SENSITIVE, ask floor same
Archive your own dry-run filter ingestion_filter.archive ELEVATED, allow same
Disable, move down, release ingestion_filter.disable SENSITIVE, ask dashboard and REST only

Disabling, releasing and moving a filter down each re-admit events into every insight at once, so the MCP tool has no field for them and refuses a diff that amounts to one. An agent may archive only a dry-run filter it authored itself.

Every write is one governed operation (governedOperation, family ingestion_filter): a plan, a digest, the engine's decision, a human's yes bound to that digest, one apply against the version the plan read, a signed receipt and a stored inverse. A preview is its own route (POST .../ingestion-filters/preview), never a flag. operation_revert restores the previous mode.

A pattern copied from a quarantined sample is a pattern driven by payload text: every condition value and test string is in the plan's free text, the operation is marked untrusted_text and the approver is told.

Interfaces

REST, under /orgs/:orgSlug/projects/:projectSlug/ingestion-filters:

  • GET the overview: each filter and rule with mode, time in mode, 7-day counts, test report, what is held, and the moves open to it with the reason any is closed (read_analytics);
  • GET /:filterId/quarantine samples, scrubbed and labelled untrusted (read_analytics);
  • POST /preview what a change would be, the act it would derive and what refuses it; writes nothing (configure_analytics);
  • POST the governed write: 200 applied, 202 asked, retried with approvalRequestId (configure_analytics).

MCP: ingestion_filters_update (held, configure_analytics, toolset analytics) and the filters block of taxonomy_get, which counts rows by mode. Both are held until the SG-059 review: no token a production path can mint sees them.

Caching across machines

The set of filters is cached per process for a short TTL under <orgId>:<projectId>. The machine that applied a change evicts its copy at once; another api machine keeps its own until the TTL ends. A new filter therefore takes up to that long to start counting, and a disabled one up to that long to stop. What a machine still holding the old set quarantined after the disable is not lost: { name, lifecycle: "disable", release: true } on an already-disabled filter is a release on its own (ingestion_filter.disable, asked, the mode unmoved). A filter disabled from reject may drop events on such a machine for the same window; those are not recoverable.