Alert rules

Bucker has one alert router. An issue rule watches issues (a new issue, a regression, a count of events on one issue, a lifecycle change). A metric rule watches an aggregate over a window. Both kinds of rule reach the same channels, use the same debounce ledger, and write the same alert-event history. A rule's type picks what it watches. It does not pick a separate delivery system.

Rules live under a project:

POST   /orgs/{org}/projects/{project}/alert-rules
GET    /orgs/{org}/projects/{project}/alert-rules
PATCH  /orgs/{org}/projects/{project}/alert-rules/{ruleId}
DELETE /orgs/{org}/projects/{project}/alert-rules/{ruleId}

In the dashboard, open Alerts › Rules.

New user feedback

The feedback_received condition (New user feedback in the editor) fires when a user report linked to the issue arrives and is not filed as spam. It is its own signal, like a lifecycle change. It never fires on telemetry, telemetry never fires it, and it cannot be combined with first_seen, regression, event_count, user_count or issue_lifecycle in one rule. The usual debounce applies, so a burst of reports pages once per cooldown. A report that arrives before its error is grouped pages as soon as it is linked to the issue. See User feedback.

{ "type": "issue", "conditions": [{ "id": "feedback_received" }], "actions": [...] }

Org-wide rules

An org-wide rule covers every project in the organization, including projects created after the rule. "Page me on any new FATAL in production, any project" is one rule, not one rule per project.

POST   /orgs/{org}/alert-rules
GET    /orgs/{org}/alert-rules
GET    /orgs/{org}/alert-rules/{ruleId}
PATCH  /orgs/{org}/alert-rules/{ruleId}
DELETE /orgs/{org}/alert-rules/{ruleId}

Same DSL. The body is the same as for a project rule. Bucker evaluates the rule separately for each project it covers. Each project gets its own cooldown and its own alert events, so an org-wide metric rule that breaches in one project pages for that project alone.
A project filter limits coverage. { "id": "project", "slugs": ["sandbox"], "match": "not_in" } skips a project. "match": "in" covers only the listed projects. A project rule cannot carry this filter.
Owners and admins only. Creating, editing or deleting an org-wide rule needs write on the organization. Anyone who can read the organization can list the rules.
Restrictions. An org-wide rule can only notify org-wide channels. It cannot use the event anomaly condition, because that condition's baseline is learned per project. Use telemetry_anomaly instead, or a project rule.

In the dashboard, an org-wide rule appears in every project's rule list, marked "all projects". The rule editor offers Applies to › Every project in the organization to owners and admins.

What a metric rule can watch

A metric rule has exactly one watched condition:

Condition	Watches	Threshold
`metric`	error events: `event_count`, `user_count`, `issue_count`	typed
`anomaly`	error events, as an hourly series	learned
`telemetry`	spans, logs or custom metrics: any Explore aggregate, or a counter's delta or rate	typed
`telemetry_anomaly`	spans, logs or custom metrics, as an hourly series	learned

Spans and logs

A telemetry condition is an Explore query that has a threshold. It uses Explore's filter operators, aggregate functions and attribute keys. Bucker compiles the condition with the same builder that draws the Explore chart, so the chart and the alert read the query the same way.

"p95 of checkout spans above 2 s for 10 minutes":

{
  "name": "Slow checkout",
  "type": "metric",
  "conditions": [
    {
      "id": "telemetry",
      "dataset": "spans",
      "query": [{ "key": "span.name", "op": "eq", "value": "checkout" }],
      "aggregate": "p95",
      "field": "span.duration",
      "comparator": "gt",
      "threshold": 2000,
      "windowMinutes": 10
    }
  ],
  "actions": [{ "kind": "notify_channel", "channelId": "…" }]
}

"More than 50 ERROR logs from billing in 5 minutes":

{
  "id": "telemetry",
  "dataset": "logs",
  "query": [
    { "key": "log.severity", "op": "gte", "value": "error" },
    { "key": "service.name", "op": "eq", "value": "billing" }
  ],
  "aggregate": "count",
  "comparator": "gt",
  "threshold": 50,
  "windowMinutes": 5
}

Datasets. spans, logs and metrics (see below).
Aggregates. count, count_unique, sum, avg, min, max, p50, p75, p95 and p99. Every aggregate except count needs a field.
Operators. eq, neq, contains, not_contains, starts_with, gt, gte, lt, lte, exists and not_exists. Filters in query are combined with AND.
Span keys. These are the same keys Explore uses: span.name, span.kind, span.status, span.duration (in milliseconds), environment, release, resource.* and any attribute name.
Log keys. These are log.severity, log.severity_number, log.body, trace_id, environment, release and any attribute name. log.severity gte error compares OTel severity numbers, so warn counts as less severe than error. Log records have no span.* or resource.* fields. Bucker refuses a rule that names one when you save it, not later when the rule runs.
Rule filters. A telemetry rule takes only environment and release filters. A span has no event level, so put severity or status in the query instead.
No data is not a breach. An avg or p95 over an empty window has no value. The rule reads "no data" and does not fire, even if its comparator is lt. A count over an empty window is a real zero.
Time. The window is measured on the span's start time or the log record's timestamp. Event-count metrics are measured on arrival time instead.

Custom metrics

With "dataset": "metrics", the condition watches metrics your services send over OTLP /v1/metrics. The rule has to name the metric: a metric.name eq filter is required, and Bucker refuses the rule at save time without one. The other keys are Explore's metrics dataset: metric.value (a gauge or sum point), metric.count, metric.sum, metric.min, metric.max, metric.unit, metric.kind, service, environment, release and any attribute.

The aggregate is always explicit. Gauges and sampled values use the ordinary aggregates over the window: "queue depth above 100 at any point in 10 minutes" is aggregate: "max", field: "metric.value", and "average cache hit rate under 80%" is avg. p95 is the 95th percentile of the stored point values, not of a histogram's buckets; for a histogram use metric.sum and metric.count.

A counter needs one more word. Summing the stored values of a cumulative counter adds up running totals, which is not a number anyone wants. Set counter to read it properly:

`counter`	Value
`delta`	The increase over the window.
`rate`	The increase per second: the delta divided by the window length.

Per series, a point with delta temporality is its own increase. A cumulative point is its difference from the previous point, and a drop is read as a restart, so the new value is the increase. The increases of all matching series are then summed. The first point of a series that has no earlier point within 15 minutes contributes nothing. counter needs "aggregate": "sum" (no field, or metric.value) and applies only to monotonic sum metrics; up-down counters are read as gauges.

"More than 5 errors per second for 10 minutes":

{
  "id": "telemetry",
  "dataset": "metrics",
  "query": [{ "key": "metric.name", "op": "eq", "value": "http.errors" }],
  "aggregate": "sum",
  "counter": "rate",
  "comparator": "gt",
  "threshold": 5,
  "windowMinutes": 10
}

The rule editor has a Custom metrics option under Watch. It seeds the metric.name filter and offers a Read the metric as choice (stored points, counter increase, counter rate). Grouping, warning thresholds, comparison with an earlier period, recovery messages and the backtest work as they do for spans and logs. A window with no points reads "no data" and does not fire.

telemetry_anomaly takes the same dataset, query, aggregate and field, plus the event anomaly's sensitivity, direction and minValue. Bucker scores the last complete hour against an hour-of-week baseline. It learns that baseline from the last four weeks of the same query. The learned threshold switches on only after four weeks of history. Until then, the rule's reason says when it will switch on.

Warning, recovery, percent change and fan-out

The metric and telemetry conditions accept four optional fields. A rule that leaves them out behaves as it did before they existed.

Field	Effect
`warningThreshold`	A second line on the same comparator. It must be crossed before `threshold` is. Crossing it fires a warning. Crossing `threshold` fires critical, even inside the cooldown, because the alert has escalated. Going from critical back to warning is recorded but not announced.
`notifyOnRecovery`	When a warning or critical series returns to normal, Bucker sends a resolved message. The message goes out even inside the cooldown. On a PagerDuty channel the incident gets a `resolve`. `telemetry_anomaly` accepts this field too.
`comparison`	`{ "kind": "previous_period", "offsetMinutes": 1440 }` compares the window with the same window that many minutes earlier. The value and the thresholds then count as percent change. If the earlier window held nothing, the rule reads "no baseline" and does not fire.
`groupBy`	`{ "by": "environment" }`, `{ "by": "release" }` or `{ "by": "tag", "key": "region" }` fans the rule out over that key. Each value gets its own cooldown, state and recovery. On a span or log rule, `tag` names an attribute. A rule evaluates at most 50 groups per pass, the busiest first, and its reason says when it cut the rest.

A group that goes quiet still recovers. If an environment stops erroring completely, it drops out of the query. Bucker still evaluates every group that is breaching or warning in its ledger, so that environment's incident gets its resolved message.

{
  "id": "metric",
  "aggregate": "event_count",
  "windowMinutes": 10,
  "comparator": "gt",
  "threshold": 200,
  "warningThreshold": 100,
  "comparison": { "kind": "previous_period", "offsetMinutes": 1440 },
  "groupBy": { "by": "environment" },
  "notifyOnRecovery": true
}

The webhook payload carries severity, recovered, group and comparison in alert.metric. See alert-webhooks.md.

Maintenance windows and mutes

A maintenance window keeps alert notifications quiet during planned work. It covers alert rules, the anomaly detector, and uptime and cron monitors, because monitors raise their alerts through the same router. A window covers one project, or every project in the org, including projects created later.

GET    /orgs/{org}/maintenance-windows
POST   /orgs/{org}/maintenance-windows        { name, startsAt, endsAt, projectSlug?, reason? }
POST   /orgs/{org}/maintenance-windows        { name, statusMaintenanceId }   # follow a status-page maintenance
POST   /orgs/{org}/maintenance-windows/{id}/end
DELETE /orgs/{org}/maintenance-windows/{id}

Suppressed, never dropped. Bucker still records every fire a window covers. It is written to the alert history as an alert event with status SUPPRESSED, and its suppression field names the window. Nothing is delivered. The fire still counts for the rule's cooldown, so if a condition is still true when the window ends, the alert goes out when the cooldown next expires.
Following a status page. A window bound to a status-page maintenance follows that maintenance's state. It is quiet while the maintenance runs, or while it is scheduled and inside its times. When the maintenance is marked complete, the quiet ends too.
Bounds. A window lasts at most 14 days. A window over one project needs write on that project. A window over the whole org needs write on the org.

A rule mute silences one rule until a given time:

POST   /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute   { minutes | until, scope: "rule" | "me" }
DELETE /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute?scope=rule|me

scope: "rule" mutes the rule for everyone. Its fires are recorded as SUPPRESSED. It needs project write and writes an alert_rule.mute audit row.
scope: "me" keeps the rule firing for everyone else. Only your own address is left out of its email deliveries.
A mute lasts at most 30 days and ends by itself. The dashboard's rule menu offers 1 h, 4 h and 24 h. A Slack alert card offers Mute rule 1h and Mute rule 4h.

Backtest: how often would this rule have fired?

Before you save a rule, the editor's Backtest panel replays it over the project's last 7 days. The API is:

POST /orgs/{org}/projects/{project}/alert-rules/backtest   { "rule": { …the rule… }, "days": 7 }

The replay uses the live evaluator. Bucker turns the stored history into the points the live pass would have evaluated: a metric value per group at each step, or an issue signal for each new issue, regression, lifecycle change and event. It runs each point through the same rule code, with the rule's own cooldown kept in memory. Nothing is written and nothing is delivered. The call needs only read on the project.

The response gives the count and its denominators:

Field	Meaning
`fires`	Notifications after the cooldown. This is the headline number.
`matches`	Evaluations that met the rule. This is what would have fired with no cooldown.
`suppressedByCooldown`	Matches the cooldown swallowed.
`evaluations`	Points or signals replayed.
`recoveries`	Resolved messages, for a rule with `notifyOnRecovery`.
`delivery.chatMessagesPerDestination`	Messages each Slack, Discord, Teams or chat-webhook destination would have received after the chat volume ceiling (12 per 5 minutes, then one rollup).
`fireTimes`, `firesByDay`	When the fires would have happened.
`approximations`	Each way the replay is coarser than the live pass.

The replay is coarser than the live pass in a few ways, and the response lists the ones that apply:

A metric rule is evaluated once per window (5 to 60 minutes) rather than every minute, so a breach shorter than that can be missed.
An issue's status is taken from the signal (new, regressed, ongoing), not from its stored history, so mutes that have since ended are not replayed.
At most 20,000 events are replayed.
The event anomaly condition is not backtested, because its baseline lives in the detector's stored profile. telemetry_anomaly is replayed hour by hour.

Where this is defined

The rule DSL: code/domain/src/modules/alerts/rules.ts.
The telemetry builder: code/domain/src/modules/alerts/telemetry-metric.ts, which reuses code/domain/src/modules/explore/span-sql-builder.ts.
The learned threshold: code/domain/src/modules/alerts/telemetry-anomaly.ts, which uses code/domain/src/modules/anomaly/seasonality.ts.
Groups, comparison and the points a rule is evaluated at: code/domain/src/modules/alerts/metric-measure.ts. Warning, critical and recovery: evaluateMetricRule in code/domain/src/modules/alerts/evaluator.ts.