# Alert rules

Bucker has one alert router. An **issue rule** watches issues (a new issue, a
regression, a count of events on one issue, a lifecycle change). A **metric rule**
watches an aggregate over a window. Both kinds of rule reach the same channels, use the
same debounce ledger, and write the same alert-event history. A rule's `type` picks what
it watches. It does not pick a separate delivery system.

Rules live under a project:

```
POST   /orgs/{org}/projects/{project}/alert-rules
GET    /orgs/{org}/projects/{project}/alert-rules
PATCH  /orgs/{org}/projects/{project}/alert-rules/{ruleId}
DELETE /orgs/{org}/projects/{project}/alert-rules/{ruleId}
```

In the dashboard, open **Alerts › Rules**.

### New user feedback

The `feedback_received` condition (**New user feedback** in the editor) fires when a user
report linked to the issue arrives and is not filed as spam. It is its own signal, like a
lifecycle change. It never fires on telemetry, telemetry never fires it, and it cannot be
combined with `first_seen`, `regression`, `event_count`, `user_count` or
`issue_lifecycle` in one rule. The usual debounce applies, so a burst of reports pages
once per cooldown. A report that arrives before its error is grouped pages as soon as it
is linked to the issue. See [User feedback](user-feedback.md).

```json
{ "type": "issue", "conditions": [{ "id": "feedback_received" }], "actions": [...] }
```

### Org-wide rules

An **org-wide rule** covers every project in the organization, including projects
created after the rule. "Page me on any new FATAL in production, any project" is one
rule, not one rule per project.

```
POST   /orgs/{org}/alert-rules
GET    /orgs/{org}/alert-rules
GET    /orgs/{org}/alert-rules/{ruleId}
PATCH  /orgs/{org}/alert-rules/{ruleId}
DELETE /orgs/{org}/alert-rules/{ruleId}
```

- **Same DSL.** The body is the same as for a project rule. Bucker evaluates the rule
  separately for each project it covers. Each project gets its own cooldown and its own
  alert events, so an org-wide metric rule that breaches in one project pages for that
  project alone.
- **A `project` filter limits coverage.** `{ "id": "project", "slugs": ["sandbox"],
  "match": "not_in" }` skips a project. `"match": "in"` covers only the listed projects.
  A project rule cannot carry this filter.
- **Owners and admins only.** Creating, editing or deleting an org-wide rule needs
  `write` on the organization. Anyone who can read the organization can list the rules.
- **Restrictions.** An org-wide rule can only notify org-wide channels. It cannot use
  the event `anomaly` condition, because that condition's baseline is learned per
  project. Use `telemetry_anomaly` instead, or a project rule.

In the dashboard, an org-wide rule appears in every project's rule list, marked "all
projects". The rule editor offers **Applies to › Every project in the organization**
to owners and admins.

## What a metric rule can watch

A metric rule has exactly one *watched* condition:

| Condition | Watches | Threshold |
| --- | --- | --- |
| `metric` | error events: `event_count`, `user_count`, `issue_count` | typed |
| `anomaly` | error events, as an hourly series | learned |
| `telemetry` | spans, logs or custom metrics: any Explore aggregate, or a counter's delta or rate | typed |
| `telemetry_anomaly` | spans, logs or custom metrics, as an hourly series | learned |

### Spans and logs

A `telemetry` condition is an Explore query that has a threshold. It uses Explore's
filter operators, aggregate functions and attribute keys. Bucker compiles the condition
with the same builder that draws the Explore chart, so the chart and the alert read the
query the same way.

"p95 of checkout spans above 2 s for 10 minutes":

```json
{
  "name": "Slow checkout",
  "type": "metric",
  "conditions": [
    {
      "id": "telemetry",
      "dataset": "spans",
      "query": [{ "key": "span.name", "op": "eq", "value": "checkout" }],
      "aggregate": "p95",
      "field": "span.duration",
      "comparator": "gt",
      "threshold": 2000,
      "windowMinutes": 10
    }
  ],
  "actions": [{ "kind": "notify_channel", "channelId": "…" }]
}
```

"More than 50 ERROR logs from billing in 5 minutes":

```json
{
  "id": "telemetry",
  "dataset": "logs",
  "query": [
    { "key": "log.severity", "op": "gte", "value": "error" },
    { "key": "service.name", "op": "eq", "value": "billing" }
  ],
  "aggregate": "count",
  "comparator": "gt",
  "threshold": 50,
  "windowMinutes": 5
}
```

- **Datasets.** `spans`, `logs` and `metrics` (see below).
- **Aggregates.** `count`, `count_unique`, `sum`, `avg`, `min`, `max`, `p50`, `p75`,
  `p95` and `p99`. Every aggregate except `count` needs a `field`.
- **Operators.** `eq`, `neq`, `contains`, `not_contains`, `starts_with`, `gt`, `gte`,
  `lt`, `lte`, `exists` and `not_exists`. Filters in `query` are combined with AND.
- **Span keys.** These are the same keys Explore uses: `span.name`, `span.kind`,
  `span.status`, `span.duration` (in milliseconds), `environment`, `release`,
  `resource.*` and any attribute name.
- **Log keys.** These are `log.severity`, `log.severity_number`, `log.body`,
  `trace_id`, `environment`, `release` and any attribute name. `log.severity gte
  error` compares OTel severity numbers, so `warn` counts as less severe than
  `error`. Log records have no `span.*` or `resource.*` fields. Bucker refuses a rule
  that names one when you save it, not later when the rule runs.
- **Rule filters.** A telemetry rule takes only `environment` and `release` filters.
  A span has no event level, so put severity or status in the query instead.
- **No data is not a breach.** An `avg` or `p95` over an empty window has no value.
  The rule reads "no data" and does not fire, even if its comparator is `lt`. A count
  over an empty window is a real zero.
- **Time.** The window is measured on the span's start time or the log record's
  timestamp. Event-count metrics are measured on arrival time instead.

### Custom metrics

With `"dataset": "metrics"`, the condition watches metrics your services send over OTLP
`/v1/metrics`. The rule has to name the metric: a `metric.name` `eq` filter is required, and
Bucker refuses the rule at save time without one. The other keys are Explore's `metrics`
dataset: `metric.value` (a gauge or sum point), `metric.count`, `metric.sum`, `metric.min`,
`metric.max`, `metric.unit`, `metric.kind`, `service`, `environment`, `release` and any
attribute.

The aggregate is always explicit. Gauges and sampled values use the ordinary aggregates over
the window: "queue depth above 100 at any point in 10 minutes" is
`aggregate: "max", field: "metric.value"`, and "average cache hit rate under 80%" is `avg`.
`p95` is the 95th percentile of the stored point values, not of a histogram's buckets; for
a histogram use `metric.sum` and `metric.count`.

A counter needs one more word. Summing the stored values of a cumulative counter adds up
running totals, which is not a number anyone wants. Set `counter` to read it properly:

| `counter` | Value |
| --- | --- |
| `delta` | The increase over the window. |
| `rate` | The increase per second: the delta divided by the window length. |

Per series, a point with delta temporality is its own increase. A cumulative point is its
difference from the previous point, and a drop is read as a restart, so the new value is the
increase. The increases of all matching series are then summed. The first point of a series
that has no earlier point within 15 minutes contributes nothing. `counter` needs
`"aggregate": "sum"` (no `field`, or `metric.value`) and applies only to monotonic `sum`
metrics; up-down counters are read as gauges.

"More than 5 errors per second for 10 minutes":

```json
{
  "id": "telemetry",
  "dataset": "metrics",
  "query": [{ "key": "metric.name", "op": "eq", "value": "http.errors" }],
  "aggregate": "sum",
  "counter": "rate",
  "comparator": "gt",
  "threshold": 5,
  "windowMinutes": 10
}
```

The rule editor has a **Custom metrics** option under *Watch*. It seeds the `metric.name`
filter and offers a *Read the metric as* choice (stored points, counter increase, counter
rate). Grouping, warning thresholds, comparison with an earlier period, recovery messages
and the backtest work as they do for spans and logs. A window with no points reads "no
data" and does not fire.

`telemetry_anomaly` takes the same `dataset`, `query`, `aggregate` and `field`, plus
the event anomaly's `sensitivity`, `direction` and `minValue`. Bucker scores the last
complete hour against an hour-of-week baseline. It learns that baseline from the last
four weeks of the same query. The learned threshold switches on only after four weeks
of history. Until then, the rule's reason says when it will switch on.

## Warning, recovery, percent change and fan-out

The `metric` and `telemetry` conditions accept four optional fields. A rule that leaves
them out behaves as it did before they existed.

| Field | Effect |
| --- | --- |
| `warningThreshold` | A second line on the same comparator. It must be crossed before `threshold` is. Crossing it fires a **warning**. Crossing `threshold` fires **critical**, even inside the cooldown, because the alert has escalated. Going from critical back to warning is recorded but not announced. |
| `notifyOnRecovery` | When a warning or critical series returns to normal, Bucker sends a **resolved** message. The message goes out even inside the cooldown. On a PagerDuty channel the incident gets a `resolve`. `telemetry_anomaly` accepts this field too. |
| `comparison` | `{ "kind": "previous_period", "offsetMinutes": 1440 }` compares the window with the same window that many minutes earlier. The value and the thresholds then count as **percent change**. If the earlier window held nothing, the rule reads "no baseline" and does not fire. |
| `groupBy` | `{ "by": "environment" }`, `{ "by": "release" }` or `{ "by": "tag", "key": "region" }` fans the rule out over that key. Each value gets its own cooldown, state and recovery. On a span or log rule, `tag` names an attribute. A rule evaluates at most 50 groups per pass, the busiest first, and its reason says when it cut the rest. |

A group that goes quiet still recovers. If an environment stops erroring completely, it
drops out of the query. Bucker still evaluates every group that is breaching or warning
in its ledger, so that environment's incident gets its resolved message.

```json
{
  "id": "metric",
  "aggregate": "event_count",
  "windowMinutes": 10,
  "comparator": "gt",
  "threshold": 200,
  "warningThreshold": 100,
  "comparison": { "kind": "previous_period", "offsetMinutes": 1440 },
  "groupBy": { "by": "environment" },
  "notifyOnRecovery": true
}
```

The webhook payload carries `severity`, `recovered`, `group` and `comparison` in
`alert.metric`. See [alert-webhooks.md](alert-webhooks.md).

## Maintenance windows and mutes

A **maintenance window** keeps alert notifications quiet during planned work. It covers
alert rules, the anomaly detector, and uptime and cron monitors, because monitors raise
their alerts through the same router. A window covers one project, or every project in
the org, including projects created later.

```
GET    /orgs/{org}/maintenance-windows
POST   /orgs/{org}/maintenance-windows        { name, startsAt, endsAt, projectSlug?, reason? }
POST   /orgs/{org}/maintenance-windows        { name, statusMaintenanceId }   # follow a status-page maintenance
POST   /orgs/{org}/maintenance-windows/{id}/end
DELETE /orgs/{org}/maintenance-windows/{id}
```

- **Suppressed, never dropped.** Bucker still records every fire a window covers. It is
  written to the alert history as an alert event with status `SUPPRESSED`, and its
  `suppression` field names the window. Nothing is delivered. The fire still counts for
  the rule's cooldown, so if a condition is still true when the window ends, the alert
  goes out when the cooldown next expires.
- **Following a status page.** A window bound to a status-page maintenance follows that
  maintenance's state. It is quiet while the maintenance runs, or while it is scheduled
  and inside its times. When the maintenance is marked complete, the quiet ends too.
- **Bounds.** A window lasts at most 14 days. A window over one project needs `write` on
  that project. A window over the whole org needs `write` on the org.

A **rule mute** silences one rule until a given time:

```
POST   /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute   { minutes | until, scope: "rule" | "me" }
DELETE /orgs/{org}/projects/{project}/alert-rules/{ruleId}/mute?scope=rule|me
```

- `scope: "rule"` mutes the rule for everyone. Its fires are recorded as `SUPPRESSED`.
  It needs project `write` and writes an `alert_rule.mute` audit row.
- `scope: "me"` keeps the rule firing for everyone else. Only your own address is left
  out of its email deliveries.
- A mute lasts at most 30 days and ends by itself. The dashboard's rule menu offers 1 h,
  4 h and 24 h. A Slack alert card offers **Mute rule 1h** and **Mute rule 4h**.

## Backtest: how often would this rule have fired?

Before you save a rule, the editor's **Backtest** panel replays it over the project's
last 7 days. The API is:

```
POST /orgs/{org}/projects/{project}/alert-rules/backtest   { "rule": { …the rule… }, "days": 7 }
```

The replay uses the live evaluator. Bucker turns the stored history into the points the
live pass would have evaluated: a metric value per group at each step, or an issue
signal for each new issue, regression, lifecycle change and event. It runs each point
through the same rule code, with the rule's own cooldown kept in memory. Nothing is
written and nothing is delivered. The call needs only `read` on the project.

The response gives the count and its denominators:

| Field | Meaning |
| --- | --- |
| `fires` | Notifications after the cooldown. This is the headline number. |
| `matches` | Evaluations that met the rule. This is what would have fired with no cooldown. |
| `suppressedByCooldown` | Matches the cooldown swallowed. |
| `evaluations` | Points or signals replayed. |
| `recoveries` | Resolved messages, for a rule with `notifyOnRecovery`. |
| `delivery.chatMessagesPerDestination` | Messages each Slack, Discord, Teams or chat-webhook destination would have received after the chat volume ceiling (12 per 5 minutes, then one rollup). |
| `fireTimes`, `firesByDay` | When the fires would have happened. |
| `approximations` | Each way the replay is coarser than the live pass. |

The replay is coarser than the live pass in a few ways, and the response lists the ones
that apply:

- A metric rule is evaluated once per window (5 to 60 minutes) rather than every minute,
  so a breach shorter than that can be missed.
- An issue's status is taken from the signal (new, regressed, ongoing), not from its
  stored history, so mutes that have since ended are not replayed.
- At most 20,000 events are replayed.
- The event `anomaly` condition is not backtested, because its baseline lives in the
  detector's stored profile. `telemetry_anomaly` is replayed hour by hour.

## Where this is defined

- The rule DSL: `code/domain/src/modules/alerts/rules.ts`.
- The telemetry builder: `code/domain/src/modules/alerts/telemetry-metric.ts`, which
  reuses `code/domain/src/modules/explore/span-sql-builder.ts`.
- The learned threshold: `code/domain/src/modules/alerts/telemetry-anomaly.ts`, which
  uses `code/domain/src/modules/anomaly/seasonality.ts`.
- Groups, comparison and the points a rule is evaluated at:
  `code/domain/src/modules/alerts/metric-measure.ts`. Warning, critical and recovery:
  `evaluateMetricRule` in `code/domain/src/modules/alerts/evaluator.ts`.