# Uptime monitoring

An uptime monitor probes a public URL on an interval and opens an issue when it has failed enough
times in a row to rule out a blip. A monitor opens one issue per outage, adds nothing while it stays
down, and resolves the issue when it recovers. A TLS certificate close to expiry is its own issue,
because a site can be up with a certificate nine days from ending the party.

Create one in the console under **Monitors → Uptime**, or with `POST
/orgs/{org}/projects/{project}/uptime-monitors`.

## What counts as a failure

Every probe is judged in this order, and the first thing that fails names the failure:

1. The request itself: a timeout, a refused connection, too many redirects.
2. **Expected status** (default 200). Skipped when you assert on the status code yourself (below).
3. **Expected body contains** (optional substring).
4. **Assertions** (optional): the structured checks below.

`failureThreshold` (default 3) consecutive failures open the issue; `recoveryThreshold` (default 2)
consecutive successes resolve it.

## Assertions

A 200 with `{"status":"degraded"}` is down, and a substring cannot say so reliably. An assertion is
a check on the response, and they combine with **all**, **any** and **not**:

```json
{
  "all": [
    { "source": "json", "path": "$.status", "op": "eq", "value": "ok" },
    { "source": "latency_ms", "op": "lt", "value": 800 },
    { "not": { "source": "header", "name": "x-maintenance", "op": "exists" } }
  ]
}
```

| `source` | Fields | `op` |
| --- | --- | --- |
| `status` | `value` (100-599) | `eq` `neq` `lt` `lte` `gt` `gte` |
| `latency_ms` | `value` (ms) | `lt` `lte` `gt` `gte` |
| `header` | `name`, and `value` unless the op is `exists` or `not_exists` | `exists` `not_exists` `eq` `neq` `contains` |
| `json` | `path`, and `value` unless the op is `exists` or `not_exists` | `exists` `not_exists` `eq` `neq` `lt` `lte` `gt` `gte` `contains` |
| `body` | `value` | `contains` `not_contains` |

- **JSON paths** are `$`, `.key`, `['key']` and `[0]`: no wildcards, filters or recursive descent.
  The body is read up to 64 KiB; a longer body is cut, and a path into the cut part reports that.
- **Combinators** nest up to four deep and 25 assertions in total.
- **No regular expressions.** A pattern you supply would run against a response your server
  controls, which is a denial-of-service waiting for a bad pattern.
- A tree that asserts on `status` **replaces** the single expected status, so "any 2xx" or "200 or
  204" can be said. Otherwise the expected status and body checks still apply.

When one fails, the check's error names the assertion and what was seen: `Assertion failed: json
$.status = "ok", got "degraded"`. It appears in the result list and on the outage issue. What was
seen comes from your server's response, so it is shortened and shown as text.

Set `assertions` to `null` on a `PATCH` to remove them. The console edits flat lists (all of / any
of, each row optionally negated); a deeper tree saved through the API is shown as such and left as it
is when you save the form.

## Probe locations

One probe location cannot tell "your site is down" from "our probe's network blipped". With more
than one, a monitor is **DOWN only when at least two locations have each failed their threshold**; a
single location failing is recorded against that location and pages nobody. With a single location
listed, that one is enough.

- Pick locations with `regions` (create or `PATCH`); `GET .../uptime-monitors/regions` lists what
  your deployment runs. No `regions` means the primary location only, scheduled as it always was.
- Each location probes on the monitor's own interval and keeps its own streaks, shown per monitor.
- Every result carries the `region` it was probed from.
- These are **probe** locations, not data residency. A probe stores nothing regional: every result
  is written to the one database, and the name says where the request came from, not where any data
  lives.

On Bucker's hosted service the primary location, `us`, is the only one today, so
`GET .../uptime-monitors/regions` answers `{"primary":"us","available":["us"]}` and every monitor
is probed from there. Further locations are added when customers need two-location quorum; when
they are, they appear in that list and in the monitor form's location picker, and nothing about
existing monitors changes until you list a location on them.

**Operators.** A location exists when a prober runs in it. Run a worker machine
there with `MONITORS_PROBE_REGION=<name>` (default `us`) and list every location in
`UPTIME_PROBE_REGIONS` (comma-separated) on the api, which refuses a monitor naming any other, since
a location with no prober would be a monitor that is never checked there. Only the `us` machine
probes monitors with no listed locations.

## Trace propagation

By default each probe sends a W3C `traceparent` header, so a Bucker-instrumented target continues
the probe's trace instead of starting an unrelated one. Each result records the `traceId` it sent:
search it in your tracing tools to find the target's own trace for that check. Turn it off with
`propagateTrace: false` if the target rejects unknown headers; a `traceparent` you set in the
monitor's own headers wins.

## Suggested monitors

**Monitors → Uptime → Suggest monitors from my errors** lists the hosts your recent errors were
requesting, most frequent first. It **suggests and never creates**: choosing one opens the ordinary
create form with the URL filled in, and nothing exists until you save it. Hosts that already have a
monitor are left out, and so are internal names and private addresses. (The list comes from error
events, which anyone holding a DSN can write; treat it as a convenience, not as evidence.)

## Limits

- **Targets must be public.** A probe starts from Bucker's network, so a monitor can watch only an
  absolute `http://` or `https://` address that the public internet can reach, with no credentials in
  it. `localhost`, internal-only names and private network addresses are refused when you save, and a
  name that resolves to a private address is refused before each probe, at every redirect too. For a
  service that is not reachable from the internet, a [cron monitor](crons.md) that the service checks
  in to is the alternative.
- **At most 100 uptime monitors per project**, disabled ones included. The next create is refused:
  "This project already has 100 uptime monitors; delete one before creating another".
- **At least 30 seconds between probes.** `intervalSeconds` runs from 30 to 86,400 and defaults to 60;
  `timeoutMs` runs from 500 to 60,000 and defaults to 10,000.

## On a status page

A [status page](status-pages.md) component can be bound to an uptime monitor, and the monitor then
decides the component's status: by default **Major outage** while it is DOWN and **Operational** while
it is UP. With `degradedAboveLatencyMs` on the binding, a monitor that is up but averaged slower than
that over its last five checks shows **Degraded performance**. A monitor that has not run yet, is
disabled or was deleted contributes nothing, so its component shows **No data**, never Operational.

**Monitors → Uptime → Show these on a status page** appears once the project has an uptime monitor. It
opens the status page setup with this project selected, where each web address you add becomes an
uptime monitor in the project, a component on the page, and the binding between them in one request.

## Owner and mute

Uptime monitors take the same `ownerTeamId` / `ownerPrincipalId` and `mutedUntil` as
[cron monitors](crons.md): a `notify_owner` alert action routes the monitor's issues to its owner,
and a mute (up to 30 days) holds the alerts back while the monitor keeps checking. Muted alerts are
recorded as *suppressed* in the alert history, naming the mute.