# OTLP ingest

A separate surface from the [Sentry-compatible envelope protocol](ingest-protocol.md):
Bucker also accepts OpenTelemetry traces and logs over OTLP/HTTP, so a stock OTel SDK
or collector can point at Bucker with nothing but an endpoint and a header — no
Bucker-specific exporter required.

Metrics are accepted too ([below](#metrics)); a signal Bucker does not store is not
registered, because a route that accepted data and silently dropped it would be worse than the
404 an exporter gets.

## Endpoints

```
POST /v1/traces
POST /v1/logs
POST /v1/metrics
POST /api/<projectId>/otlp/v1/traces
POST /api/<projectId>/otlp/v1/logs
POST /api/<projectId>/otlp/v1/metrics
```

The bare `/v1/*` set is what an unmodified OTel exporter reaches when
`OTEL_EXPORTER_OTLP_ENDPOINT` points at Bucker. The project-scoped set is for anyone
who prefers an explicit project in the URL; both sets run the same handler.

## Authentication and project resolution

OTLP has no DSN-shaped credential and no project segment in its own protocol, so both
are recovered from the request, in this order:

1. The `:projectId` route parameter (project-scoped pair only).
2. `Authorization: DSN <full dsn>` — the simplest option for a stock exporter, since
   it supplies the public key and the project id together.
3. The `x-bucker-project-id` header, paired with a public key from `X-Sentry-Auth`,
   `Authorization: Sentry sentry_key=<publicKey>`, or `?sentry_key=<publicKey>`.
4. A `project_id` query parameter, paired with a public key the same way.

A key that does not belong to the resolved project is refused, exactly as on the
envelope route. The recommended exporter configuration is:

```
OTEL_EXPORTER_OTLP_ENDPOINT=https://<region>.<ingest-host>
OTEL_EXPORTER_OTLP_HEADERS=Authorization=DSN <full dsn>
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

See also [Log drains](log-drains.md) for platform drains and Collector, Fluent Bit and Vector
examples.

## Encodings

Both OTLP/HTTP encodings are accepted, decided from `Content-Type`:

| Content-Type | Encoding |
| --- | --- |
| `application/x-protobuf` or `application/protobuf` | protobuf |
| `application/json`, or no `Content-Type` at all | JSON |

`http/protobuf` is OTLP's spec default and what an SDK sends unless told otherwise, so
both are first-class rather than JSON-only. An absent `Content-Type` is treated as
JSON rather than refused: a bare `curl -d @batch.json` with no `-H` is a common first
attempt, and guessing protobuf from bytes would be worse. The response is encoded in
whatever was received — a protobuf request gets a protobuf response, including on
partial success — because several exporters log a deserialization warning on every
successful export otherwise.

Protobuf decoding is hand-written rather than pulled from a library: it produces the
same OTLP/JSON object shape the JSON path validates, so a protobuf batch and a JSON
batch converge on one validated object before anything downstream runs, and quota,
scrubbing and outcome accounting cannot fork between the two encodings.

### Compression

`Content-Encoding: gzip` (or `x-gzip`), `deflate` or `br` — the same bounded
decompressor the envelope route uses, so a decompression bomb is refused as `413`
(too large) rather than expanded.

## Limits

| Limit | Value | On exceeding |
| --- | --- | --- |
| Request body (received or decompressed) | 4 MiB | `413`, whole request refused |
| Records per request (spans + log records) | 5,000 | `400`, whole request refused |
| Resource wrapper objects (`ResourceSpans`/`ResourceLogs`) | 512 | `400` (schema validation), whole request refused |
| Scope wrapper objects, per resource (`ScopeSpans`/`ScopeLogs`) | 512 | `400` (schema validation), whole request refused |
| Scope wrapper objects, total across the request | 4,096 | `400` (schema validation), whole request refused |

The wrapper ceilings exist because a wrapper is allocated and walked whether or not it
holds any records — a batch of a million empty wrappers costs the same CPU as a
million empty spans, with no per-record charge to catch it. The per-resource and
total scope caps are enforced identically on both encodings, so an exporter cannot be
accepted on `http/json` and refused on `http/protobuf` for the same batch.

Unknown fields — either encoding — are always skipped rather than refused, because
`opentelemetry-proto` adds fields in minor releases and an exporter one version ahead
must not start failing.

## Responses

Success is an empty JSON object (or a zero-byte protobuf body) unless some records
were rejected on content grounds, in which case the OTLP partial-success shape is
used:

```jsonc
{ "partialSuccess": { "rejectedSpans": 3, "errorMessage": "…" } }
```

```jsonc
{ "partialSuccess": { "rejectedLogRecords": 1, "errorMessage": "…" } }
```

**Durable by default.** `/v1/traces`, `/v1/logs` and `/v1/metrics` (and the platform log
drains) store the decoded batch as a queue job and answer before any row is written; a worker
then maps and writes it. Success is therefore the empty object even when some records will turn
out to be unusable: those records are counted as `invalid` outcome rows (the Drops page), not
returned as `partialSuccess`. A body that cannot be decoded, a batch over the per-request record
ceiling and a quota refusal are decided before anything is stored and are still answered in the
request. Self-hosters can set `INGEST_OTLP_DURABLE=false` to write inside the request, as before,
and get the partial-success shape back.

An exporter logs a partial success and does **not** retry, which is correct for
records refused on content grounds — retrying would only reproduce the same
rejection. Authentication, rate-limit and decompression failures instead raise the
ordinary HTTP error statuses (`400`, `401`, `413`, `429`) described in
[ingest-protocol.md](ingest-protocol.md#responses); those are transport failures,
which an exporter should retry, and partial success is not overloaded to mean both.

## Rate limits and quota

Ingested through the same auth, rate-limiting and quota machinery as the envelope
route: a coarse per-IP ceiling ahead of any database lookup, then a per-key token
bucket keyed on the resolved DSN key (not a separate per-IP limit — a NAT egress or a
collector fronting a fleet is not an attacker). Over quota, traces and log records
degrade to stratified sampling rather than an outright block, exactly as ingest does
for Sentry events, and every drop is recorded against a data category
(`transaction` for spans, `log` for log records) the same way.

## Exceptions recorded on spans

An OpenTelemetry SDK's `recordException` adds an `exception` event to the span, carrying
`exception.type`, `exception.message` and `exception.stacktrace`. Each distinct exception on a
**stored** span becomes an occurrence in the same issue stream the logs path and the envelope path
feed, linked to its trace, span, service, release and environment. The message and stack are
scrubbed at the stored tier before anything is written, and symbolication is not attempted (OTLP
carries no Debug IDs); the event says so in `extra.symbolication: "not_attempted"`.

- **One occurrence per trace, span and exception type.** A log record carrying the same
  `exception.type` on the same trace and span is not a second occurrence of a span exception that
  opened it within the last minute, and the other way round. The log record is still stored and
  links to the issue. Two log records with one triple remain two occurrences.
- **At most 100 grouped exceptions per request.** Each is an `error` outcome at one unit. Past the
  cap the spans are still stored; the rest are counted as a `rate_limited` / `error` outcome with the
  reason `otlp_span_exception_tail_capped`, which the drops view words as "stored, not lost" (the log
  twin is `otlp_exception_tail_capped`). Nothing needs resending.
- **Error spans with no exception event** open an issue only when the project's performance setting
  `error_span_issues` is on (`PUT .../performance-settings` with `{"errorSpanIssues":{"enabled":true}}`,
  off by default). They group by service, endpoint and status, so a thousand 500s on one route are one
  issue.
- Re-sent batches and redelivered durable jobs do not duplicate occurrences: an occurrence's event id
  is derived from the project, trace, span and type.

## What this document does not cover

Grouping an exception-carrying log record into an issue reuses the same
`ingestNormalizedEvent` + essence pipeline the envelope path uses — see
[error-essence.md](error-essence.md) — rather than a second one for OTLP. This
document is the wire layer only: encodings, auth, limits, and what a refusal looks
like.

## Metrics

`POST /v1/metrics` takes `ExportMetricsServiceRequest` in the same two encodings, with the same
authentication, compression, rate limit and size limits as traces and logs. Gauge, sum, histogram,
exponential histogram and summary points are stored one row per data point and are queried in
[Explore](explore.md) as the **Metrics** dataset. There is no Bucker metrics SDK and no metrics API
to learn: you keep exporting with OpenTelemetry.

```
OTEL_EXPORTER_OTLP_ENDPOINT=https://<region>.<ingest-host>
OTEL_EXPORTER_OTLP_HEADERS=Authorization=DSN <full dsn>
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
```

- **Exemplars link to traces.** A data point's first exemplar with a trace id becomes the row's
  `trace_id` (all of them, up to five, are kept), so a metric spike is one click from the trace
  that produced it. Enable exemplars in your SDK (`OTEL_METRICS_EXEMPLAR_FILTER=trace_based`).
- **Cardinality is capped, and the cap is reported.** Each metric may have 1,000 distinct series
  (one per distinct set of attributes) over a rolling day of arrival. A data point belonging to a
  series past that is dropped and counted on the drops panel under `metric_series_cap`; series
  that already exist keep reporting. At most 200 distinct metric names per request
  (`metric_name_cap`). The fix for either is to remove the unbounded attribute before export.
- **Priced as logs.** A data point weighs what a log record weighs (0.05 units) and shows on the
  usage page and the drops panel as its own `metric` category, apart from logs. There is no
  separate metrics meter: it is one weight in the one budget, and a histogram point costs one
  point whatever its bucket count. Over quota, points are sampled the way log records are.
- **PII scrubbing** runs on attributes, the metric name and the unit before anything is stored,
  as for spans and logs.
- **Retention** follows the plan's window, like the other telemetry.
- **Units come from the name when the sender sent none.** A point that carries a `unit` keeps
  it, whatever its name says. A point with no unit gets one from the metric name's suffix, by the
  OpenTelemetry and Prometheus naming conventions: `_seconds` `s`, `_milliseconds` `ms`,
  `_microseconds` `us`, `_nanoseconds` `ns`, `_minutes` `min`, `_hours` `h`, `_days` `d`, `_bytes`
  `By`, `_kilobytes` `kBy`, `_megabytes` `MBy`, `_gigabytes` `GBy`, `_bits` `bit`, `_percent` `%`,
  `_ratio` `1`, `_celsius` `Cel`, `_hertz` `Hz`, `_meters` `m`, `_volts` `V`, `_amperes` `A`,
  `_watts` `W`, `_joules` `J`. A Prometheus `_total` suffix is read past, so
  `process_cpu_seconds_total` is `s`, and a counter whose name is only `<things>_total` is `1`. A
  name that ends in none of these keeps no unit. Unit is part of a series' identity, so a metric
  that gains an inferred unit starts a new series the first time it is seen with one.
- **Start times are honoured.** `startTimeUnixNano` on a cumulative point is stored, and a change
  in it between two points of one series is read as a reset by the cumulative differencing, so a
  restarted process is not a negative rate. A start time after the point's own timestamp is
  ignored (the point is kept; it is never read as a reset).
- **Old points are refused, and counted.** A point whose UTC day is already past its retention
  class window would be removed by the next sweep, so it is not stored and not billed. It is
  counted on the drops panel as `invalid` with the reason `metric_point_expired`; the rest of the
  request is stored. Any timestamp inside the window is accepted, so a backfill within it works.
- **Partial success.** A point with no timestamp, no finite value, or an unreadable bucket layout
  is rejected and reported as `rejectedDataPoints`; the rest of the request is stored.
- **Alerts** do not read metrics yet. A metric alert needs a rate/delta semantic per `kind` and
  temporality that Explore's per-point aggregates deliberately do not have.

## Collector recipes

Bucker accepts OTLP over HTTP, with `http/protobuf` or `http/json`, and nothing else on the wire:
no gRPC listener, no Prometheus remote-write endpoint, no StatsD port. Everything else reaches
it through an OpenTelemetry Collector (the upstream `otelcol` or `otelcol-contrib` distribution)
running next to the source. Each recipe below is a complete file in
[`collector-recipes/`](collector-recipes/), and a test checks that each parses and that every
pipeline names only components it defines. The test does not run `otelcol validate`: run that on
your own copy before you deploy it.

All three use the same exporter. `BUCKER_OTLP_ENDPOINT` is `https://<region>.<ingest-host>` with no
path (the exporter appends `/v1/metrics` and the other signal paths), and `BUCKER_DSN` is the
project's full DSN, sent as `Authorization: DSN <dsn>`. Keep the DSN in the environment, never in the
file you commit.

| Source | Recipe |
| --- | --- |
| An application that exports OTLP over gRPC | [`grpc-sidecar.yaml`](collector-recipes/grpc-sidecar.yaml) |
| A Prometheus `/metrics` endpoint | [`prometheus-scrape.yaml`](collector-recipes/prometheus-scrape.yaml) |
| StatsD datagrams | [`statsd.yaml`](collector-recipes/statsd.yaml) |
| A Prometheus server in agent mode | the two files below |

### gRPC through a sidecar, with batches under 4 MiB

Run the Collector as a sidecar and point the application at `127.0.0.1:4317`. The batch processor
counts items, not bytes, while Bucker refuses a body over 4 MiB (`413`) and a request over 5,000
records (`400`), in both cases refusing the whole request. The recipe sends at most 2,500 items and
compresses with gzip, which keeps ordinary telemetry far under the byte limit; lower
`send_batch_max_size` if your records are large (long log bodies, wide span attributes). A
refused request is not retried by an exporter, so the ceiling is something to set, not to hit.

### A Collector scraping Prometheus

The `prometheus` receiver scrapes `/metrics` and the same exporter forwards it. Counters arrive as
cumulative monotonic sums, histograms as explicit-bucket histograms, and units are inferred from
the names (`_seconds`, `_bytes`). Every distinct label set is a series, so scrape what you need.

### Prometheus in agent mode, remote-writing to a Collector

Prometheus in agent mode keeps no local storage and remote-writes. Point it at a Collector with the
contrib `prometheusremotewrite` receiver, and the Collector forwards over OTLP/HTTP:

```yaml
# prometheus.yml
global:
  scrape_interval: 30s
scrape_configs:
  - job_name: app
    static_configs:
      - targets: ['127.0.0.1:9100']
remote_write:
  - url: http://127.0.0.1:19291/api/v1/write
    protobuf_message: io.prometheus.write.v2.Request
```

Start Prometheus 3 with `--agent`. The `protobuf_message` line is not optional: the
`prometheusremotewrite` receiver accepts Remote-Write 2.0 only and answers a 1.0 request (what
Prometheus sends by default) with `415`. The receiver is still at development stability in the
contrib distribution, so pin the Collector version you tested. The Collector side:

```yaml
receivers:
  prometheusremotewrite:
    endpoint: 127.0.0.1:19291
processors:
  batch:
    timeout: 10s
    send_batch_size: 2000
    send_batch_max_size: 2500
exporters:
  otlphttp/bucker:
    endpoint: ${env:BUCKER_OTLP_ENDPOINT}
    encoding: proto
    compression: gzip
    headers:
      Authorization: DSN ${env:BUCKER_DSN}
service:
  pipelines:
    metrics:
      receivers: [prometheusremotewrite]
      processors: [batch]
      exporters: [otlphttp/bucker]
```

### StatsD through the Collector

Bucker runs no StatsD server. The Collector's `statsd` receiver listens on UDP 8125, aggregates for
`aggregation_interval`, and exports OTLP metrics: counters become delta sums, gauges gauges, and
timers and histograms (with `observer_type: histogram`) exponential histograms. Datagram tags become
attributes, so the cardinality advice above applies to them as well.

Every recipe states `send_batch_size` beside `send_batch_max_size`. The Collector refuses a
configuration whose maximum is below the batch size, and the batch size defaults to 8192, so a
recipe that set only the maximum would not start.
