Skip to content
bucker

Agentic observability

Monitoring your AI application, in the same product as your crashes.

The thesis

Agents fail with HTTP 200 and plausible text. Every span is ok, no exception is raised, the latency looks fine — and the user got a confidently wrong answer, or a tool was called eleven times with the same broken arguments, or the retriever returned nothing and the model happily made something up.

Status codes are useless as the only signal, so the signal is derived from the shape of the run.

Today error tracking and LLM observability are separate products with separate on-call surfaces. Here they are one: an agent failure and a null-pointer exception land in the same issue stream, with one lifecycle, one alerting path and one remediation loop. That is enforced structurally — the detectors emit occurrences through ingestNormalizedEvent, the same function the crash pipeline calls. There is no parallel "agent incident" table with its own status vocabulary.

Agent issues

Each detector turns a failure shape into an occurrence. Because these occurrences have no stack trace, they carry an explicit [agent_issue, kind, subject] fingerprint — stable across runs, and legible in the grouping rationale rather than a hash nobody can explain.

False positives are the failure mode. A detector that fires on healthy traffic makes the whole product unusable, so every threshold is explicit, every detector has negative tests, and none of them fire on "the model said something I dislike".

Agent issues v2: evidence, quiet windows and a register per kind (LLM-07)

Every occurrence carries a versioned evidence block, contexts.agent_issue (agent_issue_evidence/v1), instead of the free-form bag it used to carry. It names the detector and its version, up to five session and trace ids, the step, model and prompt version, the counters that made the detector fire, the day's cost per completed session (from the AI usage rollup; empty when the rollup is off, never estimated from spans) and a validation recipe, the frozen case or rate window that would show the fix. issue_essence renders it as the last [AGENT] breadcrumb, labeled and scanned like the rest of the essence; ai_session_get serves it on each finding without any string a sender chose (the model, agent and prompt are already on the session beside it); GET .../agent/issues/:issue/evidence adds the sample traces, what ran when the issue was first seen, and the kind's precision with n.

An issue resolves itself when its kind stops firing. The published window is 7 days AND 200 sessions of the same agent (24 hours and 20 sessions for model_call_error); the audit row for the resolution states the days and the sessions observed since the last firing, so a quiet agent with no traffic never looks fixed. The next firing reopens it as a regression through the ordinary lifecycle. Spend stops (provider_cap_hit) and cost anomalies keep their own lifecycles and are not resolved by this pass. A person's archive is never touched.

model_call_error groups failed model calls by provider, model and error class from the hourly usage rows and fires at 5 or more failures that are at least 2% of the agent's model calls in a complete hour, so one overloaded response never opens an issue. A tool error is not a model call failure, and a spend stop is its own kind.

Each kind has a per-project switch and a published precision. GET .../agent/issue-kinds lists every kind with its quiet window and its precision from human dispositions (dismissed or resolved by a person), with the number of labels behind it. A kind whose precision falls below 0.7 over at least 30 labels is demoted in that project by the system principal (agent_issue_kind.demote, audited) and restored above 0.75; a project admin can demote a kind by hand, and only a person signed in to the dashboard can turn a kind back on or off (agent_issue_policy.update). A demoted kind opens nothing in that project and is untouched in every other; its firings are still recorded as held detector runs, so the decision can be explained.

Guard verdicts and evaluation results (LLM-10)

A GUARDRAIL span (or any span carrying a guard verdict attribute) that denied, aborted or flagged a call becomes a guardrail_block issue; a gen_ai.evaluation.result event or EVALUATOR span with a failed label becomes an evaluation_failed issue. Both are fingerprinted per guard (or evaluator) and prompt version and say "guard X blocked N of M calls on prompt v14", never that an attack happened. The facts are counts and closed vocabularies: the verdict, the category and the CWE it maps to (CWE-1427 prompt injection, CWE-540 system-prompt leakage, CWE-250 excessive agency, CWE-400 unbounded consumption). The guard's reason text and the evaluation's explanation are never read, so there is no copy of them to serve. The same facts file one SEC-02 guardrail_block finding per guard and prompt version (provenance dsn, the weakest, so it ranks and opens an issue only through that kind's published threshold). Nothing is blocked by Bucker.

Ingest

GenAI spans arrive over OTLP. The OTel GenAI semantic conventions are pre-stable and have been renamed repeatedly (gen_ai.system → gen_ai.provider.name; prompt_tokens/completion_tokens → input_tokens/output_tokens), and a large share of already-instrumented apps emit a different vocabulary entirely — OpenInference's llm.*/tool.*, OpenLLMetry's traceloop.*.

Hard-coding any one generation is a named risk in the design doc, so every field is read through an ordered candidate list (newest naming first, because when both are present the newer one is what the instrumentation meant) and normalized into one internal schema. Adding a fourth vocabulary is a change to one table.

Six vocabularies are read: the OTel GenAI conventions (three generations, including the registry's own v1.38+ spellings), OpenInference, OpenLLMetry, Langfuse (langfuse.*, which wins a conflict) and MLflow (mlflow.*), plus Bucker's own bucker.* escape hatch. Each fact records which vocabulary it came from. genai-conformance.md lists, per attribute key, whether it is read, aliased or ignored; it is generated from the code and a test fails when it is stale.

Three things the contract guarantees:

  • Nothing is dropped for being unfamiliar. A trace with no GenAI marker is stored as ordinary spans (genai: none). An unrecognised gen_ai.* key stays raw on the span and its name (never its value) is counted per project, per day, for seven days, so you can tell instrumentation newer than Bucker from no instrumentation.
  • Requested tool calls are facts. The names of the calls the model asked for (at most 64 characters each, from a safe character set) and their count are read from the response before content capture strips it, so a project that wraps only the provider client still gets tool counts.
  • A vocabulary cannot widen capture. Every content key a vocabulary adds is stripped under metadata capture; a test walks every candidate key to prove it.

One pipeline on every door

A GenAI span is handled the same way whichever door it arrives on: the agent route (/v1/agent/traces), /v1/traces (any OTLP exporter), Sentry spans (v2 streamed spans and transactions) and Bucker's own SDK span items (including the Python adapters). Every GenAI span is normalised through the same tables, checked by the same six detectors, priced by the same price book, counted in the same usage rollup, and then stripped by the project's content policy, in that order: detection and metering see the full content in memory and only then does the policy remove it, which is how a metadata-only project still gets schema-invalid-tool-argument findings.

  • Metadata-first capture means the same thing everywhere. Under metadata (the default) the content keys of every vocabulary are stripped from GenAI spans on every door and bucker.content.stripped is set. An exporter that records prompts by default (the Vercel AI SDK does) and points at /v1/traces is stripped, not stored. A door that cannot read the project's policy strips as metadata does: it never stores content on a guess, and a stored value other than full is read as metadata.
  • Plain spans are untouched. Only spans that carry GenAI facts go through the content policy, so an AI policy never edits the APM spans that share a trace.
  • Tail sampling stays the agent route's. The generic doors keep their own sampling rules, so an AI policy never drops an APM span. Replay recordings stay the agent route's under full capture.
  • Every GenAI span records its door as bucker.genai.door (agent_route, otlp, sentry, sdk). It is Bucker's observation of where the span arrived: a value the sender put under that key is overwritten.
  • Behind a switch. On the generic doors the pass runs only while the operator has switched on pillar.ai_usage_rollup; with it off a generic door behaves exactly as it did before.

Tiered evaluation

The tiering is an economic argument, not an architectural preference. Running an LLM judge over 100% of an agent's traffic costs more than the traffic did — so teams run judges on a 1% sample, miss the failures, and conclude evals do not work.

Tier Runs on What it is
1 100% of traffic Deterministic. Free and instant: schema-invalid tool arguments, retry loops, empty retrieval, output-format violations, truncation. The same detectors that produce agent issues — one implementation, so an eval and a production alert cannot disagree.
2 Only what tier 1 flagged Cheap deterministic scorer: token-overlap against expected output. No model, no network.
3 Only escalations LLM judge. Version-pinned, and the version is recorded on every result.

A score whose judge version is unknown is not comparable to any other score.

Judge drift

Uncalibrated LLM-as-judge is a documented failure mode with two halves that are constantly confused:

  • Meta-evaluation drift — the judge is nominally the same, but the model behind it moved (a provider updated a checkpoint, a temperature default changed). Yesterday's 0.82 and today's 0.71 were measured with different rulers.
  • Evaluator replacement ambiguity — the team upgraded the judge deliberately, and now nobody can say whether a score drop is the new judge being stricter or the system actually getting worse.

The module distinguishes them and attributes drift rather than reporting one number and letting you guess.

Cost

The customer-facing cost engine prices your AI application from your GenAI spans, rolled up by org, project, agent, feature and user, with cost anomalies raised as ordinary issues.

It is deliberately not the same code as our internal spend meter (which prices our own remediation agents against agent budgets). Different inputs, different consumers, different lifetimes — collapsing them would tie our internal budget arithmetic to a pricing table that changes whenever a vendor announces something.

Only inference spans count toward Bucker's own meter. Tool calls, retrieval and agent orchestration spans cost nothing. An embedding call is your provider's cost, so it is priced on its input tokens from the price book, while Bucker's meter still keeps it at weight 0.

The span kinds workflow, plan, skill, guardrail, evaluator, prompt and decision are read and counted but never priced.

Cache tokens are priced by vocabulary, not by provider. The OTel registry says input_tokens already includes cache reads and cache writes, for every provider. A span whose cache counts arrive under the registry names (gen_ai.usage.cache_read.input_tokens, cache_write.input_tokens, cache_creation.input_tokens) is inclusive: both are subtracted before the input rate applies. Any other spelling (the underscore names, OpenInference, Langfuse, MLflow) is exclusive and keeps the provider's own rule, and a span with no cache count is unknown. Each priced span records which rule applied, so a reconciliation against a vendor invoice can say why.

The usage rollup, and what a cost figure is made of

Ingest writes what every GenAI span that reached Bucker cost and did into three small tables, in memory where the batch is priced, before tail sampling, the quota sampler and the content policy:

Table Key Kept
ai_usage_days project, environment, UTC day, model, provider, service, agent, feature, prompt key and version, flag key and variant, release the plan's retention ceiling
ai_usage_hours the same without release, for intraday reads 14 days
ai_subject_days project, day, subject, feature 35 days, and erased with the subject

A row holds inference spans and step spans (every other GenAI kind, counted and weighted 0), tokens by type, micro-USD with the price table version and the price source, unpriced spans, errors by class (a closed list: rate_limit, timeout, auth, context_length, content_filter, provider_unavailable, invalid_request, tool_error, other), sessions started and sessions that reached a terminal state, agent issues by kind, and spans seen against spans stored. At most 5,000 keys per project per day; a key beyond that lands in an __overflow__ row that keeps environment, model and provider and replaces every other label, so a feature label that carries an id cannot explode the table and every span is still counted. A full day holds at most 50 such overflow rows; past that, a key lands in the one row whose every label is __overflow__ (a model name can carry an id too). The subject row holds only a per-project pseudonym of the end-user id, never the id.

Every cost figure states its basis, and the two are never mixed in one number:

  • rollup_unsampled: summed from the rollup rows. Exact for every GenAI span that reached Bucker, however many traces tail sampling kept. Bucketed: a window that starts in the last 13 days is read from the hour rows, an older one from the day rows, and the report's from and to are the span actually summed (the window widened to whole hours or days), not the range asked for. Available for the agent, feature, model and provider dimensions, and only for a window that starts at or after the project's first rollup row.
  • stored_sampled: summed from stored spans. It carries sampling (the number of spans read, what they stand for, the keep rate and how many carried no rate) and truncated. Every span kept on the agent route now records the rate it was kept at: the baseline rate times the quota gate's rate for a healthy trace, the gate's rate for a failing one. The user and session dimensions, and any window the rollup does not cover, are read this way.

With the switch off nothing is written and every cost read says stored_sampled.

The meter. With the rollup on, the agent route charges 0.2 units per inference span and weights tool, retrieval, agent, workflow and embedding spans 0 (they are stored and counted in the rollup's step column). Without it every kept span is charged as before.

The price book

Every cost figure says where its price came from and which version of the table priced it.

priceSource Meaning
org_contract Your own negotiated rate, an ai_prices row you added for the model.
global A row from Bucker's price book, maintained per model family in code review, each with a source URL.
ingested A cost your app or gateway sent with the span (gen_ai.usage.cost, in USD). It wins over any table.
default No row matched. The span is unpriced: tokens are counted, and the cost is a range from zero up to the default rate ($3 / $15 per million tokens).

A rollup spanning several sources says mixed and lists them. Every figure also carries unpricedShare (unpriced spans over costed spans; null, never a bare 0, when nothing was costed) and microUsdLow, the low end of the range; microUsd is the high end. The dashboard's cost tab shows the range, the source, the table version and the share.

What a row holds. Input, output, cache-read, cache-write at 5 minutes and at 1 hour, reasoning and embedding rates; batch and priority multipliers (applied when the span reports that service tier); length tiers (above a prompt size the whole request reprices); the tokenizer note and the source URL. Reasoning tokens are a subset of output tokens: with a reasoning rate they are billed at it, otherwise as output. The cache rule (inclusive, exclusive, unknown) is the one above.

Effective dating. A row applies from effectiveFrom up to, not including, effectiveTo, and a span is priced by the row in force at its own start time. A contract row written today with an effectiveFrom last month prices last month's spans under it in your next read, but nothing already recorded changes: re-pricing past days is an explicit, separate backfill, never a side effect of editing a price. An organization row beats a global row whenever it matches, and inside one source the most specific pattern wins (gpt-5-mini* over gpt-5*).

Your rates. GET /orgs/{org}/ai/prices lists your rows, GET .../ai/prices/effective?at= shows what prices each model at an instant, POST adds a row (closing the previous open-ended row of the same pattern) and DELETE removes one and re-opens what it closed. Writing a row is a person's act for now and is audited with the row before and after. Rows are yours alone: another organization never sees them, and they never price its traffic.

Unpriced models. A daily check records, per project, which models in the last 24 hours matched no row and how many spans each cost at the default rate (instrumentation_findings, check llm.unpriced_models). Add a contract row, or tell us, and the finding clears on the next pass.

Prompts and trace replay

Prompt versions are tracked, and traces can be replayed against a different prompt or model version — which is how "did the prompt change break this, or did the model?" becomes answerable rather than a debate.

Content policy

Prompts and completions are the most sensitive payload the platform accepts. Capture is governed by a per-project policy rather than captured by default. The policy applies on every door a GenAI span can arrive on (see "One pipeline on every door").

Where it lives

domain/src/modules/agent-observability/ — agent-issues, genai-ingest, genai-pipeline, ai-usage-rollup, evals, judge-drift, cost, prompts, trace-replay, content-policy.

Not built

  • No agent-simulation or synthetic-traffic testing.
  • No cross-tenant benchmark ("your agent vs the median agent").
  • Tier-3 judge results are recorded with their judge version, but there is no automatic re-scoring of history when a judge version changes.

Tools and models

The AI view's Tools and Models tabs list, for the chosen window, each tool (execute_tool spans) and each model (chat spans, keyed on the response model): calls, error rate (a failed span, or a tool span that set gen_ai.tool.error), p50 and p95 latency, and cost by the same arithmetic as the Cost tab. Rows are sorted failing-first (GET /orgs/:org/projects/:project/agent/breakdowns?from=…&to=…).

Your own MCP server

wrapMcpServer instruments a server built on the official TypeScript MCP SDK, without importing it, and emits spans in the OpenTelemetry MCP vocabulary: one per tools/call and per resources/read, and one mcp.connection span per connect(). A tool that throws, or returns isError: true, counts as failed. Arguments and results are not recorded unless you pass captureContent: true.

import { createAgentRecorder, wrapMcpServer } from '@bucker/node/integrations/genai'
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'

const recorder = createAgentRecorder({ dsn: process.env.BUCKER_DSN ?? '', serviceName: 'acme-mcp' })
const server = wrapMcpServer(new McpServer({ name: 'acme', version: '1.0.0' }), {
  recorder,
  serverName: 'acme-mcp',
})
// register tools and resources as usual; they now appear on the Tools tab

A server with its own JSON-RPC dispatcher records the same spans with recordMcpToolCall(recorder, { name }, () => handler()) from its dispatch funnel — which is how Bucker's own MCP server reports to its bucker-mcp project when BUCKER_DSN_MCP is set.

Reading agent runs over MCP (LLM-03)

Seven reads in the ai toolset (a tsets claim or a grant loads it) answer the questions the AI view answers, bounded and with every customer-written string fenced as untrusted telemetry.

Tool Answers Needs
ai_sessions_list Agent sessions in a window: status, agent, models, steps, cost, issues, prompt version and the door the spans arrived on. Filter by agent, status, model, prompt version, issue kind, minimum cost and recording; sort by recency or cost. It reads the newest 20,000 stored GenAI spans of the window and says when that ceiling bit. event:read
ai_session_get One session as its turn, step and tool tree with linked issues, findings and cost. include: ["transcript"] adds the newest turns, each clipped, only where content was stored; every transcript read is audited, and a metadata-only project answers not_captured. event:read
ai_cost_get Cost and tokens over up to 90 days by one dimension, from the unsampled usage rollup, with the basis, the price-table version and the unpriced share. The user dimension is a distribution, never a list of people. cost:read (held)
ai_prompts_list, ai_prompt_get Prompts with version counts, labels, label history and live usage; the template on request, labelled as code and still scanned for injection. project:read
ai_evals_list, ai_eval_run_get Datasets, recent runs with their gate decision, judges with their canary verdict, and one run's failing cases with clipped messages and the delta against the baseline. project:read

ai_cost_get is held until the SG-059 security review: cost:read is outside the scopes an MCP token can hold today, so the tool is registered and counted but is not listed for any token the server can mint. Every list says what it scanned and which cap bit, and a cost figure says whether it is sampled or unsampled. The write side (ai_prompts_update, ai_prices_update) is described under "Shipping prompts and pricing models" below.

Building evals and replaying from the dashboard

  • Sessions → Create dataset from these sessions. Tick sessions and name a dataset. Each becomes an eval case carrying the session's steps (and its final response, when content capture kept it), checked for no agent issues and at most this many steps. Freeze it so later runs compare against exactly these cases (POST …/agent/eval-datasets/from-sessions).
  • Evals → Register judge. Key, version, model, prompt and a rubric (key | weight | description per line). A judge scores cases the ladder escalates; it never signs or certifies anything.
  • Prompts → New version. Save a template as the next version, optionally labelled; moving a label onto an older version is how you roll back.
  • Replay. Pick a recording to run the control replay — the recording against itself, which must match — and read the bisect over the candidate replays the CLI recorded: the last prompt version or model that matched and the first that diverged. The bisect warns when the outcomes are not monotonic, because then "first bad" is a guess. Candidate replays need your agent in the loop, so they run from the CLI, not from a button.

Shipping prompts and pricing models (LLM-05, LLM-06)

A person ships production prompts in the dashboard. The production label is what live traffic resolves, so its moves are split by who may make them. A new key, a new version and a non-production label go through the prompt_label operation family: a new version is inert (no label, nothing serves it) and a credential-shaped template is refused before anyone is asked; a non-production label asks (prompt_label.update) and applies against the prompt's revision. A label that served production traffic in the last 7 days is treated as production, so it keeps asking. Moving production back to the version it held before is a rollback draft: it changes nothing and files a mitigation request (mitigation.prompt_label_rollback, at most 24 hours, reversal debt, fingerprint-rate signal) that only a person holding deploy_hotfix applies. Anything else on production is prompt_label.ship, human-only: refused to every agent and OAuth client, and done by a person with project write. An organization that wrote a policy row for prompt_label.ship gets dual control (the human-requester path); otherwise the ship is the person's own act. Every production move writes a PromptLabelEvent and a ChangeEvent(PROMPT_LABEL) and opens a 24-hour watch. MCP: ai_prompts_update. REST: POST …/prompts/:key/labels, POST …/prompts/:key/rollback-drafts.

A price needs a source. ai_prices_update (the ai_price family) adds an effective-dated row to the organization's price book only with a cited https URL or invoice reference; it always asks, and the plan carries the 30-day cost delta (an estimate from daily token totals) and the anomalies it would open or clear. The inverse is the previous row. ai_usage.backfill re-prices at most 90 days of past usage as its own act; tiered rows are skipped and counted. REST for people: POST /orgs/:o/ai/prices/backfill/preview and …/backfill.