Agentic observability
Monitoring your AI application, in the same product as your crashes.
The thesis
Agents fail with HTTP 200 and plausible text. Every span is ok, no exception is
raised, the latency looks fine — and the user got a confidently wrong answer, or a tool
was called eleven times with the same broken arguments, or the retriever returned
nothing and the model happily made something up.
Status codes are useless as the only signal, so the signal is derived from the shape of the run.
Today error tracking and LLM observability are separate products with separate on-call
surfaces. Here they are one: an agent failure and a null-pointer exception land in the
same issue stream, with one lifecycle, one alerting path and one remediation loop.
That is enforced structurally — the detectors emit occurrences through
ingestNormalizedEvent, the same function the crash pipeline calls. There is no
parallel "agent incident" table with its own status vocabulary.
Agent issues
Each detector turns a failure shape into an occurrence. Because these occurrences have
no stack trace, they carry an explicit [agent_issue, kind, subject] fingerprint —
stable across runs, and legible in the grouping rationale rather than a hash nobody can
explain.
False positives are the failure mode. A detector that fires on healthy traffic makes the whole product unusable, so every threshold is explicit, every detector has negative tests, and none of them fire on "the model said something I dislike".
Agent issues v2: evidence, quiet windows and a register per kind (LLM-07)
Every occurrence carries a versioned evidence block, contexts.agent_issue
(agent_issue_evidence/v1), instead of the free-form bag it used to carry. It names the detector
and its version, up to five session and trace ids, the step, model and prompt version, the
counters that made the detector fire, the day's cost per completed session (from the AI usage
rollup; empty when the rollup is off, never estimated from spans) and a validation recipe, the
frozen case or rate window that would show the fix. issue_essence renders it as the last
[AGENT] breadcrumb, labeled and scanned like the rest of the essence; ai_session_get serves
it on each finding without any string a sender chose (the model, agent and prompt are already on
the session beside it); GET .../agent/issues/:issue/evidence adds the sample traces, what ran
when the issue was first seen, and the kind's precision with n.
An issue resolves itself when its kind stops firing. The published window is 7 days AND 200
sessions of the same agent (24 hours and 20 sessions for model_call_error); the audit row for
the resolution states the days and the sessions observed since the last firing, so a quiet agent
with no traffic never looks fixed. The next firing reopens it as a regression through the
ordinary lifecycle. Spend stops (provider_cap_hit) and cost anomalies keep their own
lifecycles and are not resolved by this pass. A person's archive is never touched.
model_call_error groups failed model calls by provider, model and error class from the
hourly usage rows and fires at 5 or more failures that are at least 2% of the agent's model calls
in a complete hour, so one overloaded response never opens an issue. A tool error is not a model
call failure, and a spend stop is its own kind.
Each kind has a per-project switch and a published precision. GET .../agent/issue-kinds lists
every kind with its quiet window and its precision from human dispositions (dismissed or resolved
by a person), with the number of labels behind it. A kind whose precision falls below 0.7 over at
least 30 labels is demoted in that project by the system principal (agent_issue_kind.demote,
audited) and restored above 0.75; a project admin can demote a kind by hand, and only a person
signed in to the dashboard can turn a kind back on or off (agent_issue_policy.update). A demoted
kind opens nothing in that project and is untouched in every other; its firings are still
recorded as held detector runs, so the decision can be explained.
Guard verdicts and evaluation results (LLM-10)
A GUARDRAIL span (or any span carrying a guard verdict attribute) that denied, aborted or flagged
a call becomes a guardrail_block issue; a gen_ai.evaluation.result event or EVALUATOR span
with a failed label becomes an evaluation_failed issue. Both are fingerprinted per guard (or
evaluator) and prompt version and say "guard X blocked N of M calls on prompt v14", never that an
attack happened. The facts are counts and closed vocabularies: the verdict, the category and the
CWE it maps to (CWE-1427 prompt injection, CWE-540 system-prompt leakage, CWE-250 excessive
agency, CWE-400 unbounded consumption). The guard's reason text and the evaluation's explanation
are never read, so there is no copy of them to serve. The same facts file one SEC-02
guardrail_block finding per guard and prompt version (provenance dsn, the weakest, so it
ranks and opens an issue only through that kind's published threshold). Nothing is blocked by
Bucker.
Ingest
GenAI spans arrive over OTLP. The OTel GenAI semantic conventions are pre-stable and
have been renamed repeatedly (gen_ai.system → gen_ai.provider.name;
prompt_tokens/completion_tokens → input_tokens/output_tokens), and a large share
of already-instrumented apps emit a different vocabulary entirely — OpenInference's
llm.*/tool.*, OpenLLMetry's traceloop.*.
Hard-coding any one generation is a named risk in the design doc, so every field is read through an ordered candidate list (newest naming first, because when both are present the newer one is what the instrumentation meant) and normalized into one internal schema. Adding a fourth vocabulary is a change to one table.
Six vocabularies are read: the OTel GenAI conventions (three generations, including the
registry's own v1.38+ spellings), OpenInference, OpenLLMetry, Langfuse (langfuse.*, which
wins a conflict) and MLflow (mlflow.*), plus Bucker's own bucker.* escape hatch. Each fact
records which vocabulary it came from. genai-conformance.md lists,
per attribute key, whether it is read, aliased or ignored; it is generated from the code and a
test fails when it is stale.
Three things the contract guarantees:
- Nothing is dropped for being unfamiliar. A trace with no GenAI marker is stored as
ordinary spans (
genai: none). An unrecognisedgen_ai.*key stays raw on the span and its name (never its value) is counted per project, per day, for seven days, so you can tell instrumentation newer than Bucker from no instrumentation. - Requested tool calls are facts. The names of the calls the model asked for (at most 64 characters each, from a safe character set) and their count are read from the response before content capture strips it, so a project that wraps only the provider client still gets tool counts.
- A vocabulary cannot widen capture. Every content key a vocabulary adds is stripped under
metadatacapture; a test walks every candidate key to prove it.
One pipeline on every door
A GenAI span is handled the same way whichever door it arrives on: the agent route
(/v1/agent/traces), /v1/traces (any OTLP exporter), Sentry spans (v2 streamed spans and
transactions) and Bucker's own SDK span items (including the Python adapters). Every
GenAI span is normalised through the same tables, checked by the same six detectors,
priced by the same price book, counted in the same usage rollup, and then stripped by the
project's content policy, in that order: detection and metering see the full content in
memory and only then does the policy remove it, which is how a metadata-only project still gets
schema-invalid-tool-argument findings.
- Metadata-first capture means the same thing everywhere. Under
metadata(the default) the content keys of every vocabulary are stripped from GenAI spans on every door andbucker.content.strippedis set. An exporter that records prompts by default (the Vercel AI SDK does) and points at/v1/tracesis stripped, not stored. A door that cannot read the project's policy strips asmetadatadoes: it never stores content on a guess, and a stored value other thanfullis read asmetadata. - Plain spans are untouched. Only spans that carry GenAI facts go through the content policy, so an AI policy never edits the APM spans that share a trace.
- Tail sampling stays the agent route's. The generic doors keep their own sampling rules, so an AI policy never drops an APM span. Replay recordings stay the agent route's under full capture.
- Every GenAI span records its door as
bucker.genai.door(agent_route,otlp,sentry,sdk). It is Bucker's observation of where the span arrived: a value the sender put under that key is overwritten. - Behind a switch. On the generic doors the pass runs only while the operator has switched on
pillar.ai_usage_rollup; with it off a generic door behaves exactly as it did before.
Tiered evaluation
The tiering is an economic argument, not an architectural preference. Running an LLM judge over 100% of an agent's traffic costs more than the traffic did — so teams run judges on a 1% sample, miss the failures, and conclude evals do not work.
| Tier | Runs on | What it is |
|---|---|---|
| 1 | 100% of traffic | Deterministic. Free and instant: schema-invalid tool arguments, retry loops, empty retrieval, output-format violations, truncation. The same detectors that produce agent issues — one implementation, so an eval and a production alert cannot disagree. |
| 2 | Only what tier 1 flagged | Cheap deterministic scorer: token-overlap against expected output. No model, no network. |
| 3 | Only escalations | LLM judge. Version-pinned, and the version is recorded on every result. |
A score whose judge version is unknown is not comparable to any other score.
Judge drift
Uncalibrated LLM-as-judge is a documented failure mode with two halves that are constantly confused:
- Meta-evaluation drift — the judge is nominally the same, but the model behind it moved (a provider updated a checkpoint, a temperature default changed). Yesterday's 0.82 and today's 0.71 were measured with different rulers.
- Evaluator replacement ambiguity — the team upgraded the judge deliberately, and now nobody can say whether a score drop is the new judge being stricter or the system actually getting worse.
The module distinguishes them and attributes drift rather than reporting one number and letting you guess.
Cost
The customer-facing cost engine prices your AI application from your GenAI spans, rolled up by org, project, agent, feature and user, with cost anomalies raised as ordinary issues.
It is deliberately not the same code as our internal spend meter (which prices our own remediation agents against agent budgets). Different inputs, different consumers, different lifetimes — collapsing them would tie our internal budget arithmetic to a pricing table that changes whenever a vendor announces something.
Only inference spans count toward Bucker's own meter. Tool calls, retrieval and agent orchestration spans cost nothing. An embedding call is your provider's cost, so it is priced on its input tokens from the price book, while Bucker's meter still keeps it at weight 0.
The span kinds workflow, plan, skill, guardrail, evaluator, prompt and decision
are read and counted but never priced.
Cache tokens are priced by vocabulary, not by provider. The OTel registry says
input_tokens already includes cache reads and cache writes, for every provider. A span whose
cache counts arrive under the registry names (gen_ai.usage.cache_read.input_tokens,
cache_write.input_tokens, cache_creation.input_tokens) is inclusive: both are subtracted
before the input rate applies. Any other spelling (the underscore names, OpenInference,
Langfuse, MLflow) is exclusive and keeps the provider's own rule, and a span with no cache
count is unknown. Each priced span records which rule applied, so a reconciliation against a
vendor invoice can say why.
The usage rollup, and what a cost figure is made of
Ingest writes what every GenAI span that reached Bucker cost and did into three small tables, in memory where the batch is priced, before tail sampling, the quota sampler and the content policy:
| Table | Key | Kept |
|---|---|---|
ai_usage_days |
project, environment, UTC day, model, provider, service, agent, feature, prompt key and version, flag key and variant, release | the plan's retention ceiling |
ai_usage_hours |
the same without release, for intraday reads | 14 days |
ai_subject_days |
project, day, subject, feature | 35 days, and erased with the subject |
A row holds inference spans and step spans (every other GenAI kind, counted and weighted 0),
tokens by type, micro-USD with the price table version and the price source, unpriced spans,
errors by class (a closed list: rate_limit, timeout, auth, context_length,
content_filter, provider_unavailable, invalid_request, tool_error, other), sessions
started and sessions that reached a terminal state, agent issues by kind, and spans seen
against spans stored. At most 5,000 keys per project per day; a key beyond that lands in an
__overflow__ row that keeps environment, model and provider and replaces every other label, so
a feature label that carries an id cannot explode the table and every span is still counted. A
full day holds at most 50 such overflow rows; past that, a key lands in the one row whose every
label is __overflow__ (a model name can carry an id too). The
subject row holds only a per-project pseudonym of the end-user id, never the id.
Every cost figure states its basis, and the two are never mixed in one number:
rollup_unsampled: summed from the rollup rows. Exact for every GenAI span that reached Bucker, however many traces tail sampling kept. Bucketed: a window that starts in the last 13 days is read from the hour rows, an older one from the day rows, and the report'sfromandtoare the span actually summed (the window widened to whole hours or days), not the range asked for. Available for the agent, feature, model and provider dimensions, and only for a window that starts at or after the project's first rollup row.stored_sampled: summed from stored spans. It carriessampling(the number of spans read, what they stand for, the keep rate and how many carried no rate) andtruncated. Every span kept on the agent route now records the rate it was kept at: the baseline rate times the quota gate's rate for a healthy trace, the gate's rate for a failing one. Theuserandsessiondimensions, and any window the rollup does not cover, are read this way.
With the switch off nothing is written and every cost read says stored_sampled.
The meter. With the rollup on, the agent route charges 0.2 units per inference span and weights tool, retrieval, agent, workflow and embedding spans 0 (they are stored and counted in the rollup's step column). Without it every kept span is charged as before.
The price book
Every cost figure says where its price came from and which version of the table priced it.
priceSource |
Meaning |
|---|---|
org_contract |
Your own negotiated rate, an ai_prices row you added for the model. |
global |
A row from Bucker's price book, maintained per model family in code review, each with a source URL. |
ingested |
A cost your app or gateway sent with the span (gen_ai.usage.cost, in USD). It wins over any table. |
default |
No row matched. The span is unpriced: tokens are counted, and the cost is a range from zero up to the default rate ($3 / $15 per million tokens). |
A rollup spanning several sources says mixed and lists them. Every figure also carries
unpricedShare (unpriced spans over costed spans; null, never a bare 0, when nothing was costed)
and microUsdLow, the low end of the range; microUsd is the high end. The dashboard's cost tab
shows the range, the source, the table version and the share.
What a row holds. Input, output, cache-read, cache-write at 5 minutes and at 1 hour,
reasoning and embedding rates; batch and priority multipliers (applied when the span reports that
service tier); length tiers (above a prompt size the whole request reprices); the tokenizer note and
the source URL. Reasoning tokens are a subset of output tokens: with a reasoning rate they are billed
at it, otherwise as output. The cache rule (inclusive, exclusive, unknown) is the one above.
Effective dating. A row applies from effectiveFrom up to, not including, effectiveTo, and a
span is priced by the row in force at its own start time. A contract row written today with an
effectiveFrom last month prices last month's spans under it in your next read, but nothing already
recorded changes: re-pricing past days is an explicit, separate backfill, never a side effect of
editing a price. An organization row beats a global row whenever it matches, and inside one source the
most specific pattern wins (gpt-5-mini* over gpt-5*).
Your rates. GET /orgs/{org}/ai/prices lists your rows, GET .../ai/prices/effective?at= shows what
prices each model at an instant, POST adds a row (closing the previous open-ended row of the same
pattern) and DELETE removes one and re-opens what it closed. Writing a row is a person's act for now
and is audited with the row before and after. Rows are yours alone: another organization never sees
them, and they never price its traffic.
Unpriced models. A daily check records, per project, which models in the last 24 hours matched no
row and how many spans each cost at the default rate (instrumentation_findings, check
llm.unpriced_models). Add a contract row, or tell us, and the finding clears on the next pass.
Prompts and trace replay
Prompt versions are tracked, and traces can be replayed against a different prompt or model version — which is how "did the prompt change break this, or did the model?" becomes answerable rather than a debate.
Content policy
Prompts and completions are the most sensitive payload the platform accepts. Capture is governed by a per-project policy rather than captured by default. The policy applies on every door a GenAI span can arrive on (see "One pipeline on every door").
Where it lives
domain/src/modules/agent-observability/ — agent-issues, genai-ingest, genai-pipeline,
ai-usage-rollup, evals, judge-drift, cost, prompts, trace-replay, content-policy.
Not built
- No agent-simulation or synthetic-traffic testing.
- No cross-tenant benchmark ("your agent vs the median agent").
- Tier-3 judge results are recorded with their judge version, but there is no automatic re-scoring of history when a judge version changes.
Tools and models
The AI view's Tools and Models tabs list, for the chosen window, each tool
(execute_tool spans) and each model (chat spans, keyed on the response model): calls, error
rate (a failed span, or a tool span that set gen_ai.tool.error), p50 and p95 latency, and cost
by the same arithmetic as the Cost tab. Rows are sorted failing-first
(GET /orgs/:org/projects/:project/agent/breakdowns?from=…&to=…).
Your own MCP server
wrapMcpServer instruments a server built on the official TypeScript MCP SDK, without importing
it, and emits spans in the OpenTelemetry MCP vocabulary: one per tools/call and per
resources/read, and one mcp.connection span per connect(). A tool that throws, or returns
isError: true, counts as failed. Arguments and results are not recorded unless you pass
captureContent: true.
import { createAgentRecorder, wrapMcpServer } from '@bucker/node/integrations/genai'
import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js'
const recorder = createAgentRecorder({ dsn: process.env.BUCKER_DSN ?? '', serviceName: 'acme-mcp' })
const server = wrapMcpServer(new McpServer({ name: 'acme', version: '1.0.0' }), {
recorder,
serverName: 'acme-mcp',
})
// register tools and resources as usual; they now appear on the Tools tab
A server with its own JSON-RPC dispatcher records the same spans with
recordMcpToolCall(recorder, { name }, () => handler()) from its dispatch funnel — which is how
Bucker's own MCP server reports to its bucker-mcp project when BUCKER_DSN_MCP is set.
Reading agent runs over MCP (LLM-03)
Seven reads in the ai toolset (a tsets claim or a grant loads it) answer the questions the AI
view answers, bounded and with every customer-written string fenced as untrusted telemetry.
| Tool | Answers | Needs |
|---|---|---|
ai_sessions_list |
Agent sessions in a window: status, agent, models, steps, cost, issues, prompt version and the door the spans arrived on. Filter by agent, status, model, prompt version, issue kind, minimum cost and recording; sort by recency or cost. It reads the newest 20,000 stored GenAI spans of the window and says when that ceiling bit. | event:read |
ai_session_get |
One session as its turn, step and tool tree with linked issues, findings and cost. include: ["transcript"] adds the newest turns, each clipped, only where content was stored; every transcript read is audited, and a metadata-only project answers not_captured. |
event:read |
ai_cost_get |
Cost and tokens over up to 90 days by one dimension, from the unsampled usage rollup, with the basis, the price-table version and the unpriced share. The user dimension is a distribution, never a list of people. |
cost:read (held) |
ai_prompts_list, ai_prompt_get |
Prompts with version counts, labels, label history and live usage; the template on request, labelled as code and still scanned for injection. | project:read |
ai_evals_list, ai_eval_run_get |
Datasets, recent runs with their gate decision, judges with their canary verdict, and one run's failing cases with clipped messages and the delta against the baseline. | project:read |
ai_cost_get is held until the SG-059 security review: cost:read is outside the scopes an MCP
token can hold today, so the tool is registered and counted but is not listed for any token the
server can mint. Every list says what it scanned and which cap bit, and a cost figure says whether
it is sampled or unsampled. The write side (ai_prompts_update, ai_prices_update) is described
under "Shipping prompts and pricing models" below.
Building evals and replaying from the dashboard
- Sessions → Create dataset from these sessions. Tick sessions and name a dataset. Each becomes
an eval case carrying the session's steps (and its final response, when content capture kept it),
checked for no agent issues and at most this many steps. Freeze it so later runs compare
against exactly these cases (
POST …/agent/eval-datasets/from-sessions). - Evals → Register judge. Key, version, model, prompt and a rubric (
key | weight | descriptionper line). A judge scores cases the ladder escalates; it never signs or certifies anything. - Prompts → New version. Save a template as the next version, optionally labelled; moving a label onto an older version is how you roll back.
- Replay. Pick a recording to run the control replay — the recording against itself, which must match — and read the bisect over the candidate replays the CLI recorded: the last prompt version or model that matched and the first that diverged. The bisect warns when the outcomes are not monotonic, because then "first bad" is a guess. Candidate replays need your agent in the loop, so they run from the CLI, not from a button.
Shipping prompts and pricing models (LLM-05, LLM-06)
A person ships production prompts in the dashboard. The production label is what live traffic resolves, so its moves are split by who may make them. A new key, a new version and a non-production label go through the prompt_label operation family: a new version is inert (no label, nothing serves it) and a credential-shaped template is refused before anyone is asked; a non-production label asks (prompt_label.update) and applies against the prompt's revision. A label that served production traffic in the last 7 days is treated as production, so it keeps asking. Moving production back to the version it held before is a rollback draft: it changes nothing and files a mitigation request (mitigation.prompt_label_rollback, at most 24 hours, reversal debt, fingerprint-rate signal) that only a person holding deploy_hotfix applies. Anything else on production is prompt_label.ship, human-only: refused to every agent and OAuth client, and done by a person with project write. An organization that wrote a policy row for prompt_label.ship gets dual control (the human-requester path); otherwise the ship is the person's own act. Every production move writes a PromptLabelEvent and a ChangeEvent(PROMPT_LABEL) and opens a 24-hour watch. MCP: ai_prompts_update. REST: POST …/prompts/:key/labels, POST …/prompts/:key/rollback-drafts.
A price needs a source. ai_prices_update (the ai_price family) adds an effective-dated row to the organization's price book only with a cited https URL or invoice reference; it always asks, and the plan carries the 30-day cost delta (an estimate from daily token totals) and the anomalies it would open or clear. The inverse is the previous row. ai_usage.backfill re-prices at most 90 days of past usage as its own act; tiered rows are skipped and counted. REST for people: POST /orgs/:o/ai/prices/backfill/preview and …/backfill.