AI Observability: What to Log When Your Product Is Nondeterministic

The Short Answer

Log the version quadruple on every request: response model, prompt version, retrieval index version, and code SHA. Add token counts including cache reads, retrieved document IDs, tool error count, schema-valid boolean, stop reason, and conversation id. Capture full prompts and completions only on a tail-sampled subset that keeps every error and every slow trace. Most production LLM incidents are deterministic version-skew bugs misfiled as model randomness.

"Nondeterministic" Is Usually the Wrong Diagnosis

Teams treat output variance as the thing to instrument. The variance is real but small, and the incidents that cost money are ordinary system bugs that nothing in the trace pins down.

Thinking Machines Lab measured the variance directly. Sampling 1,000 completions from Qwen3-235B at temperature 0 with the same prompt produced 80 unique completions, all identical through token 102 before diverging at token 103. Their stated cause is lack of batch invariance: inference endpoints are nondeterministic because load, and therefore batch size, varies. That is a kernel property, not a product defect.

You can buy bit-exactness and you should decline. LMSYS reports SGLang's deterministic inference costing 27.2% to 35.7% end-to-end on FlashAttention 3 and 42.6% to 46.0% on FlashInfer, and its authors recommend the mode primarily for debugging and reproducibility rather than production serving. A third of your throughput is a bad price for removing the smallest source of variance in the system.

Make the harness replayable instead. Pin the prompt, the model id, the sampling parameters, the retrieved document IDs, and the tool outputs, and you can re-run every stage of the pipeline except the sampler. That is where the bugs live anyway.

Now look at what broke in production. Anthropic's September 2025 postmortem on three overlapping infrastructure bugs describes a context-window routing error introduced on August 5 that, in the worst affected hour on August 31, hit 16% of Sonnet 4 requests and was not fixed until September 4. A month of degraded traffic. That is not sampling variance. That is a request landing on a server the caller never asked for, and one logged attribute would have shown it.

The Four Fields That Answer "The Answers Got Worse"

When a user reports degradation, you need to answer four questions in one query: which model version served it, which prompt version, which retrieval index version, and which code SHA. If you cannot, stop building evals and add those four attributes to every span first.

The model field is the load-bearing one. Log the requested model and the responded model separately. OpenTelemetry's GenAI semantic conventions define both, gen_ai.request.model and gen_ai.response.model, and most instrumentation records only the first. Provider-side routing changes stay invisible until you record the second.

Two more failure shapes from the same postmortem matter here. Degradation was sticky per session, because once a request landed on the wrong server, later requests from the same user tended to follow it. Fleet-wide error rate and p95 latency stayed flat while a subset of users had a consistently bad time. Without a conversation id on every span you cannot cluster by session, and that pattern never surfaces. Anthropic also says plainly that their evaluations did not capture the degradation users reported, in part because Claude often recovers well from isolated mistakes. Aggregate pass rates absorb per-token corruption.

What to Log on 100% of Traffic

These are cheap, low-cardinality, deterministic, and they cannot lie. Put all of them on every request:

• gen_ai.response.model, separate from gen_ai.request.model
• prompt version id and retrieval index version
• code SHA or release id
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens
• cache read tokens and cache creation tokens
• retrieved document IDs, the IDs only, never the text
• tool call count and tool error count
• output schema valid, as a boolean
• finish or stop reason
• gen_ai.conversation.id

Note what is not on that list. Prompts and completions. They are the most expensive, most legally exposed, and least diagnostic part of a trace, and they are what nearly every team captures first.

One warning: conversation id, user id, and prompt text belong on span attributes, not metric labels. Attaching them to a metric multiplies your time series count and either wrecks the bill or gets the metric silently dropped by the backend cardinality limiter.

Why Payload Capture Is the Wrong First Move

OpenTelemetry's own spec treats message content as Opt-In, gated on user consent, with instrumentations expected to expose a gate such as OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT. The spec attaches an explicit warning to gen_ai.input.messages and gen_ai.output.messages that the attribute is likely to contain sensitive information including user and PII data. Retrieval spans follow the same pattern: gen_ai.retrieval.top_k is recommended, while the document text and query text are Opt-In and flagged sensitive.

The cost side is worse than most teams model. Braintrust bills observability by gigabytes of processed data rather than per span, so a 20 KB system prompt on every request makes your spend track prompt length instead of traffic. Add retrieved document text and a RAG application can spend more logging its context than generating from it.

The argument that should settle it comes from the same postmortem. Anthropic states that internal privacy and security controls prevent their engineers from examining the problematic interactions needed to identify or reproduce bugs. They had the data. They could not look at it. Capturing payloads at 100% before you have solved access control and redaction buys a compliance liability and a bill, not a debugging capability.

The counterweight is real, so do not over-correct. Cursor's support bot invented a device-limit login policy that did not exist, recorded as Incident 1039 in the AI Incident Database with an April 2025 date and coverage in Fortune and The New York Times. Users hit unexpected logouts and cancelled subscriptions. Catching that requires clustering over low-frequency assertions in model outputs, which requires payloads for some fraction of traffic. Some, not all.

Head Sampling Is a Bug in Your Default Configuration

Most LLM SDKs sample at trace start. Langfuse's sampling is head-based, using OpenTelemetry's TraceIdRatioBasedSampler configured through LANGFUSE_SAMPLE_RATE, and their docs are clear that an unsampled trace drops all of its observations and scores. Set that to 0.2 and you have discarded four out of five failures alongside four out of five successes. The traces you need most are the rare ones, and head sampling is by construction indifferent to outcome.

Move the decision into an OpenTelemetry Collector tail-sampling processor. Its policy types compose into what you actually want kept: status_code for errors, latency with a threshold_ms boundary, probabilistic for the baseline. Keep 100% of error traces, 100% of traces past a latency boundary, and a 1% to 5% probabilistic baseline for everything else.

Retained volume then tracks your error and slow-request rates rather than a flat percentage, so a healthy service keeps a low single-digit share of its traces and still keeps every trace worth an incident. Compute span metrics before the sampling decision, so your rates and percentiles stay calculated over the full population.

Count Your Billable Units Before You Instrument

Pricing shape matters more than the interface. These are list prices loaded on 2026-08-25.

The Langfuse unit definition surprises people, because every observation and every score counts, not just the trace. Doing the arithmetic on their published graduated rates: a five-observation agent trace at 1M requests per month is 5M units. On Core that is 900k units at $8 per 100k plus 4M units at $7 per 100k, about $352 of overage on the $29 base, so roughly $380 per month at list before storage. That is our calculation from their published rates, not a figure Langfuse publishes.

The widely circulated per-span Datadog LLM Observability figures do not appear on its public pricing page. Get that rate in writing from an account team rather than budgeting from a blog post.

Metrics That Change Meaning Under You

Two numbers you alert on can change definition without a code change on your side.

Anthropic's prompt caching documentation states that prompts shorter than the model minimum cannot be cached, no error is returned, and the request processes without caching. Those minimums range from 512 tokens on Opus 5 up to 4,096 on Opus 4.6, Opus 4.5, and Haiku 4.5. An innocuous edit that trims the cacheable prefix takes your input bill up 10x on that prefix with nothing in the error logs, because a cache read is priced at 0.1x base input. Alert on cache_read_input_tokens as a ratio of total input tokens and you catch it the same day.

The second is the tokenizer. Anthropic's pricing documentation notes that Claude 4.7 and later models plus Claude Mythos Preview use a newer tokenizer producing approximately 30% more tokens for the same text. Every cost-per-request alert, context-window guard, and capacity model calibrated on an older model misfires at once on cutover. Recalibrate the thresholds before you switch, not while on call.

Do Not Page on an LLM Judge

A judge is a ranker for a human review queue. It is not a monitor.

A March to April 2026 evaluation of 21 judge models across nine providers, published as arXiv preprint 2606.19544, covered roughly 541,000 judgments and found chance-corrected agreement with human labels on MT-Bench between Cohen's kappa 0.376 and 0.511. Raw exact-match scores overstated that agreement by 33.8 to 41.2 percentage points for every judge tested. Worse for anyone treating a judge score as an SLO: self-consistency does not imply validity. Qwen 3 8B scored 0.992 test-retest reliability with a position bias of 0.192. The peer-reviewed anchor is arXiv 2406.07791, accepted to AACL-IJCNLP 2025, which measured 15 judges over more than 150,000 instances and found position bias varies significantly by judge and task and grows with the quality gap between candidates.

If you run a judge anyway, store the judge model id and judge prompt version on the score itself. Otherwise a provider updating your evaluator model shifts your quality graph with no product change, and you open an incident against the wrong system.

Retention Is a Compliance Input, Not a Budget Line

If your system is high-risk under the EU AI Act, Article 12 requires it to technically allow automatic recording of events over the system lifetime, and Article 19 requires providers to keep those automatically generated logs for at least six months. A 14-day default retention, which is what LangSmith base traces and Braintrust Starter ship with, does not satisfy that.

Design redaction before you turn logging on, because you cannot retroactively un-log PII you were obliged to keep. Two related traps: OpenAI's data controls documentation states that abuse monitoring logs are generated by default for all API feature usage and retained up to 30 days, with Zero Data Retention requiring prior approval and not covering the conversations, assistants, or threads endpoints. And Langfuse's masking documentation notes the mask hook affects only spans exported by that Langfuse client, so a second OTLP exporter to your APM receives an unmodified copy.

Pick a Convention and Put a Mapping Layer Behind It

OpenTelemetry's GenAI semantic conventions now live in a dedicated repository, open-telemetry/semantic-conventions-genai, which as of 2026-08-25 has no tagged releases, and every convention document in its README is marked Development status. Standardizing on gen_ai today means building on an untagged main branch.

The competing convention, OpenInference from Arize, names the same numbers differently: llm.token_count.prompt where OpenTelemetry says gen_ai.usage.input_tokens, plus a required openinference.span.kind attribute. Dashboards written against one do not transfer. Pin the schema version and keep a thin mapping layer between your application and your dashboards, so migration is a config change instead of a rewrite.

If you run agents, add two histograms early: gen_ai.invoke_agent.inference_calls and gen_ai.invoke_agent.tool_calls. The spec's bucket boundaries run to 128 calls per invocation, which tells you what loop depth its authors expect. Loop count regresses first when a tool starts failing, and it multiplies every other cost you have.

How We Instrument at SASID

Across 7 production AI systems in 5 industries, the pattern has held. A cybersecurity plain-English query engine has run more than six months in production with zero bad queries, a claim we can make only because retrieval IDs and query outcomes were recorded on every request from day one. A customer platform processing more than 30,000 reviews a day across 200+ locations holds a sub-2-minute response SLA and moved response rate from 60% to 99.8%, numbers that are defensible only because the counters behind them are logged per request rather than sampled. On HIPAA-compliant retrieval over medical records, redaction was designed before logging was switched on.

Get a Free Technical Assessment

If you are instrumenting an LLM product or trying to explain a regression you cannot reproduce, we offer a free technical assessment: a 30-minute call about your system, followed by a written roadmap within 48 hours covering what to log, what to sample, retention and redaction, and expected cost. There is no obligation, and the roadmap is yours to keep. Book at sasid.ai.