LLM observability

What is LLM observability?

LLM observability is being able to follow, understand and troubleshoot an AI application through its whole life. Not just whether it is up, but what it did on a given request and whether the answer was any good.

The reason it needs its own name is that a language model breaks differently from ordinary software. Normal software fails loudly. It throws an error, returns a 500, or stops. A language model almost never does that. It gives you a fluent, confident answer that happens to be wrong, and your monitoring dashboard shows a healthy green because the request returned a 200 in 900 milliseconds.

So the question shifts. With a database you ask whether it responded. With an AI application you have to ask whether it responded correctly, whether it used the right source, whether it stayed within the task, and whether it did anything it should not have done.

The three layers you need

Microsoft groups this into three capabilities that work together, and that split is a good way to think about it whatever tooling you use.

Tracing records the execution of a single request. Which model was called, with which prompt, which tools it used, what those tools returned, and how the agent got from the question to the answer. In an agent that runs eight steps, this is the only way to find out which step went wrong.

Evaluation judges the quality of the output itself. This is what a normal monitoring stack cannot do, because there is no error to catch. You run a set of evaluators against answers and get scores back on things like whether the answer is grounded in the source material, whether it is relevant to the question, and whether it is coherent.

Monitoring is what runs in production once the thing is live. Token consumption, latency, error rates and quality scores over time, with alerts when output falls below a threshold or something harmful gets through.

The useful part is how they connect. Monitoring tells you something changed. Evaluation tells you what got worse. Tracing tells you why. Skip one of the three and you are left with a gap you will notice at an unpleasant moment.

The standard underneath

There is an open standard for this, which matters if you would rather not be locked into one vendor's dashboard.

OpenTelemetry, the same project behind tracing in ordinary software, has semantic conventions for generative AI. They define what a span looks like for a model call, an embedding call or a retrieval, and which attributes go on it. You get names like gen_ai.operation.name for the kind of operation, gen_ai.provider.name for the provider, gen_ai.request.model for the model being called, and gen_ai.usage.input_tokens and gen_ai.usage.output_tokens for consumption.

Because those names are agreed, a trace from one framework can land in the same tooling as a trace from another, and your token cost per request becomes a number you can group and filter like any other metric. Microsoft Foundry builds its tracing on OpenTelemetry and feeds it into Application Insights, and it supports traces from several frameworks including LangChain, LangGraph and the Microsoft Agent Framework.

One caveat worth knowing before you build a lot on top of it: these conventions are still marked as being in development, so attribute names can still change.

What you measure on an agent

Once your application stops being a single model call and becomes an agent, the questions change, and so do the measurements.

For answers grounded in your own documents, there are three things to look at. Is the answer actually supported by the material that was retrieved. Did the retrieval bring back the right thing in the first place. And is the answer relevant to the question that was asked.

For an agent that takes actions, you look at different things. Did it call the right tool with the right arguments. Did it stay within the task it was given, including the rules and constraints. Did it work out what the user actually wanted. Did it finish.

There is one more measurement people forget: cost per successful outcome. An agent that answers correctly nine times out of ten but spends four times as many tokens getting there is not obviously better than one that answers correctly eight times out of ten. You cannot have that conversation without the token attributes on your traces.

What to watch out for with LLM observability

Prompts and answers are data you are now storing. A full trace contains what the user typed and what the model answered, which can easily include personal data or something commercially sensitive. Decide what you keep, for how long, and who can look at it, before you switch on full logging.

Do not evaluate everything in production. Running an evaluator on every single answer means paying for a second model call on every request. Sampling is normal, and it is enough as long as the sample is genuinely random rather than whatever was cheap to catch.

An evaluator is a model too. Automated scoring usually comes from a language model, which means it has its own biases and its own error rate. Treat a score as a signal that something deserves a look, not as a verdict.

Traces without an evaluation set only explain the past. They tell you what happened on a request that already went out. To know whether a change makes things better, you need a fixed set of test questions with expected answers that you can run before you deploy.

Build it before you go live, not after. Retrofitting tracing onto an agent that is already in production means you have no baseline, and no way to prove whether last month was better or worse.