LLM Observability Tools Compared

February 11, 2026 / Arvid Andersson

Once you ship LLM features to production, you need visibility into what's happening: what prompts are being sent, how much each call costs, whether quality is holding up, and where failures occur. LLM observability tools fill this gap. This post compares the leading options and what each does well.

Why LLM observability matters

Traditional application monitoring tracks request times and error rates. LLM applications add new dimensions: token costs that can spike unexpectedly, prompt quality that degrades over time, hallucinations that are hard to catch without systematic evaluation, and complex chains where a failure in one step cascades. LLM observability tools capture the full trace of each request, from prompt to completion, and give you the data to debug, optimize, and evaluate.

Open-source options

Langfuse is the most established open-source option. It provides tracing, prompt management, evaluation, and cost tracking. You can self-host it or use their cloud offering. The integration works with most LLM frameworks (LangChain, LlamaIndex, OpenAI SDK) through decorators or callbacks. The self-hosted option is attractive for teams with strict data residency requirements.

Pezzo and lunary are also open-source, with more focused feature sets. Pezzo emphasizes prompt management and versioning. Lunary focuses on agent tracing and user analytics, with built-in guardrails for content filtering.

Platform-integrated tools

LangSmith is built by the LangChain team and integrates tightly with the LangChain ecosystem. If your application already uses LangChain, LangSmith gives you tracing, evaluation datasets, and prompt playgrounds with minimal setup. The trade-off is vendor lock-in to the LangChain stack.

Braintrust takes a different angle, focusing on evaluation and experimentation. It treats LLM outputs as data that should be scored, compared, and iterated on. Braintrust is a good fit for teams that want rigorous, data-driven prompt optimization rather than just logging.

Proxy-based monitoring

Helicone works as a proxy that sits between your application and the LLM provider. You change your API base URL to point at Helicone, and it logs every request with zero code changes. This approach is appealing for teams that want observability without modifying their application code. Helicone tracks costs, latency, and usage patterns across providers.

Agent-focused observability

As AI agents become more common, tools like AgentOps specialize in agent-specific observability. AgentOps records every LLM call, tool invocation, and decision point in an agent session, letting you replay runs step-by-step. It integrates with frameworks like CrewAI, AutoGen, and OpenAI Agents SDK. If you're building autonomous agents rather than simple LLM chains, agent-specific tooling gives you better visibility into multi-step reasoning and tool use.

ML platforms with LLM support

Weights & Biases started as an experiment tracking platform for ML training and has expanded into LLM observability with its Weave product. If your team already uses W&B for model training, adding LLM tracing keeps everything in one platform. Arize AI takes a similar approach, combining traditional ML monitoring with LLM-specific features like prompt tracing and embedding drift detection. Their open-source Phoenix library provides local tracing and evaluation without sending data to a cloud service.

AI gateways

Portkey combines an AI gateway with observability. It sits between your application and LLM providers, handling request routing, fallbacks, and load balancing while logging every request. This gives you both reliability features (automatic retries, provider failover) and observability (cost tracking, latency monitoring) in one layer. Useful for teams running multiple LLM providers in production.

Evaluation and testing

Several tools in this space go beyond logging to offer evaluation capabilities. Humanloop provides prompt management with built-in A/B testing and evaluation workflows. Giskard focuses on AI testing, scanning models for vulnerabilities and quality issues before deployment. Patronus AI specializes in automated evaluation with LLM-as-judge approaches.

Comparison

How to choose

If you want open-source and self-hosting, start with Langfuse. If you're already using LangChain, LangSmith is the path of least resistance. If you want the simplest possible setup, Helicone's proxy approach gets you logging without touching your code. For teams focused on prompt quality and evaluation, Braintrust or Humanloop offer more structured workflows. If you're building agents, AgentOps gives you the right level of detail for multi-step execution traces. If your team already uses Weights & Biases or Arize for ML monitoring, their LLM features keep everything in one place. And if you need an AI gateway with built-in observability, Portkey covers both routing and logging.

Most of these tools offer free tiers, so the practical approach is to try two or three with your actual workload before committing.