LLM Observability Tools Compared (2026)

March 28, 2026 / Arvid Andersson

Once you ship LLM features to production, you need visibility into what's happening: what prompts are being sent, how much each call costs, whether quality is holding up, and where failures occur. LLM observability tools fill this gap. This post compares the leading options and what each does well.

Why LLM observability matters

Traditional application monitoring tracks request times and error rates. LLM applications add new dimensions: token costs that can spike unexpectedly, prompt quality that degrades over time, hallucinations that are hard to catch without systematic evaluation, and complex chains where a failure in one step cascades. LLM observability tools capture the full trace of each request, from prompt to completion, and give you the data to debug, optimize, and evaluate.

Open-source options

Langfuse is the most established open-source option. It provides tracing, prompt management, evaluation, and cost tracking, with integrations across most LLM frameworks (LangChain, LlamaIndex, OpenAI SDK). ClickHouse acquired Langfuse in January 2026 alongside a $400M Series D. The MIT license and self-hosting remain unchanged. Self-hosting is attractive for teams with strict data residency requirements.

Opik (by Comet) is an Apache-2.0 option with built-in evaluation metrics for hallucination, moderation, and relevance. It includes an agent optimizer SDK for automated prompt tuning. Backed by Comet, an established MLOps company, so the project has staying power.

Lunary is also open-source (Apache-2.0), focused on chatbot and RAG evaluation with the cheapest paid tier in the category at $20/user/month. Good for smaller teams that want observability without a large budget.

Platform-integrated tools

LangSmith is built by the LangChain team and integrates tightly with the LangChain ecosystem. If your application already uses LangChain, LangSmith gives you tracing, evaluation datasets, and prompt playgrounds with minimal setup. The trade-off is vendor lock-in to the LangChain stack.

Braintrust takes a different angle, focusing on evaluation and experimentation. It treats LLM outputs as data that should be scored, compared, and iterated on. After raising an $80M Series B in February 2026 (customers include Notion, Replit, Cloudflare, and Ramp), the team built Brainstore, a custom database designed for querying complex AI traces from multi-step agents. Braintrust is a good fit for teams that want rigorous, data-driven prompt optimization rather than just logging.

Proxy-based monitoring

Helicone works as a proxy that sits between your application and the LLM provider. You change your API base URL to point at Helicone, and it logs every request with zero code changes. This approach is appealing for teams that want observability without modifying their application code. Helicone tracks costs, latency, and usage patterns across providers.

Agent-focused observability

As AI agents become more common, tools like AgentOps specialize in agent-specific observability. AgentOps records every LLM call, tool invocation, and decision point in an agent session, letting you replay runs step-by-step. It integrates with frameworks like CrewAI, AutoGen, and OpenAI Agents SDK. If you're building autonomous agents rather than simple LLM chains, agent-specific tooling gives you better visibility into multi-step reasoning and tool use.

HoneyHive covers similar ground with distributed tracing and drift detection for production agents, plus flexible deployment options (SaaS, hybrid, or self-hosted). Useful for teams that need on-premises deployment alongside agent observability.

ML platforms with LLM support

Weights & Biases started as an experiment tracking platform for ML training and has expanded into LLM observability with its Weave product. If your team already uses W&B for model training, adding LLM tracing keeps everything in one platform. Arize AI takes a similar approach, combining traditional ML monitoring with LLM-specific features like prompt tracing and embedding drift detection. Their open-source Phoenix library provides local tracing and evaluation without sending data to a cloud service.

AI gateways

Portkey combines an AI gateway with observability. It sits between your application and LLM providers, handling request routing, fallbacks, and load balancing while logging every request. In March 2026, Portkey fully open-sourced its production gateway, including features that previously required a SaaS subscription: circuit breakers, semantic caching, budget limits, and real-time metrics. The gateway processes over 1 trillion tokens daily across 24,000+ organizations. Portkey also added an MCP Gateway for governing AI agents across enterprise tools. Useful for teams running multiple LLM providers in production who want both reliability and observability in one open-source layer.

Evaluation and testing

Several tools in this space go beyond logging to offer evaluation capabilities. Giskard focuses on AI testing, scanning models for vulnerabilities and quality issues before deployment. Patronus AI specializes in automated evaluation with LLM-as-judge approaches. (Note: Humanloop, previously listed here for prompt management and A/B testing, was acqui-hired by Anthropic in August 2025 and shut down in September 2025. Separately, Traceloop, whose open-source OpenLLMetry library provides OpenTelemetry instrumentation used by several tools on this page, was acquired by ServiceNow in March 2026.)

LangWatch bridges evaluation and observability with agent simulation. It runs thousands of synthetic conversations across scenarios, languages, and edge cases, then feeds results back into prompt optimization via DSPy integration. Netherlands-based and open-source. Useful for teams that want to test agent behavior systematically before shipping, not just monitor after the fact.

Comparison

How to choose

If you want open-source and self-hosting, start with Langfuse. If you're already using LangChain, LangSmith is the path of least resistance. If you want the simplest possible setup, Helicone's proxy approach gets you logging without touching your code. For teams focused on prompt quality and evaluation, Braintrust offers structured scoring and comparison workflows. If you're building agents, AgentOps or HoneyHive give you the right level of detail for multi-step execution traces. If your team already uses Weights & Biases or Arize for ML monitoring, their LLM features keep everything in one place. If you need an AI gateway with built-in observability, Portkey's fully open-source gateway covers both routing and logging. For budget-conscious teams building chatbots or RAG pipelines, Lunary's $20/user/month tier is the cheapest entry point. Opik is worth evaluating if you want open-source with built-in evaluation metrics and prompt optimization. And LangWatch is a good fit for teams that want to test agent behavior with synthetic conversations before deploying.

Most of these tools offer free tiers, so the practical approach is to try two or three with your actual workload before committing.

Frequently asked questions