LLM Observability Tools Compared (2026)

March 28, 2026 / Updated October 25, 2026 / Arvid Andersson

An LLM observability tool records every model call and the steps around it (prompt, response, tokens, cost, latency, tool calls) so you can debug failures and watch quality in production. The tools differ on three things: whether you can self-host the data, whether they also run evaluations or a gateway, and who owns them. Five of the tools below changed hands in 2026; ownership is part of the comparison.

Looking for a side-by-side table with filters? See the Observability & Analytics pricing comparison.

Why LLM observability matters

Traditional application monitoring tracks request times and error rates. LLM applications add new dimensions: token costs that can spike unexpectedly, prompt quality that degrades over time, hallucinations that are hard to catch without systematic evaluation, and complex chains where a failure in one step cascades. LLM observability tools capture the full trace of each request, from prompt to completion, and give you the data to debug, optimize, and evaluate.

Open-source options

Langfuse is one of the more widely adopted open-source options, with about 36k GitHub stars as of October 2026. It provides tracing, prompt management, evaluation, and cost tracking, with integrations across most LLM frameworks (LangChain, LlamaIndex, OpenAI SDK). Langfuse joined ClickHouse in January 2026 (announced as an acquisition, no funding figures disclosed). The core is MIT-licensed, with enterprise features in separate directories under their own licence. The team has said it plans no changes to the MIT license or self-hosting, which matters for teams with strict data residency requirements. Hosted pricing (October 2026): free Hobby plan with 50k units a month, Core at $29/month, Pro at $199/month.

Opik (by Comet) is an Apache-2.0 option (about 23k GitHub stars in October 2026) with built-in evaluation metrics for hallucination, moderation, and relevance. It includes an agent optimizer SDK for automated prompt tuning. Backed by Comet, an established MLOps company.

Laminar is an Apache-2.0, OpenTelemetry-native platform for agent traces, with about 3.4k GitHub stars. Each run renders as a readable transcript, and its Signals feature reads every trace for a behavior you describe in plain English, or surfaces failure modes you never defined. It can be self-hosted or used as a hosted service: free with 1 GB of data a month and 7 days of access, Starter at $30/month for 3 GB (October 2026).

Future AGI is Apache-2.0 and self-hostable, covering tracing, evaluation, simulation, guardrails, and an OpenAI-compatible gateway in one platform. Broader surface area than single-purpose tools, which can mean more to learn at setup, but useful for teams that prefer a single feedback loop over stitching together separate observability, evaluation, and guardrail tools.

Platform-integrated tools

LangSmith is built by the LangChain team and integrates tightly with the LangChain ecosystem. If your application already uses LangChain, LangSmith gives you tracing, evaluation datasets, and prompt playgrounds with minimal setup. The trade-off is vendor lock-in to the LangChain stack.

Braintrust takes a different angle, focusing on evaluation and experimentation. It treats LLM outputs as data that should be scored, compared, and iterated on. After raising an $80M Series B in February 2026 (customers include Vercel, Notion, Coursera, Dropbox, Replit, Graphite, and Navan), the team built Brainstore, a custom database designed for querying complex AI traces from multi-step agents. The Loop agent generates better prompts, scorers, and datasets automatically. A good fit for teams that want rigorous, data-driven prompt optimization rather than just logging.

Lunary offers prompt management, human reviews, and an AI playground. Its Team plan is $20/user/month with 50,000 events included, then $10 per additional 50,000 (October 2026). Two things changed since this post first covered it: its pricing page now lists hosting on your own servers under the Enterprise plan only, and new workspaces require payment before access. There is no free tier for new accounts. Its GitHub repository (lunary-ai/lunary) is no longer available.

Datadog LLM Observability is the option for teams that already run Datadog. Datadog now presents it as Agent Observability: tracing, offline experimentation with golden datasets built from annotated traces, and production monitoring, correlated with the rest of the APM data. Access controls and HIPAA compliance are listed on its product page. It is proprietary and part of the Datadog platform; it fits teams that already pay for Datadog more than teams starting from nothing.

Proxy-based monitoring

Helicone works as a proxy that sits between your application and the LLM provider. You change your API base URL to point at Helicone, and it logs every request with zero code changes. This approach is appealing for teams that want observability without modifying their application code. Helicone tracks costs, latency, and usage patterns across providers. It is Apache-2.0 (about 6.2k GitHub stars) with a free plan of 10,000 requests a month; Pro is $79/month and Team $799/month (October 2026). One thing to weigh before adopting it for a new project: Mintlify acquired Helicone in March 2026 and moved it to maintenance mode, with the founding team joining Mintlify. As of October 25, 2026, Helicone's pricing page says it is no longer part of Mintlify and that details are to come. The repository is still receiving commits, but we found no announcement explaining what changed; check its status before committing to it.

Agent-focused observability

As AI agents become more common, tools like AgentOps specialize in agent-specific observability. AgentOps records every LLM call, tool invocation, and decision point in an agent session, letting you replay runs step-by-step. It integrates with frameworks like CrewAI, AutoGen, and OpenAI Agents SDK. If you're building autonomous agents rather than simple LLM chains, agent-specific tooling gives you better visibility into multi-step reasoning and tool use.

HoneyHive covers similar ground with distributed tracing and drift detection for production agents, plus flexible deployment options (SaaS, hybrid, or self-hosted). Useful for teams that need on-premises deployment alongside agent observability.

The Context Company (YC Fall 2025) approaches agent observability from the conversation side. Alongside the usual traces (tool calls, latency, token cost), it analyzes the conversations themselves: topic clustering, custom pattern tracking, and detection of silent failures like wrong tool calls, loops, and user frustration. Instrumentation is an open-source MIT monorepo, with TypeScript support for the Vercel AI SDK, Claude Agent SDK, and Mastra, and Python support for LangChain, LangGraph, CrewAI, and Agno. A free Developer tier keeps the latest 1,000 runs with 7 days of history; the Pro plan is $400/month with 25,000 runs included, then $0.001 per additional run (October 2026). Early-stage, but the angle is useful if your main question is what users experience, not just what the stack did.

ML platforms with LLM support

Weights & Biases (part of CoreWeave since May 2025) started as an experiment tracking platform for ML training and has expanded into LLM observability with its Weave product. If your team already uses W&B for model training, adding LLM tracing keeps everything in one platform. Arize AI takes a similar approach, combining traditional ML monitoring with LLM-specific features like prompt tracing and embedding drift detection. Their Phoenix library provides local tracing and evaluation without sending data to a cloud service; it is source-available under the Elastic License 2.0, not an OSI-approved open-source licence. Dynatrace completed its acquisition of Arize in October 2026 and says it will keep supporting Phoenix and the Arize AX platform. AX has a free plan with 25k spans a month and a Pro plan at $50/month (October 2026).

AI gateways

Portkey combines an AI gateway with observability. It sits between your application and LLM providers, handling request routing, fallbacks, and load balancing while logging every request. Gateway 2.0 shipped in March 2026 and moved previously enterprise-only features into the MIT-licensed open-source core, including circuit breakers, semantic caching, budget limits, real-time metrics, and an MCP Gateway for governing AI agents across enterprise tools. Portkey's release announcement (March 2026) reports over 1 trillion tokens processed daily, with 50+ guardrails built in. Palo Alto Networks acquired Portkey in May 2026 to serve as the AI gateway in its Prisma AIRS platform. Useful for teams running multiple LLM providers in production who want both reliability and observability in one layer; where the standalone product goes under its new owner is worth watching.

Evaluation and testing

LangWatch bridges evaluation and observability with agent simulation. It runs thousands of synthetic conversations across scenarios, languages, and edge cases, then feeds results back into prompt optimization via DSPy integration. Netherlands-based and open-source. Useful for teams that want to test agent behavior systematically before shipping, not just monitor after the fact.

Two notes on the wider ecosystem: Humanloop, previously listed here for prompt management and A/B testing, was acquired by Anthropic in 2025 and the Humanloop platform was shut down in September 2025. Separately, Traceloop, whose open-source OpenLLMetry library provides OpenTelemetry instrumentation used by several tools on this page, was acquired by ServiceNow in March 2026.

Guardrails and safety

Catching bad outputs before they reach users requires more than logging. Guardrails sit on top of LLM calls to block hallucinations, PII leaks, prompt injection attempts, or policy violations at request time.

Patronus AI focuses on automated evaluation with LLM-as-judge approaches, useful when you want to score outputs at scale without writing custom evaluators. Giskard takes a pre-deployment angle, scanning models for vulnerabilities and quality issues before they reach production. Portkey's gateway includes 50+ built-in guardrails, which fits teams that already want a gateway layer. Future AGI bundles guardrails with simulation and evaluation in one platform, with 18 built-in scanners (PII, jailbreak, prompt injection, and others) and adapters for Lakera, Presidio, and Llama Guard.

The trade-off across this category is breadth versus depth: dedicated safety tools (Patronus, Giskard) tend to go deeper on a narrower problem, while platforms that combine guardrails with tracing or gateway features reduce integration cost but cover less ground per area.

Comparison

How to choose

The category is broad enough that the right pick depends on what you optimize for. A few starting points:

• Open-source and self-hosted: Langfuse for general-purpose tracing, Arize Phoenix for OpenTelemetry-native (Elastic License 2.0), Opik for built-in evaluation metrics, Laminar for agent traces, Future AGI for an all-in-one stack covering evals, simulation, guardrails, and gateway.
• Already on LangChain: LangSmith is the path of least resistance.
• Simplest possible setup: Helicone's proxy gets you logging with a one-line base URL change (it entered maintenance mode in March 2026 and its ownership is unclear, so weigh that for new projects).
• Prompt quality and structured evals: Braintrust for scoring and comparison workflows, Patronus for LLM-as-judge.
• Building agents: AgentOps for session replay and time-travel debugging, HoneyHive for production agent monitoring with flexible deployment.
• Already using W&B or Arize for ML: Their LLM features keep everything in one platform.
• AI gateway plus observability: Portkey for the open-source gateway with routing and logging in one layer.
• Already on Datadog: Datadog LLM Observability keeps AI traces next to your APM data.
• Prompt management with human review: Lunary's Team plan is $20/user/month, with no free tier for new accounts.
• Agent simulation before shipping: LangWatch for synthetic conversation testing with DSPy integration.
• Guardrails, evals, and simulation under one roof: Future AGI bundles all of these with a free tier that covers prototype-scale usage.

Most of these tools offer free tiers, so the practical approach is to try two or three with your actual workload before committing.

Frequently asked questions