AI Inference API Providers Compared (2026)

March 4, 2026 / Updated November 3, 2026 / Arvid Andersson

An inference API provider runs AI models on its own hardware and sells access to them per token, through an API. They come in three kinds: model labs selling their own models (OpenAI, Anthropic, Google, Mistral), hosts running open models on GPUs or custom chips (DeepInfra, Together AI, Fireworks, Groq, Cerebras), and routers that resell many providers through one key (OpenRouter). This post compares the hosts and routers on price, speed, API compatibility and what each does best.

On gpt-oss-120B, the open model the most providers publish a price for, CoreWeave, DeepInfra, and Melious AI list the lowest blended rates of the 17 providers in the price table below, checked against each provider's pricing page between 25 September and 15 October 2026.

Looking for a side-by-side table with filters? See the Inference APIs pricing comparison. For per-token prices on one model across providers, see LLM API pricing by model.

Three kinds of inference provider

• Model lab APIs (11 listed): the company that trains the model sells it. OpenAI and Anthropic (Claude) are where most teams start.
• Hosted inference APIs (48 listed): open models such as gpt-oss, DeepSeek, Qwen and Llama, run on the host's own GPUs or custom chips. This is where prices and speed differ most, and what most of this post compares.
• LLM routers (20 listed): one API key in front of many providers, reselling their tokens. Convenient for switching models; you pay their margin and add a hop.

At one end, providers like Groq compete primarily on speed, using custom hardware (LPUs) to deliver the lowest latency. At the other end, platforms like RunPod and Modal give you raw GPU access to run any model with full control over the stack.

In between sit the serverless inference providers: DeepInfra, Together.ai, Fireworks, and Replicate. These handle the infrastructure and expose models through APIs, typically with OpenAI-compatible endpoints.

Speed vs. cost

The speed picture changed in 2026 with the rise of specialized inference hardware. Cerebras reports roughly 3000 tokens/sec on gpt-oss-120B using its WSE chips; Artificial Analysis measures ~1,700 tokens/sec sustained (July 2026), still far ahead of the field. Groq runs the same model at around 478 tokens/sec on custom LPUs with a consistently low time-to-first-token (~0.7s). The trade-off on both is a narrower model catalog than GPU-based providers, and picking between them depends on whether throughput or first-token latency matters more for the workload.

Both hardware players were pulled into bigger orbits recently: Nvidia licensed Groq's LPU technology in a deal reported at around $20B (December 2025), with Groq raising fresh capital in June 2026 to keep building its inference cloud, and OpenAI committed a reported $20B+ to Cerebras compute (April 2026). Neither changes the public APIs today, but both are worth knowing when betting on either provider long-term.

On price, CoreWeave, DeepInfra, and Melious AI list the lowest rates on gpt-oss-120B in the price table above. The gap between the cheapest provider and the fastest on the same model can be several times, so the right pick depends on whether a workload is batch (price first) or interactive (speed first).

Together.ai and Fireworks sit in a similar space: broad model support and solid performance, with the added option of fine-tuning on the same platform. Together.ai's fine-tuning workflow is convenient when iterating on a custom fine-tune without moving weights to a separate training service.

API compatibility and routing

Most providers now support the OpenAI API format, meaning you can switch between them by changing a base URL and API key. DeepInfra, Together.ai, Fireworks, and novita.ai all work with the standard OpenAI Python and Node.js SDKs. This makes it practical to route different models through different providers without rewriting integration code.

OpenRouter takes this a step further by acting as a unified API gateway across multiple providers. You send requests to OpenRouter, and it routes them to the cheapest or fastest provider for a given model. This is useful for teams that want to avoid vendor lock-in or automatically fail over between providers. LowRouter, a French router, passes provider prices through with no per-token markup and charges a fee when credits are bought; its auto mode prefers EU-sovereign routes, then the lowest grid carbon intensity, then price.

Beyond text: image, audio, video

Some providers go beyond LLM inference. Replicate hosts a wide range of open-source models across modalities, from Stable Diffusion to Whisper. fal focuses on fast image and video generation. novita.ai covers LLMs, image generation, video, and audio APIs in one platform. inference shell sells one API key for image, video, audio, text, search and 3D apps, billed per call, with a skill registry and a CLI around it. If your application needs multiple model types, consolidating on a multi-modal provider can simplify your infrastructure.

GPU cloud for custom setups

When you need full control, whether for custom models, specific framework versions, or multi-GPU training, GPU cloud providers are the way to go. RunPod offers on-demand and spot GPU instances with per-second billing and no egress fees. Runcrate rents GPUs by the second on most models and sells 125 models behind one OpenAI-compatible API, paid from one prepaid balance (its pricing page, November 2026). Modal takes a code-first approach where you define your environment in Python and Modal handles the provisioning. Baseten and Cerebrium focus on model deployment, letting you package models in containers and serve them as auto-scaling endpoints. For the inference engine itself, vLLM has become the standard open-source serving framework that many of these providers use under the hood.

Provider comparison

The price column repeats each provider's verified gpt-oss-120B rate from the table above; a dash means it publishes no price for that model or runs your own models instead.

How to choose

Start with your constraints. If sustained throughput is the primary concern, test Cerebras and Groq against each other on the specific model you plan to run. If first-token latency drives the experience (chat, interactive agents), Groq's LPU stack has a consistent edge. If cost dominates, benchmark DeepInfra and novita.ai against your expected volume. If you need fine-tuning alongside inference, Together.ai and Fireworks both offer that. If you want a single API across multiple providers, OpenRouter handles the routing. For deploying custom models, Baseten, Cerebrium, RunPod, and Modal each take a different approach to packaging and serving.

The OpenAI-compatible API standard makes it easy to test multiple providers without significant code changes. Many teams run different providers for different use cases: a fast provider for user-facing chat, a cheaper one for batch processing, and a GPU cloud for custom models.

Frequently asked questions