LocalAI

LocalAI is a self-hosted inference runtime that ships as a single binary and exposes an OpenAI-compatible API over 60+ swappable backends, including llama.cpp, vLLM, SGLang, transformers, whisper.cpp, diffusers and MLX. It also emulates the Anthropic, Ollama and ElevenLabs APIs, and covers text, audio, vision and image generation rather than text alone. Install paths include Docker, a CLI, Kubernetes and source.

The distinguishing choice is that no GPU is required. Every feature ships a CPU path first and is tested in CI on ordinary hardware, with optional acceleration through CUDA, ROCm, Intel oneAPI, Apple Metal, Vulkan and NVIDIA Jetson. The practical consequence is that CPU-first means CPU-speed: the documentation recommends small models such as qwen3-4b for CPU use, and larger models will want acceleration to stay responsive. Minimum RAM and storage are not published, only a general claim of consumer-grade hardware.

The licence is MIT. Worth weighing before it goes into production: the project is led by an individual, Ettore Di Giacinto, rather than a company, and there is no commercial support contract or SLA available. Development is active and the issue count is healthy, but 60+ backends across several modalities is a large surface for a very small team, so backend maturity varies.