Local LLM inference in September 2026: what actually works

The pitch for running LLMs locally used to be "free, private, and slow." In September 2026, the "slow" part is increasingly optional. Consumer hardware has caught up to the point where a £500 GPU runs inference at speeds that would have required a data centre two years ago, and the software stack has matured from "impressive demo" to "boring reliable tool."

This is a practical guide. If you are considering running models locally — for development, for privacy, for cost control, or simply because you want to understand what is happening under the hood — here is what the landscape looks like right now.

The hardware reality

The single most important variable is VRAM. Everything else — clock speed, memory bandwidth, PCIe generation — matters, but VRAM determines which models you can run at all. Here is the rough mapping as of this month:

8 GB VRAM (RTX 4060, RTX 3070): runs 7B–8B parameter models at Q4 quantisation comfortably. Think Llama 3.1 8B, Mistral 7B, Qwen 2.5 7B. You get 30–50 tokens per second, which is faster than most people can read. Context windows above 8K start to get tight.

12 GB VRAM (RTX 4070, RTX 3060 12GB): opens up 13B models at Q4 and 7B models at Q8 (full quality). This is the sweet spot for most developers — enough headroom for a decent context window without constant memory pressure.

24 GB VRAM (RTX 4090, RTX 3090, M2/M3/M4 Max with unified memory): runs 70B models at Q4 quantisation. This is where local inference stops being a compromise and starts being genuinely competitive with API endpoints. Llama 3.1 70B at Q4 on an RTX 4090 produces 15–20 tokens per second — fast enough for interactive use.

Apple Silicon deserves its own mention. The M4 Pro and M4 Max share memory between CPU and GPU, which means a 64 GB M4 Max can run models that would require a 48 GB GPU on the Nvidia side. The trade-off is memory bandwidth — Apple's unified memory is fast, but not as fast as GDDR6X. For models that fit in memory, Apple Silicon is price-competitive and remarkably power-efficient.

The software stack

Three tools dominate local inference, and each has a clear niche:

llama.cpp is the foundation. Written in C/C++, it runs GGUF-quantised models on everything from a Raspberry Pi to a multi-GPU server. If you want a single tool that works everywhere, this is it. The project has matured to the point where building from source is a five-minute process on any platform, and pre-built binaries are available for every major OS. The llama-server binary gives you an OpenAI-compatible API endpoint, which means any tool that speaks the OpenAI API can talk to your local model without modification.

Ollama wraps llama.cpp in a user-friendly layer. ollama run llama3.1 downloads the model, quantises it if needed, and starts an interactive session. It is the fastest path from "I want to try a local model" to having one running. The downside is that it abstracts away enough details that debugging performance issues becomes harder. For production use, most people eventually graduate to llama.cpp directly.

vLLM is the throughput engine. If you are serving multiple concurrent users — an internal tool for a team, a research pipeline, or a local API for an agent framework — vLLM's PagedAttention and continuous batching give you 2–5x the throughput of naive inference. It requires a CUDA GPU and Linux, but for server-side local inference, nothing else comes close.

What quantisation actually means

Quantisation reduces the precision of model weights to save memory and speed up computation. A 70B parameter model at full precision (FP16) requires about 140 GB of VRAM. At Q4_K_M — the most common quantisation for local use — it requires about 40 GB. The quality loss is measurable on benchmarks but rarely noticeable in practice for most tasks.

The naming convention matters: Q4_K_M is a good default. Q4_K_S is slightly smaller and slightly worse. Q5_K_M is better quality but larger. Q8_0 is essentially lossless but uses twice the memory of Q4. For most use cases, Q4_K_M is the right trade-off.

Newer quantisation methods like GGUF's imatrix and EXL2's variable bit rates are improving quality at lower sizes, but Q4_K_M remains the safe, well-tested default.

When local beats the API

The economics are straightforward. An RTX 4090 costs about £1,600. Running Llama 3.1 70B on it costs nothing per token after that. If you are processing more than about 2 million tokens per month, the GPU pays for itself in three months compared to mid-tier API pricing. If you are processing 10 million tokens per month, it pays for itself in three weeks.

Privacy is the other obvious advantage. If you are working with sensitive data — medical records, legal documents, financial information — keeping everything on-premise eliminates an entire category of compliance risk. No data leaves your network, no terms of service to negotiate, no vendor to audit.

Latency can also be better locally. A well-tuned local setup produces first-token latency under 100 milliseconds. API endpoints, even fast ones, add network round-trip time. For interactive use, local feels noticeably snappier.

When the API is still the right call

Frontier models — GPT-5, Claude Opus 4, Gemini 2.5 Pro — are not available locally. If you need the absolute best reasoning, longest context windows, or multimodal capabilities, the API is still the only option. The gap between the best local model (Llama 3.1 70B or Qwen 2.5 72B) and the best API model is still significant on complex reasoning tasks.

Scaling is the other consideration. If your usage is bursty — quiet most of the time, then heavy for a few hours — the API's pay-per-token model is more efficient than maintaining hardware that sits idle 90% of the time.

The practical starting point

If you want to try local inference today, here is the fastest path:

Install Ollama (ollama.com). Run ollama run llama3.1. If it is usable on your hardware — and it will be on anything with 8 GB of VRAM or 16 GB of unified memory — you have your answer. If you want an API endpoint, run ollama serve and point your tools at http://localhost:11434.

From there, graduate to llama.cpp when you want more control over quantisation, context size, and performance tuning. Move to vLLM when you need to serve multiple users or run batch inference at scale.

Local inference is no longer the enthusiast's hobby. It is a production-ready option for anyone who wants control over their AI stack. The tools are stable, the hardware is affordable, and the quality is good enough for the vast majority of real-world tasks.