The best local LLMs for private, regulated work (2026)
If your work falls under an NDA, GDPR, professional secrecy rules, or just a client who asked, the cloud AI question answers itself: the data stays home. That decision leads straight to a harder one. Local model quality now varies enormously, the marketing does not match the hardware reality, and a wrong pick means either an assistant too weak to be useful or a model that will not fit on the machine you own. Here is the field as of mid-2026, with every spec checked against the vendor's own documentation.
Quick answer: For private, regulated work in 2026: Qwen 3.6 35B-A3B is the best all-rounder (24GB of VRAM, or 16GB with expert offload), Mistral Small 3.2 24B is the strongest dense pick for 20-24GB cards, Gemma 4 is the writing and multimodal pick, and Phi-4-mini covers 8GB machines. If your assistant calls tools, choose Qwen, Mistral, or Granite.
What makes a local LLM good for regulated work?
Four things, in order:
- It fits your hardware honestly. A model you cannot run is not a model. Vendor size claims usually describe the unquantized weights; what matters is the 4-bit quantized file plus working memory.
- The license permits commercial use without surprises. Apache 2.0 and MIT are unconditional. Meta's community license mostly is not a problem, but it is not "open source, no strings."
- Tool calling, if you need it. An assistant that creates tasks, searches notes, or queries a database does it through function calls, and a model not trained for them will fail in ways that look random.
- It is good at grounded tasks. Local models shine when the answer comes from your data (summarize, extract, draft, answer-from-notes) rather than from open-ended reasoning.
Which models are worth running?
Qwen 3.6 35B-A3B: the all-rounder. A mixture-of-experts model: 35B parameters total, only about 3B active per token, so it delivers big-model quality at small-model speed. Apache 2.0, 262K context window, and vendor-documented function calling that works in agentic flows. The honest hardware note: the default 4-bit build is a ~24GB download, so it lives fully in VRAM only on a 24GB GPU; on a 16GB card it still runs well by offloading experts to system RAM. On Ollama as qwen3.6:35b.
Mistral Small 3.2 24B: the dependable dense pick. Apache 2.0, and a 15GB download at 4-bit on Ollama. The honest hardware note: 15GB of weights on a 16GB card leaves almost no room for the working memory a model needs at real context lengths, so treat it as comfortable on 20-24GB and workable on 16GB with reduced context or a few layers offloaded to CPU. Mistral's own model card highlights stronger function calling and better instruction following over version 3.1. If you own one good 24GB GPU and want one model that just behaves, this is the pick.
Gemma 4: the writer, and the family with range. Google's April 2026 release moved Gemma to Apache 2.0 and spans four sizes: E2B and E4B for edge devices (E4B runs happily in an 8GB machine), a 26B mixture-of-experts, and a 31B dense model that fits 24GB quantized. Natively multimodal, strong at prose. Gemma 4 also added native function calling, which Gemma 3 never had; there is not much independent benchmark data on it yet, so for tool-heavy work the other picks are safer until numbers land.
Phi-4-mini: the small-machine pick. Microsoft's 3.8B model, MIT-licensed, built for memory-constrained environments, roughly 2.5GB at 4-bit. On an 8GB laptop it handles summaries, cleanup, and simple extraction without drama.
Granite 4: the enterprise pick. IBM's models, Apache 2.0, explicitly trained for function calling and retrieval-augmented workloads, with sizes from 3B to 32B. Less charismatic than the others as a chat partner, but predictable in exactly the way regulated environments want.
What about Llama 4?
Skip it for local work, and be suspicious of guides that recommend it. Llama 4's smallest model (Scout) is 109B total parameters and needs a datacenter-class GPU even at aggressive quantization; Meta's own launch materials describe it fitting on a single H100. Nothing with a consumer graphics card runs it. The Llama that runs on ordinary hardware is still the 3.x line, and Llama 3.2 3B remains a fine 8GB pick with tool-calling support.
One more nuance: Llama ships under Meta's community license, not Apache 2.0. It is free for almost everyone, but it carries conditions (attribution, a separate license above 700M monthly users, and EU restrictions on the multimodal variants) that a compliance-minded reader should at least know exist.
How do you match a model to your hardware?
The rule of thumb: usable memory must cover the quantized model file plus 1-2GB of working overhead.
- 8GB: Phi-4-mini, Gemma 4 E4B, Llama 3.2 3B.
- 16GB: Qwen 3.6 35B-A3B with expert offload, or Mistral Small 3.2 24B with reduced context or partial offload (its 15GB of weights leaves little headroom on a 16GB card). An 8B-class dense model fits with room to spare.
- 24GB: Qwen 3.6 35B fully in VRAM, Mistral Small 3.2 comfortably, or Gemma 4 31B (a ~20GB download at 4-bit).
On Apple Silicon, unified memory mostly counts toward these numbers, with one catch: macOS reserves a share of it, and the GPU can typically use about two-thirds to three-quarters of total RAM. A 16GB MacBook is really a ~10-11GB machine for models; a 32GB one comfortably runs the 24B class.
How do you actually use one of these?
The model is half the setup. The other half is serving it where your apps can reach it, which in practice means Ollama's OpenAI-compatible endpoint at http://localhost:11434/v1. The full walkthrough is in how to run an AI assistant with zero cloud egress. In Calmara that endpoint plugs into the Pure Local connection mode, where the browser talks to your model directly and a hard privacy setting blocks any cloud call. That combination is the working answer for anyone whose compliance rules keep client data out of cloud AI.
And if your constraint is softer, you want control over where data goes but a cloud model is acceptable, bring your own key is the intermediate option: your data still travels, but under your account and the provider's API terms rather than an app vendor's.
FAQ
What is the best local LLM overall in 2026?
Qwen 3.6 35B-A3B, if your hardware allows it. Apache 2.0 license, 262K context, reliable tool calling, and mixture-of-experts efficiency that keeps it fast. It needs a 24GB GPU to sit fully in VRAM, or a 16GB GPU with experts offloaded to system RAM.
What is the best local LLM for a 16GB GPU?
It is a trade-off. Qwen 3.6 35B-A3B runs well via expert offload to system RAM. Mistral Small 3.2 24B's 4-bit build is 15GB, which technically loads but leaves almost no working-memory headroom, so expect reduced context or light offloading. For zero compromises on 16GB, an 8B-class dense model is the honest fit.
Can I run Llama 4 locally?
Not on consumer hardware. Llama 4 Scout, the smallest variant, is 109B total parameters and targets datacenter GPUs like the H100. For local Llama, use the 3.x line; Llama 3.2 3B is the practical small option.
Are these models free for commercial use?
Qwen 3.6, Mistral Small 3.2, Gemma 4, and Granite 4 are Apache 2.0 and Phi-4-mini is MIT, all unconditional. Llama uses Meta's community license: free for nearly everyone, but with attribution requirements and conditions for very large companies.
Which local models can call tools and functions?
Qwen 3.6, Mistral Small 3.2, and IBM Granite 4 are the proven, vendor-documented tool callers. Llama 3.x supports tool calling too. Gemma 4 added native function calling in April 2026, but independent benchmarks on it are still thin.
Do I need a GPU to run a local LLM?
Not strictly. Small models like Phi-4-mini run on CPU, just slowly, and Apple Silicon Macs run mid-size models on the integrated GPU through unified memory. For the 24B-and-up class at comfortable speed, a discrete GPU with 16-24GB of VRAM (or a 32GB+ Mac) is the realistic requirement.
Is a local LLM automatically private?
The inference is: prompts and documents never leave your machine, so there is no provider retention, training, or breach exposure. The app around the model still matters, though. Check where it stores data and whether any telemetry leaves your network.
How much disk space do local models need?
Roughly 2 to 24GB per model at 4-bit quantization: 2.5GB for Phi-4-mini, about 5GB for an 8B model, 15GB for Mistral Small 3.2, and 24GB for Qwen 3.6 35B. Keeping two or three models installed is normal; budget 50GB and you will not think about it again.
Written by Dan Hagen