The best local LLMs for private, regulated work (2026)
If your work falls under an NDA, GDPR, professional secrecy rules, or just a client who asked, the cloud AI question answers itself: the data stays home. That decision leads straight to a harder one. Local model quality now varies enormously, the marketing does not match the hardware reality, and a wrong pick means either an assistant too weak to be useful or a model that will not fit on the machine you own. Here is the field as of October 2026, with every spec checked against the vendor's own documentation.
Quick answer: For private, regulated work in 2026: Qwen 3.6 35B-A3B is the best all-rounder (24GB of VRAM, or 16GB with expert offload), Mistral Small 3.2 24B is the strongest dense pick for 20-24GB cards, Gemma 4 is the writing and multimodal pick, and Phi-4-mini covers 8GB machines. If your assistant calls tools, choose Qwen, Mistral, or Granite.
What makes a local LLM good for regulated work?
Four things, in order:
- It fits your hardware honestly. A model you cannot run is not a model. Vendor size claims usually describe the unquantized weights; what matters is the 4-bit quantized file plus working memory.
- The license permits commercial use without surprises. Apache 2.0 and MIT are unconditional. Meta's community license mostly is not a problem, but it is not "open source, no strings."
- Tool calling, if you need it. An assistant that creates tasks, searches notes, or queries a database does it through function calls, and a model not trained for them will fail in ways that look random.
- It is good at grounded tasks. Local models shine when the answer comes from your data (summarize, extract, draft, answer-from-notes) rather than from open-ended reasoning.
Which models are worth running?
Qwen 3.6 35B-A3B: the all-rounder. A mixture-of-experts model: 35B parameters total, only about 3B active per token, so it delivers big-model quality at small-model speed. Apache 2.0, 262K context window, and vendor-documented function calling that works in agentic flows. The honest hardware note: the default 4-bit build is a ~23GB download, so it lives fully in VRAM only on a 24GB GPU; on a 16GB card it still runs well by offloading experts to system RAM. On Ollama as qwen3.6:35b. Its newer sibling, Qwen 3.8 27B (dense, Apache 2.0, August 2026, 18GB at 4-bit as qwen3.8:27b, tools and vision), is the fresher Qwen for a 24GB card; the 35B-A3B mixture-of-experts has no 3.8 successor yet and remains the pick when you need to offload to 16GB.
Mistral Small 3.2 24B: the dependable dense pick. Apache 2.0, and a 15GB download at 4-bit on Ollama. The honest hardware note: 15GB of weights on a 16GB card leaves almost no room for the working memory a model needs at real context lengths, so treat it as comfortable on 20-24GB and workable on 16GB with reduced context or a few layers offloaded to CPU. Mistral's own model card highlights stronger function calling and better instruction following over version 3.1. If you own one good 24GB GPU and want one model that just behaves, this is the pick.
One naming trap to avoid: Mistral Small 4, released March 2026, is not this model's successor for local work. It is a 119B-parameter mixture-of-experts release whose minimum hardware, by Mistral's own spec, is four H100s (or two H200s, or one DGX B200); "Small" now means small relative to the 675B Mistral Large 3. For a single GPU, 3.2 remains the Mistral to run.
Gemma 4: the writer, and the family with range. Google's spring 2026 release (the first four sizes on March 31, the 12B on June 3) moved Gemma to Apache 2.0 and spans five sizes: E2B and E4B for edge devices (on Ollama the default E4B tag is now about 6.6GB and the QAT build about 6GB, so either runs happily in an 8GB machine), a 12B whose 7.2GB QAT build squeezes onto an 8GB card with the context kept short, a 26B mixture-of-experts with about 4B active, and a 31B dense model that fits 24GB quantized. Natively multimodal, strong at prose. Gemma 4 also added native function calling, which Gemma 3 never had; there is not much independent benchmark data on it yet, so for tool-heavy work the other picks are safer until numbers land.
Phi-4-mini: the small-machine pick. Microsoft's 3.8B model, MIT-licensed, built for memory-constrained environments, roughly 2.5GB at 4-bit. On an 8GB laptop it handles summaries, cleanup, and simple extraction without drama.
Granite 4.2: the enterprise pick. IBM refreshed the line again on August 25, 2026 with 3B, 8B, and 30B models, Apache 2.0, native reasoning, and a reasoning-before-tool-call mode IBM's model documentation calls reasoning-augmented tool calling. Two credentials none of the others carry, and exactly the kind of thing a compliance file likes: IBM ships the models "with cryptographic signatures, ISO certification", so you can verify what you downloaded and point an auditor at a certified process. Less charismatic than the others as a chat partner, but predictable in exactly the way regulated environments want.
What about Llama 4?
Skip it for local work, and be suspicious of guides that recommend it. Llama 4's smallest model (Scout) is 109B total parameters and needs a datacenter-class GPU even at aggressive quantization; Meta's own launch materials describe it fitting on a single H100. Nothing with a consumer graphics card runs it. The Llama that runs on ordinary hardware is still the 3.x line, and Llama 3.2 3B remains a fine 8GB pick with tool-calling support.
One more nuance: Llama ships under Meta's community license, not Apache 2.0. It is free for almost everyone, but it carries conditions (attribution, a separate license above 700M monthly users, and EU restrictions on the multimodal variants) that a compliance-minded reader should at least know exist.
Meta's post-Llama story split in 2026. Its flagship Muse Spark (April) is fully proprietary, and Llama 4 remains the last open-weight Llama. But on August 10, 2026 Meta opened the weights of Muse Glimmer, a 30-billion-parameter dense model under Apache 2.0 built for on-device agents, with function calling and a 4-bit build "under 20 GB" (18GB on Ollama as muse-glimmer:30b). It is the Meta model to evaluate for a 24GB card; it does not change the 8GB or 16GB picks.
Are the big new open-weight models an option?
A wave of headline open-weight releases makes "best local LLM" searches confusing in 2026, because open weights and local are quietly different claims. GLM-5.2 (MIT, about 750B parameters total; its August successor GLM-5.3 is the same size under a custom license), DeepSeek V4 (MIT, up to 1.6T in its Pro variant), and Moonshot's Kimi K3 (a 2.8T mixture-of-experts, announced in July as the largest open-weight model yet) are all real and all licensed generously. None of them runs on anything resembling business hardware. A license you can read does not shrink a model, and a mixture-of-experts design lowers compute per token, not the memory that must hold the weights. For work that has to stay on machines you own, the realistic field is still the 2B-35B class above.
How do you match a model to your hardware?
The rule of thumb: usable memory must cover the quantized model file plus 1-2GB of working overhead.
- 8GB: Phi-4-mini, Gemma 4 E4B, or the 12B's QAT build, Llama 3.2 3B.
- 16GB: Qwen 3.6 35B-A3B with expert offload, or Mistral Small 3.2 24B with reduced context or partial offload (its 15GB of weights leaves little headroom on a 16GB card). An 8B-class dense model fits with room to spare.
- 24GB: Qwen 3.6 35B fully in VRAM, Qwen 3.8 27B or Muse Glimmer 30B (both ~18GB at 4-bit), Mistral Small 3.2 comfortably, or Gemma 4 31B (a ~20GB download at 4-bit).
On Apple Silicon, unified memory mostly counts toward these numbers, with one catch: macOS reserves a share of it, and the GPU can typically use about two-thirds to three-quarters of total RAM. A 16GB MacBook is really a ~10-11GB machine for models; a 32GB one comfortably runs the 24B class.
How do you actually use one of these?
The model is half the setup. The other half is serving it where your apps can reach it, which in practice means Ollama's OpenAI-compatible endpoint at http://localhost:11434/v1. The full walkthrough is in how to run an AI assistant with zero cloud egress. In Calmara that endpoint plugs into the Pure Local connection mode, where the browser talks to your model directly and a hard privacy setting blocks any cloud call. That combination is the working answer for anyone whose compliance rules keep client data out of cloud AI.
And if your constraint is softer, you want control over where data goes but a cloud model is acceptable, bring your own key is the intermediate option: your data still travels, but under your account and the provider's API terms rather than an app vendor's.
FAQ
What is the best local LLM overall in 2026?
Qwen 3.6 35B-A3B, if your hardware allows it. Apache 2.0 license, 262K context, reliable tool calling, and mixture-of-experts efficiency that keeps it fast. It needs a 24GB GPU to sit fully in VRAM, or a 16GB GPU with experts offloaded to system RAM.
What is the best local LLM for a 16GB GPU?
It is a trade-off. Qwen 3.6 35B-A3B runs well via expert offload to system RAM. Mistral Small 3.2 24B's 4-bit build is 15GB, which technically loads but leaves almost no working-memory headroom, so expect reduced context or light offloading. For zero compromises on 16GB, an 8B-class dense model is the honest fit.
Can I run Llama 4 locally?
Not on consumer hardware. Llama 4 Scout, the smallest variant, is 109B total parameters and targets datacenter GPUs like the H100. For local Llama, use the 3.x line; Llama 3.2 3B is the practical small option.
Are these models free for commercial use?
Qwen 3.6 and 3.8, Mistral Small 3.2, Gemma 4, Granite 4.2, and Muse Glimmer are Apache 2.0 and Phi-4-mini is MIT, all unconditional. Llama uses Meta's community license: free for nearly everyone, but with attribution requirements and conditions for very large companies.
Which local models can call tools and functions?
Qwen 3.6, Mistral Small 3.2, and IBM Granite 4.2 are the proven, vendor-documented tool callers, and Granite 4.2's August 2026 release adds a reasoning-before-tool-call mode on top of 4.1's tool-calling post-training. Llama 3.x supports tool calling too. Gemma 4 shipped with native function calling, but independent benchmarks on it are still thin.
What is the best local LLM for business or enterprise use?
For business use the license and the paper trail matter as much as the benchmark. Qwen 3.6, Mistral Small 3.2, Gemma 4, and Granite 4.2 are all Apache 2.0, so commercial use is unconditional. If you need something to show an auditor, Granite 4.2 is the standout: ISO certified and cryptographically signed releases. Hardware-wise, a single workstation with a 24GB GPU runs any of them, which keeps the deployment simple enough for a team without an ML engineer.
Do I need a GPU to run a local LLM?
Not strictly. Small models like Phi-4-mini run on CPU, just slowly, and Apple Silicon Macs run mid-size models on the integrated GPU through unified memory. For the 24B-and-up class at comfortable speed, a discrete GPU with 16-24GB of VRAM (or a 32GB+ Mac) is the realistic requirement.
Is a local LLM automatically private?
The inference is: prompts and documents never leave your machine, so there is no provider retention, training, or breach exposure. The app around the model still matters, though. Check where it stores data and whether any telemetry leaves your network.
How much disk space do local models need?
Roughly 2 to 24GB per model at 4-bit quantization: 2.5GB for Phi-4-mini, about 5GB for an 8B model, 15GB for Mistral Small 3.2, and 23GB for Qwen 3.6 35B. Keeping two or three models installed is normal; budget 50GB and you will not think about it again.
Written by Dan Hagen