How to run an AI assistant with zero cloud egress (Ollama + Calmara)

Published 2026-07-08

Somewhere in your compliance rules, or your client contracts, or just your own judgment, there is a line that says client data does not go into a cloud AI. That line kills most AI assistants outright, because most of them are a thin interface over an API call to OpenAI, Anthropic, or Google. The workaround is not to give up on AI features. It is to move the model onto hardware you control, so the question "where does my data go" has a one-word answer: nowhere.

Quick answer: Zero cloud egress means the AI model runs on your own hardware, so prompts and data never cross the internet. The practical setup: install Ollama, pull a local model, and point your app at Ollama's OpenAI-compatible API at http://localhost:11434/v1. Any app that speaks the OpenAI API format can use it.

What does "zero cloud egress" actually mean?

It means no part of the AI request leaves your network. Not to the model provider, not to the app vendor, not to anyone. The model weights sit on your disk, inference runs on your CPU or GPU, and the "API call" is a request to your own machine.

This is a stricter guarantee than the other privacy options. Bring your own key changes whose account and contract govern your cloud traffic, but the data still travels to the provider. Provider zero-data-retention agreements limit storage, but the data still gets processed on their servers. A local model is the only setup where the data never makes the trip at all. For a consultant under NDA, a clinician, a lawyer, or anyone whose compliance rules exclude cloud AI, that difference is the whole decision.

How does Ollama make this work?

Ollama is the piece that turns "run a model locally" from a research project into a service your apps can use. It is MIT-licensed, runs as a background service that starts with your machine, and pulling a model is one command:

# install (Linux — macOS and Windows use the app download from ollama.com)
curl -fsSL https://ollama.com/install.sh | sh

# pull a model
ollama pull qwen3.6:35b

The part that matters for app integration: Ollama exposes an OpenAI-compatible API at http://localhost:11434/v1. Any app built to call OpenAI's chat-completions format can talk to your local model by swapping one base URL. No adapter, no plugin, no SDK change. That single design decision is why local AI stopped being exotic: the whole OpenAI-compatible app ecosystem became local-capable overnight.

LM Studio is the other popular runner, and the split is mostly about audience. LM Studio is proprietary freeware and GUI-first, great for a human browsing and testing models; it added a headless server mode in 2026, so it can run without the desktop app now too. Ollama is MIT-licensed and CLI-first, built to be the always-on service that code talks to. A sensible workflow: discover a model you like in LM Studio, then make Ollama the endpoint your apps depend on.

Which model should you run?

As of mid-2026, the practical picks by hardware tier:

• 8GB RAM/VRAM: Gemma 4 E4B, Llama 3.2 3B, or Phi-4-mini (3.8B). Fine for summaries, drafting, and simple extraction.
• 16GB: Qwen 3.6 35B-A3B runs well here: it is a mixture-of-experts model with only ~3B parameters active per token, so offloading experts to system RAM keeps it fast even though the full 4-bit download is around 24GB. Mistral Small 3.2 24B (a 15GB download at 4-bit) is workable too, with reduced context or a few layers offloaded.
• 24GB: Qwen 3.6 35B fully in VRAM, or Gemma 4 31B quantized. Solid general assistants.

Two caveats worth knowing before you pick:

warning: "Run Llama 4 on your laptop" is a myth. Llama 4's smallest model (Scout) is 109B total parameters and needs a datacenter GPU even quantized. The laptop-class Llama is still the 3.x line. If a guide tells you otherwise, it was written from the press release, not from a terminal.

The second caveat is tool calling. If your assistant does anything agentic, like creating tasks or querying your notes through function calls, the model must actually be trained for it. Qwen 3.6, Mistral Small 3.2, and IBM Granite 4 are proven tool callers with vendor-documented function calling. Gemma 4 added native function calling in April 2026 (Gemma 3 never had it), but independent benchmark data on how it compares is still thin, so the first three remain the safe picks for tool-heavy work.

How do you wire it into a real assistant?

This is where Calmara is the worked example, because it treats a local model as a first-class provider rather than a fallback:

1. Install Ollama and pull a model (commands above).
2. In Calmara, open Settings → AI & Automation and choose the Pure Local connection mode.
3. Set the endpoint URL to your Ollama address: http://localhost:11434 on the same machine, or a LAN address like http://192.168.0.50:11434 if the model runs on a beefier box elsewhere in the house.
4. Click Detect Models and pick the one you pulled.

Two implementation details make this genuinely zero-egress rather than nominally so. First, in local mode the browser talks to the model directly, skipping Calmara's own backend, so not even the app vendor sits in the request path. Second, a hard privacy mode (local_only) makes the backend refuse any cloud provider call with an error, so a misconfiguration fails loudly instead of leaking quietly.

If you want the whole stack under your control, pair the local model with Calmara's self-hosted database option: your notes and tasks live in your own PostgreSQL, and the AI that reads them runs on your own GPU. At that point the SaaS part of the product is the interface, and the data plane is entirely yours.

What are the tradeoffs?

An honest list, because there is no free lunch:

• A local model is not GPT-5 or Claude. A 24B model is impressively capable in 2026 and handles summarizing, drafting, extraction, and task parsing well. It will lose to frontier cloud models on hard reasoning. Match the model to the job.
• Hardware costs money. A machine with 16 to 24GB of VRAM (or an Apple Silicon Mac with that much unified memory) is the realistic entry point for the good models.
• You run the infrastructure. Model updates, disk space (models run roughly 2 to 20GB each), and the occasional restart are yours now.
• Speed varies. On the right GPU, local inference is fast. On a thin laptop with an 8GB model, expect a slower, chattier experience than a cloud API.

What you get in exchange is the only privacy guarantee that does not depend on anyone's terms of service. Not the app's, not the provider's. Data that never leaves cannot be retained, trained on, subpoenaed from a third party, or leaked in someone else's breach.

FAQ

What does zero cloud egress mean?

It means the AI model runs on hardware you control, so no prompt, document, or piece of context is ever transmitted to a cloud provider. Inference happens locally and the network sees nothing. It is a stronger guarantee than BYOK or zero-data-retention agreements, which still send data to the provider.

Is Ollama free?

Yes. Ollama is MIT-licensed open-source software, and the open-weight models it runs are free to download. Most are permissively licensed (Qwen, Gemma 4, Mistral, and Granite are Apache 2.0; Phi is MIT); Llama uses Meta's community license, which adds conditions for very large companies. Your only cost is the hardware.

What hardware do I need to run a local LLM?

As of mid-2026: 8GB of RAM or VRAM runs small 3-4B models for light tasks; 16GB runs Qwen 3.6 35B-A3B with expert offload (Mistral Small 3.2's 15GB build also works there with reduced context); 24GB runs the 24-30B class fully in VRAM. Apple Silicon unified memory largely counts toward these numbers, though macOS reserves a share of it, so a 16GB Mac gives roughly 10-11GB of model headroom.

Can I run Llama 4 on my laptop?

No. Llama 4's smallest variant (Scout) is 109B total parameters and needs a datacenter-class GPU even at aggressive quantization. For local Llama, use the 3.x models. Guides claiming laptop Llama 4 are wrong.

Is a local LLM as good as ChatGPT or Claude?

Not on hard reasoning tasks. But for the assistant workloads a productivity app actually needs, like summarizing notes, parsing tasks, drafting text, and answering questions over your own data, a well-chosen 16-24GB-class model in 2026 is genuinely good. The gap matters less the more grounded the task is.

What is the difference between Ollama and LM Studio?

Both are built on llama.cpp (LM Studio also uses Apple's MLX engine on Apple Silicon). LM Studio is proprietary freeware and GUI-first: browse models, chat, experiment, with a headless server mode added in 2026. Ollama is MIT-licensed and CLI-first: an always-on service exposing the OpenAI-compatible API apps integrate against. Discover in LM Studio, serve with Ollama.

Doesn't BYOK give me the same privacy?

No. BYOK sends your data to a cloud provider under your own API key and contract, which improves the terms governing the trip but not the fact of it. Zero cloud egress means there is no trip. They solve different problems; some apps, including Calmara, support both.

Can local models use tools and function calling?

The right ones can. Qwen 3.6, Mistral Small 3.2, and IBM Granite 4 are trained for reliable function calling and work for agentic flows. Gemma 4 gained native function calling in April 2026, but independent benchmarks on it are still thin. If your assistant creates tasks or queries data through tool calls, the first three are the proven picks.

← All articles

Written by Dan Hagen