As I wrote on my intro post, I started exploring local inference because of Immich, which has built-in image recognition powered by CLIP models running locally. I’m a tinkerer by nature, and this scratched that itch. Seeing something as powerful as Immich run fully locally, in a completely private way, made me lean even harder into open source and keeping my digital life private.
So why bother, when ChatGPT and Claude exist and are cheap? Partly because messing with this stuff is genuinely fun. But mostly because the trend looks clear to me: small models you can run at home are getting good fast, and before long they’ll handle a lot of everyday tasks without anything leaving your home. Plus I get to keep my data on my own hardware and learn how all this actually works along the way - that’s the fun of it.
Once I added a 3090 to my home server, the first thing I did was to spin up an Ollama container on the server, as well as install Ollama on my desktop.
Ollama
Ollama is the natural starting point for most people looking to run local models. They were early to the space and made it as easy as possible to be up and running in no time. Just:
curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3.5:9bWait a few minutes for the model to download, and done - you’re running a small model on your own computer (as long as you have a half-decent GPU with enough VRAM for the model you’re trying to run). Ollama has evolved really fast: it started out as a wrapper around llama.cpp, but these days it’s a whole platform - local models, a cloud subscription for bigger open models, and integrations with coding tools.
On my server I also added Open WebUI as the chat interface, and that has stuck with me ever since - though these days I mostly live in the terminal.
I didn’t stick with Ollama for long, though, mainly because of how it handles model files. Ollama can import any GGUF, but it re-stores it in its own blob-and-manifest layout under ~/.ollama, so the exact same file isn’t directly shareable between my desktop and server. And the curated ollama pull library only exposes a limited menu of quantizations - no dynamic or XL quants from the likes of Unsloth, for example. For someone who wants to grab a specific quant from HuggingFace and reuse the very same file everywhere, that friction adds up.
Performance was part of it too, at least back then - when I tested, Ollama was noticeably slower on the same model and quant than tools running llama.cpp directly. It’s possible they’ve caught up since; I honestly haven’t kept track. Either way, the flexibility is what keeps me on llama.cpp-based tools. So on my desktop I settled on LM Studio, and on my server I moved to llama-swap - both llama.cpp under the hood.
llama.cpp
llama.cpp was created by Georgi Gerganov back in 2023, days after Meta’s LLaMA weights leaked online. The idea was simple: run models in pure C++, on a CPU, no GPU needed. It spread like wildfire. Today it’s the engine under most local inference tools - including LM Studio and, originally, Ollama.
On my desktop, LM Studio gives me a GUI over the engine with model browsing and GGUF downloads straight from HuggingFace, as well as a built-in OpenAI-compatible server for when I want to point other tools at it.
On my server I use llama-swap for the quality-of-life features. It hot-swaps between multiple models on demand and is incredibly easy to set up. It exposes the most popular API endpoints (OpenAI, Anthropic, etc.), making it the ideal drop-in replacement for whatever model provider a given tool expects. The swapping is the killer feature for me: I keep several models available - a different one depending on whether I’m writing, coding, or running agents - and llama-swap loads whichever one the incoming request asks for without me having to manually load it. Perfect setup for tinkering and tuning!
A recent example that highlights how fast the local inference space is moving: llama.cpp landed support for MTP (multi-token prediction) on models that ship those weights, which gave me a ~70% (!) speed bump in token generation for free - exactly the kind of stuff that makes this worth the hassle.
Now you might be wondering “why not go vLLM or SGLang?” - those are great when you’re serving many requests or spanning multiple GPUs, but on a single 24GB card llama.cpp’s flexibility is a no-brainer. I’ll revisit if/when I upgrade to an RTX 6000… just don’t let my partner see how much that costs. :)
What models to run - and how to know if they’re good
The local model space is evolving at breakneck speed. Right now my workhorse is Qwen3.6-27b, generally seen as the most competent model you can run on a 24GB GPU at 4-bit, and which I lean on for agentic and coding work. For writing I reach for Gemma4-31b from Google, generally considered the best local model for creative work.
Those models ship at different quantization levels, which is roughly how much numerical precision you trade away to fit a bigger model into less VRAM. I won’t go deep here, but it matters a lot: Alex Ziskind made a really cool video on how quality scales with quantization, check it out.
We’ve been spoiled with strong new open source models almost monthly, so it’s important to have a thorough way to evaluate them - aka “evals”. There are plenty of benchmarks that get referenced whenever a new model drops - SWE-bench Verified for coding, MMLU-Pro for knowledge, LMArena, etc. - but these can be gamed, and they measure many things, many of which you might not care about. There’s also the contamination problem: once a benchmark is public, its questions tend to leak into the next round of training data, so a great score can just mean the model has seen the test. So I’ve set up a few different ways to evaluate models while doing the things I do care about - my own eval framework. More on that below.
So what do I do with these models?
Some examples:
- Web search - I run a local SearXNG instance to power this. Kagi has been my main search engine for a while, but a local model + SearXNG is enough for the majority of my searches.
- Coding - inside VSCode via Roo Code, or through opencode (I haven’t gotten around to testing Pi yet).
- Personal assistant - “El Barto”, built on Hermes, does a ton of useful things for me. I believe there’s a bright future ahead for “AI personal assistants”, and mine will run locally.
- Second brain - llama-swap + opencode + an Obsidian vault has turned into a very valuable workflow.
- Home automation - a local model wired into Home Assistant lets me control stuff around the house in plain language.
- Building the homelab - probably the biggest one. A model to bounce ideas off along the way is how I figured out things like my current split-DNS setup, with separate reverse proxies for inside and outside the house - all far faster than I’d have managed on my own.
What’s next
This is the first in a series. A few things I’m working on (as time allows) that should become their own posts:
- My eval framework - how I actually decide if a new model performs better than my current one, using the stuff I care about instead of public benchmarks. That’s the next post.
- El Barto - my local personal assistant built on Hermes: what it does, and how it all hangs together.
- Squeezing the hardware - playing with hybrid CPU+GPU inference (ik_llama). With 512GB of RAM in the server, I can offload bits of much bigger models to system RAM and run things like gpt-oss-120b - though I’m pretty sure there’s more speed hiding in there.
- Observability - actually knowing what is going on in my serving stack: monitoring, metrics, and where the bottlenecks are.
That’s what my setup looks like today - a bit messy, always iterating, and never a dull moment. My next post will be about evals.
If you want to follow along, the blog has an RSS feed, or find me on LinkedIn.