BLUF: It's absolutely possible to set up a functional, airgap-capable LLM system at home. It might not be overly expensive, too.
That said, when I got started in this, it was shortly after the drop of GPT3. I got deep into trying to use it for all manner of things and set about creating tooling. More in a different article, but one thing that became pretty clear after conducting research and building a few prototypes is that I shouldn't trust the cloud providers for preserving the sanctity of my requests. Things which I was guaranteed would be private soon started showing up in other folks' sessions, which, for someone doing commercially-focused research, was a bad sign. Thus began the saga into local compute.
A bit later, maybe a year, I started working with a client who would pay me in funds to build a local compute R&D setup to experiment with these tools offline. While that didn't exactly happen, by the time I realized it was too late. An order to Puget Systems was already on the way and I would soon be the proud possessor of a new computer, lovingly titled the Monolith. I believe the LLaMa 2 release was the inspiration for this foolhardy buy.
At the time, running models wasn't very clean. I did a little research on the semantic transferability of prompts between models (and I still need to write up what I found on cognitive substrates - it's pretty cool), but the infrastructure was bad and I found other uses for the machine. It also required extra power and cooling because running multiple high end GPUs on full tilt cooks you if you're not careful. It's currently 84F in here and I've only been working for part of a day.
Anyway, fast forward to about two months ago and a friend introduced me to OpenCode. I had tried aider before and realized that, at least for software, going from "spending all day solving little puzzles, from algorithmic to infrastructural" to "spending all day reviewing pull requests" was a special kind of hell. OpenCode doesn't seem to have that limitation, and best, you can point it to many, many different backends.
This isn't intended to be a plug for OpenCode, but it was the motivation to get this whole stack set up. I ended up taking advantage of the low cost-to-leverage Gemini Pro to help research and get the configuration right and eventually settled on using vllm to serve Poolside Laguna, which is a frontier model in its own weight.
The trick was to avoid any loading system that used an odd number of cards. None of these tools seem to handle asymmetric layer splits, which is a disadvantage for an at-home setup. I found that the throughput gains for even an int4 model far outweighed any issues in capability, especially with whatever OpenCode is doing for prompting and sub-agent design.
After getting that running and having a tool to help improve the tool (offline), I started looking for other options. Open WebUI (OWU) is a pretty strong replacement for the Chat interface and also plugs nicely into the vllm backend. Plus, it has support for image edit and, notionally, image/image models (although I still have yet to get that to work). The beauty of an at-home system is, in addition to being able to experiment with different vendors' approaches, you can also do side by side comparisons between models.
My current stack is something like this:
2x GPUS -> vllm -> Laguna -> OpenCode | OWU 1x GPU -> FastAPI shim -> Qwen Image or Image Edit (depending on the project) -> Open WebUI | CLI tools CPU -> FLUX Schnell -> Open WebUI | CLI tools
The following have been shelved for the moment for various reasons:
- aphrodite - seemed less developed than vllm/less supported
- FLUX 1/2 dev/ - non-commercial license (might come back to this later), huge
- Llama 3 - Weird license, less recent
- SDXL - Less quality than Qwen, seemed less fast than Schnell
- ComfyUI - I think I need to sit down with a lot more tutorials, the UI is a lot the first time
Bonus Notes:
- If you serve via vllm, definitely add the
--revisionflag, because I lost a day to debugging when Hugging Face updated my weights silently in the background and the new weights were bigger and no longer fit - Compaction in OWU is worse than OpenCode, but I think that can be fixed with some clever patching
- Integrating Exa search is a good match for OWU and their free tier is more than enough for the volume of research I'm doing
- Docker deploy for OWU is great, but I'm not sure the hassle for vllm/GPU passthrough is worth it, just make sure your
venvs are up to date uvhas been a star for most of this environment setup work. Much better than pipenv or poetry, in my experience, at least for deployment
If you have any questions or want to dive deeper, come join office hours or book a 1:1!
