Is using cloud AI models for your more delicate tasks a good idea? Here are the use cases, prerequisites, and benefits of a local AI stack.

Everybody uses frontier AI providers such as ChatGPT, Claude, or Gemini. But at the same time, more and more people complain about their data flowing into the training sets of big, foreign AI models.

The good news? There exist possibilities to run an AI stack locally.

The bad news? It will require a little technical knowledge to set this up, and people will most probably call you a prepper. You shouldn’t care. You’re not setting up a local AI stack for others, but for yourself.

Let’s look at your local AI stack in more detail.

Use Cases

Why would you even consider a local AI stack? Think about it. You’ve probably uploaded contracts to ChatGPT or Claude before, asking questions about them instead of paying a lawyer or reading the entire contract yourself.

Of course, that’s an efficient way of getting a quick answer to a contractual question. But do you really want your contracts used to train big foreign frontier models?

Another use case for AI is summarizing lengthy reports and presentations, or finding contradictions within them. For example, in my assignment as an active reserve officer in the Swiss Armed Forces, I receive many lengthy reports and presentations. Still, I’m not allowed to upload them to any cloud AI tool — irrespective of whether the provider is in Switzerland or in a foreign country.

A third use case for local AI is to use your personal note-taking tool with AI. I use Obsidian to organize my notes in graph format. That’s an ideal format for using data with an AI model, but I surely don’t want to transfer my personal notes to any cloud AI system.

Prerequisites

Now, what do you need to run a decent local AI stack?

First, you need a computer with tons of RAM. Anything below 64 GB will not give you any meaningful benefit. With the AI boom still in full swing, prices for such devices have gone up, and availability has deteriorated. The good news is that you don’t always need the latest processors, as long as you have enough RAM. Speaking for myself, I bought a MacBook Pro M2 with 96 GB of RAM — a three-year-old device that still does the trick thanks to its huge RAM.

Second, you need containerization software to run the AI models on your machine. I recommend Docker; it’s easy to set up on a local machine, and it supports many different AI models. Furthermore, destroying and recreating containers is easy.

Last but not least, you need a web interface to chat with your local AI models the same way as you know it from ChatGPT, Claude, or Gemini. Open WebUI is possibly the easiest and best way to facilitate this.

The good news? Many AI models, Docker, and Open WebUI are open-source software that you can use for free for personal use.

The bad news? Even if you buy a second-hand laptop with 64+ GB of RAM, that’s going to cost you a bit of money.

Benefits

The proposed stack brings you one step closer to data sovereignty: What happens on your machine stays on your machine. Even when using AI. Depending on your field of activity, this will boost your productivity without sacrificing your privacy.

And if you still don’t trust your local AI models to perform delicate analysis, you can destroy and recreate the Docker containers once you’re done with your delicate task.