Run AI on Your Own Machine with Ollama: Secure, Offline and User-Friendly
“How can we make sure our data shared with AI is safe?”
This question came up last week in the AI & Training Design Masterclass by Dr Philippa Hardman and Roy de Vries.
I had briefly answered the question in the chat:
— I can highly recommend https://ollama.com/ to use local AIs (Mixtral, for instance).
However, I could not go into detail about it. Therefore, I have quickly made a video on how we can run a local AI using Ollama.
I hope this gives a quick overview of what local AIs are capable of and how our data remains secure without leaving our computer.
I would love to hear about your experiences and the application areas you find most promising for a local AI!
You will find the video transcript below.
Video
Video Transcript
Have you ever hesitated to ask a question to ChatGPT and wondered what happens to your data?
Do companies like OpenAI, Microsoft and Google use your data to train their models?
What about health or financial data?
When we send our data to the cloud we must assume that it can potentially be viewed, shared or even disclosed to others.
But what if we could run an AI on our local machine?
A tool like Ollama allows us to do just that. You can download the program from ollama.com.
After installation, Ollama will ask if you want to install the command line program called Ollama.
It currently does not have a graphical user interface.
To get started, open the terminal on your Mac, which is a program that allows you to interact with the operating system directly.
You can find it through the spotlight search or by navigating to applications, utilities and then open the terminal.
Once the terminal is open, enter the command “ollama run mistral”.
This will download an AI model from Mistral AI, a Paris-based software company.
Please note that you will need a Mac with Apple Silicon, M1 and M2 or M3 chip and at least 16GB of RAM for the smaller models.
Larger models like Llama 13b from Meta or formerly Facebook or Mistral require more RAM.
The size of the downloaded model should not exceed 12GB for a Mac with 16GB of RAM.
After the model is loaded, you can start asking questions.
For example, you can ask “what type of animal in the Lama family has the finest wool?”
Ollama also comes with an API server allowing developers to create their own web tools that can access the AI model offline.
There is also the open web UI which enables you to select different models, chat with them, upload documents and write your own prompts without your text leaving your computer.
Another alternative is LM Studio which offers various models for chat, code generation and more.
While it may seem overwhelming at first, it provides a glimpse into the future of local AI.
In the future, local AI may become the preferred choice for many applications allowing us to ask questions without worrying about our data.
If you have already experimented with local AIs, I would love to hear about your experiences and the application areas you find most promising.
See you next time.