For the past few months I’ve been experimenting with AI tools to help me live more independently.
It’s very much an experiment… a way to learn about the technology behind AI and see if it can reduce some of the day to day barriers I experience.
For example, I’ve been asking it how long to microwave things for, or what clothing would be suitable for the weather.
I’m pretty skeptical of large AI services (like ChatGPT etc)… in truth they scare me a little and I find the lack of privacy alarming. I don’t trust them in the slightest.
With that in mind, my interest in AI is entirely focused on things which run ‘locally’ on my own hardware and with complete privacy.
This blog post is a little overview of the tools i’m using and how I’ve got them setup.
The basic idea is to setup a ‘AI at home’ service which is accessible to anything on my home network.
The service lets me use AI tools from anywhere in the house using whatever device I happen to be holding. It works on my phone, Mac, PC etc etc.
To access the my AI tools, I open a web browser and go to http://ai.local which loads a website where I can open ‘chats’ with whatever AI models I have downloaded.
It’s pretty neat. The ‘webUI’ & AI is entirely hosted on the Mac in my lounge so everything is kept private. It also has a sign in system so everyone in the house can have their own account. Once signed in, it can remember previous chats etc etc.
The ‘AI at home’ system has 4 basic parts. The hardware, the runtime, the models and the web UI.
Before starting this project I assumed AI would require some massive expensive server. Turns out… that’s not entirely the case.
A more powerful system can run larger and more complex models, but many of the most interesting models are designed to run on fairly modest desktop and laptop hardware.
In my case I’m using a 2022 Mac Studio with an M1 Ultra processor as it’s what I happened to have on my desk. However I’ve also used the same approach on a M4 Mac Mini and an older M1 MacBook Air.
Most AI models can be run stand alone like any other software…however that’s a bit of a faff. Each model is slightly different and needs a lot of manual setup.
With that in mind, I’ve been using an AI runtime called ‘Ollama’.
Ollama is a command line tool which lets me download and run lots of different models very easily.
With Ollama installed, I can start to access the actual AI bits.
The building blocks of AI are referred to as a ‘models’. Each model has been trained on some data with a specific goal in mind.
Training models is wildly expensive and time consuming. But once trained they can be used for fairly little energy and cost.
The ‘models’ themselves vary in size from 2-200gb. The size of a model is normally appended after the name after a semi-colon ‘:’. The size normally indicates how many ‘billion’ of data points are are inside the model.
For example, there an image analysis model I like called LLaVA. It comes in three sizes, 7b, 13b and 20b.
To run the smallest version of the model in ollama I’d use the command:
$ ollama llava:7b
The first time I run the command Ollama (the AI runtime tool) will download the model and set it up for me.
The 7b version of LLaVA is a 4.7gn download and needs around 8gb of RAM to run.
Once it’s downloaded I can then pass it an image and ask it questions like “how many items are in this image” etc.
The largest model i use is (confusingly) named ‘llama3.1:70b’. Yeah.. it has a daft name very similar to the AI runtime called ‘Ollama’.
The llama3.1:70b model is much bigger. It’s a 43gb download and needs around 50-60gb of RAM to run. It works on the Mac Studio, but it’s too big to run on my MacBook.
There are hundreds of models to choose from all in different sizes. The Ollama website has a database of models which shows the size and gives the correct command to download and run them. All of the models are free to download and use. Most of them are open source.
With Ollama setup and some models downloaded. The last task is to add the webUI and make it work across the network.
The website I sign into to access the AI at http//ai.local is powered by the ‘open-webUI’ project.
To get it working I had to install Python 3.11 from the Python website… and then run the following command:
$ pip3 install open-webui
It installed first time without any issues (!).
Once installed, I start the server with the command:
$ open-webui serve
And that’s it… I can view the webUI to sign in and create my account at http://localhost:8080.
The webUI is served to anything on the local network. By default, it won’t be accessible across the internet unless you setup specific firewall rules and routing to make it work.
To make it easy to access on my local Network, I renamed my Mac to ‘ai’ so I could get to it by typing ‘http://ai.local’ instead of the IP address.
So… that’s how it works and how I set it up. I’m still experimenting and learning about this stuff. I’ll probably write about it again in the future when I’ve got more to share.
If you get stuck or want to chat about this stuff. Feel free to message me on my new BlueSky account over at jamieandlion.bsky.social