• Ollama commands pull and run are used to download and execute LLMs respectively, just like the ones used to manage containers with podman or docker.

• Tags like 13b-python and 7b-code are used to manage different variations of an LLM.

• A Modelfile (like Dockerfile) is created to build a custom model using an existing LLM as its base. Additional parameters like TEMPLATE and PARAMETER could be used to define a prompt template or fine-tune model parameters respectively.

Deploying Ollama container with NVIDIA GPU

Deploying the Ollama container directly would allow it to utilize CPU resources for its LLM workloads, but with the parallel computation capabilities of a Graphics Processing Unit (GPU), we can improve the inference performance of all models.

In this article I’m using an NVIDIA GeForce RTX 3070 Ti GPU, if you want to use a GPU from AMD/Intel or any other manufacturer then steps like driver and container toolkit installation and GPU configuration for the container engine will differ.

GPU Passthrough to VM

I am deploying the Ollama container on a Fedora 38 virtual machine so the first step will be the GPU Passthrough from my hypervisor (Proxmox) to the VM. You can skip this step if you are deploying Ollama on a baremetal machine.

In the Proxmox’s Web UI, we can go to the VM’s Hardware section and Add your PCI Device i.e. your GPU.

Proxmox VM's Hardware Section

Make sure to mark the All Functions checkbox.

GPU Passthrough to a Proxmox VM

Once the VM is rebooted we can verify the GPU Passthrough using the following command.

1

lspci | grep NVIDIA

If the GPU name is present in the command’s output (like below) then the passthrough is successful and we can move to the next step.

1
2

06:10.0 VGA compatible controller: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti] (rev a1)
06:10.1 Audio device: NVIDIA Corporation GA104 High Definition Audio Controller (rev a1)

CUDA Toolkit Installation

To utilize the parallel computation capabilities of the CUDA cores provided in NVIDIA GPUs we have to install the CUDA Toolkit. You can follow NVIDIA’s documentation on the CUDA Toolkit installation on Linux because the steps vary depending on the host’s configuration.

Here are the steps for Fedora 38:

1. Downloading CUDA Toolkit Repo RPM.

1

wget https://developer.download.nvidia.com/compute/cuda/12.3.2/local_installers/cuda-repo-fedora37-12-3-local-12.3.2_545.23.08-1.x86_64.rpm

2. Installing CUDA Toolkit Repo RPM.

1

sudo rpm -i cuda-repo-fedora37-12-3-local-12.3.2_545.23.08-1.x86_64.rpm

3. Cleaning dnf Repository Metadata.

1

sudo dnf clean all

4. Installing cuda-toolkit package.

1

sudo dnf -y install cuda-toolkit-12-3

5. Installing legacy (proprietary) or open (open source) kernel module for nvidia-driver.

1

sudo dnf -y module install nvidia-driver:latest-dkms

or

1

sudo dnf -y module install nvidia-driver:open-dkms

NVIDIA Container Toolkit Installation

With nvidia-container-tookit, we can use our NVIDIA GPU in containerized applications. Here are the steps for installing NVIDIA Container Toolkit on Fedora 38:

1. Adding nvidia-container-tookit repository.

1
2

curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
    | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

2. Installing the nvidia-container-tookit package.

1

sudo dnf install -y nvidia-container-toolkit

3. Once the container toolkit is installed, we have to add its runtime to our container engine.

1

sudo nvidia-ctk runtime configure --runtime=docker

4. Finally, we can start using our NVIDIA GPU with Docker containers after restarting the docker Daemon.

1

sudo systemctl restart docker

Deploying Ollama as a Docker Container

1. Create a directory on our host to store LLMs to avoid re-downloading models after reprovisioning or updating the container.

1

mkdir -p ~/container-data/ollama

2. The following compose.yaml file will deploy the ollama container with our NVIDIA GPU.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

version: '3.6'

services:
  ollama:
    container_name: ollama
    image: ollama/ollama:latest
    volumes:
      - ~/container-data/ollama:/root/.ollama
    ports:
      - "11434:11434"
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

If you want to provision the container without GPU you have to remove the deploy section.

3. Deploy the ollama container using the following command.

1

docker compose down && docker compose up -d

If you want to deploy Ollama with a ChatGPT-Style Web UI then follow the deployment steps for Ollama Web UI.

Managing LLMs using Ollama

Once the container is provisioned we can start downloading and executing models.

To attach the Ollama container with a terminal use the following command

1

docker exec -it ollama /bin/bash

Downloading LLMs using the pull command

To download a model use the ollama pull command with the name of LLM and its tag (refer to the Ollama Library).

For example, to download the Code Llama model with 7 Billion parameters we have to pull the codellama:7b model.

1

ollama pull codellama:7b

The model size could range from 4 to 19 GB (or even more). So choosing the right model tag is crucial to decrease downloading time and resource utilization.

If we want to delete a downloaded model we’ll use the ollama rm command followed by the name of the model.

Executing LLMs using the run command

Before we prompt the model we have to run it first using the ollama run command followed by the name of the model.

1

ollama run codellama:7b

This command will drop us directly into the model’s prompting window.

1
2
3
4
5
6
7
8

>>> Who are you?
I am LLaMA, an AI assistant developed by Meta AI that 
can understand and respond to human input in a conversational 
manner. I am trained on a massive dataset of text from the 
internet and can generate human-like responses to a wide range 
of topics and questions. I can be used to create chatbots, 
virtual assistants, and other applications that require natural 
language understanding and generation capabilities.

Prompting LLMs from Command Line

Ollama exposes multiple REST API endpoints to manage and interact with the models

• /api/tags: To list all the local models.
• /api/generate: To generate a response from an LLM with the prompt passed as input.
• /api/chat: To generate the next chat response from an LLM. The prior chat history could be passed as input.

We can perform these API requests using curl and format the response using jq.

1
2
3
4
5

curl -d '{
  "model": "codellama:7b",
  "prompt": "Write a quicksort program in Go",
  "stream": false
}' http://localhost:11434/api/generate | jq -r ".response"

By assigning the stream as false we will receive the complete response as a single JSON object rather than a stream of multiple objects.

Ollama Web UI

With self-hosted applications, it always helps to have a web interface for management and access from any device. The Ollama Web UI provides an interface similar to ChatGPT to interact with LLMs present in Ollama.

Deploying Ollama Web UI

Similar to the ollama container deployment we will create a data directory for ollama-webui

1

mkdir -p ~/container-data/ollama-webui

Modify our existing compose.yaml.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

version: '3.6'

services:
  ollama:
    container_name: ollama
    image: ollama/ollama:latest
    volumes:
      - ~/container-data/ollama:/root/.ollama
    ports:
      - "11434:11434"
    restart: unless-stopped
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
  ollama-webui:
    container_name: ollama-webui
    image: ghcr.io/ollama-webui/ollama-webui:main
    ports:
        - "3030:8080"
    extra_hosts:
        - host.docker.internal:host-gateway
    volumes:
        - ~/container-data/ollama-webui:/app/backend/data
    restart: always

Deploy both containers using docker compose.

1

docker compose down && docker compose up -d

If the ollama container is deployed on a different host then we have to rebuild the ollama-webui container image by following the instructions from here.

Managing LLMs from Ollama Web UI

Once the deployment is completed we can visit the web UI at localhost:3030.

Ollama Web UI

Alongside prompting we can also use the Web UI to manage models.

Managing models using Ollama Web UI

Integrating Ollama with Neovim

If you are using Neovim (like me) then you can integrate models in your development environment using ollama.nvim.

ollama.nvim supports the following features:

• Code generation from a text prompt
• Generating an explanation for a code snippet
• Code modification suggestions

Code explanation from Ollama using ollama.nvim

I am using LazyVim so I’ve created ~/.config/nvim/lua/plugins/ollama.lua with the following content.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

return {
	"nomnivore/ollama.nvim",
	dependencies = {
		"nvim-lua/plenary.nvim",
	},
	-- All the user commands added by the plugin
	cmd = { "Ollama", "OllamaModel", "OllamaServe", "OllamaServeStop" },
	keys = {
		-- Sample keybind for prompt menu.
		-- Note that the <c-u> is important for selections
		-- to work properly.
		{
			"<leader>oo",
			":<c-u>lua require('ollama').prompt()<cr>",
			desc = "ollama prompt",
			mode = { "n", "v" },
		},
		-- Sample keybind for direct prompting.
		-- Note that the <c-u> is important for selections
		-- to work properly.
		{
			"<leader>oG",
			":<c-u>lua require('ollama').prompt('Generate_Code')<cr>",
			desc = "ollama Generate Code",
			mode = { "n", "v" },
		},
	},
	---@type Ollama.Config
	opts = {
		model = "codellama:7b",
		url = "http://127.0.0.1:11434",
	},
}

Integrating Ollama with VSCode

The Continue VSCode extension supports the integration of LLMs as coding assistants. To use it with Ollama we have to change the Proxy Server Url in its settings to the one used by our Ollama container.

Continue Extension Settings

Watch Ollama in action inside VSCode

Optimizing code using Ollama in VSCode

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources

Ollama
Ollama Docker Image
NVIDIA GeForce RTX 3070 Ti
NVIDIA CUDA Installation Guide for Linux
NVIDIA Container Toolkit
Ollama Library
Ollama Web UI
ollama.nvim
Continue

• Each application has a different set of terms and services. A small subset of users read it and a smaller subset of them will refuse to use the application if they disagree with it.

• These applications are critical for your daily life but if you lose access to them, there might not be a proper support channel to regain access or retrieve data.

If you lose access to your Google account you’ll also lose access to your emails, your photos (backed up on Google Photos), your YouTube account, and probably most of the services you signed in using Google.

• Many of them operate on a subscription-based model which means you don’t own the product outright, you are just renting it as a service.

The application provider can change their payment model to subscription-based, while also discontinuing your existing “lifetime” license. G Suite legacy free edition, TeamViewer

The service itself could be discontinued. Killed by Google

Content “owned” by you could be removed without your consent. Sony Removing Some Purchased Films, Movies from Users’ Accounts

Self-Hosted alternatives to some or all of these services will give you more control while also keeping your data private. There are multiple hardware options available across the budget that could be used to host services like media streaming, cloud storage, VPN server, etc. at your home.

Mini PCs

Mini PCs provide maximum utility for homelab but in a smaller package with low power consumption and noise.

If you want to get started with self-hosting or deploy a secondary server for services running round-the-clock (like a password manager, DNS, router, etc.) then a mini PC would be a great option.

Raspberry Pi and Alternatives

Photo by Harrison Broadbent on Unsplash

Raspberry Pi is a credit-card-sized mini PC that uses CPUs based on ARM architecture. Due to its modularity and open design, it is used for monitoring weather, robotics, and teaching basic computer science.

General-Purpose Input/Output (GPIO) pins on Raspberry Pi could be used for integrating it with different circuits or appliances to monitor/control them or extend their functionality.

It is a great candidate for self-hosting due to

• its smaller size
• a large community of developers
• minimal energy consumption
• relatively affordable compared to other options

At the time of writing the Raspberry Pis are unavailable in most countries or sold at a price significantly higher than their MSRP so you can explore its alternatives in this video from LinusTechTips: I Can Save You Money! – Raspberry Pi Alternatives

Intel NUC and Alternatives

Photo by Gabriel Vasiliu on Unsplash

Next Unit of Computing (NUC) kits (PC kits without memory and storage) from Intel provide a relatively powerful alternative to Raspberry Pi or similar mini PCs.

NUCs are available with four types of Intel x86 processors: Celeron, Core i3, Core i5, and Core i7. NUC Extreme could be configured with a Core i9 processor and a desktop GPU.

MiniPCs like Beelink’s GTR, SER, and SR series provide an AMD alternative to NUC in similar form factors.

Limitations of Using Mini PCs for Self-Hosting

While MiniPCs are a great starting point for self-hosting services, there are still some limitations that you have to keep in mind

• MiniPCs do provide options for expansion and upgrades but relatively less than a custom PC build or a server.
• You might not be able to run multiple intensive workloads in parallel.
• To scale your homelab running on MiniPC you have to create a cluster of mini PC nodes.

Cloud Service Providers (CSPs)

You can host your services on VMs provided by Cloud Service Providers (CSPs) like Azure, Amazon Web Services (AWS), and Google Cloud Platform (GCP). They also provide a static IP for VMs which could be useful if you want your services publicly available.

Limitations of Using CSPs for Self-Hosting

• You are dependent on your CSP for the availability of your services. A CSP outage could also mean a homelab outage.
• Depending on your workload and your geographical location, hosting and running a VM round-the-clock might be expensive due to cloud costs.
• Scaling your lab on the cloud could be expensive.

Decommissioned Servers/Workstations

Photo by Taylor Vick on Unsplash

Datacenters and IT organizations often decommission servers and sell them on the secondary market. They might not perform well with the current enterprise workloads but their performance should be adequate for homelab.

Some enterprise features could help you in managing your homelab like

• 10 Gigabit Ethernet: If you are running a media streaming service or a video editing file server then having 10GbE could significantly improve your workflow.
• Intelligent Platform Management Interface (IPMI): A Motherboard with IPMI allows you to access your system’s firmware remotely. Using this you can diagnose firmware/hardware-related issues or install OS without connecting a display or peripherals.
• Error Correction Code (ECC) memory: ECC memory unlike non-ECC memory automatically detects and corrects memory errors. It is a necessary feature for critical workloads as it prevents unexpected crashes and outages.
• Ability to swap Power Supply Units (PSUs) and storage devices easily.

Limitations of Using Decommissioned Servers for Self-Hosting

• Scarce availability of replacements for components.
• Datacenter servers are designed to provide maximum cooling with the tradeoff of the high noise level. This might not be ideal if you want to deploy it in your home.
• The footprint of a server might be larger than an average PC.

Custom PC Build

Not my PC. Photo by Amiel D Hechanova on Unsplash

Custom PC builds provide maximum upgradability and customization and that’s why I decided to build one. I’ll go through my thought process for selecting each component, while it might not be the same for you, it can provide a direction for decision-making.

My planned use case for homelab:

• Hosting a Network Attached Storage (NAS) server
• Deploying Kubernetes/Red Hat OpenShift Local cluster
• Hosting a Virtual Private Network (VPN) server
• Hosting a media streaming server
• Windows VM for gaming
• Trying different self-hosted applications

Based on my use case I decided on the following specification:

• More than 8 cores and 16 threads for simultaneous execution of multiple VMs and containers. I decided that I’ll allocate at least 4 threads to my Windows VM.
• 2TB of redundant storage i.e. two drives mirrored on NAS.
• 2.5 Gigabit networking on the motherboard as I’ll be accessing services remotely.
• 32GB memory would be adequate for VMs. I went with DDR4 because DDR5 exceeds my budget.
• 1TB NVMe SSD storage for the host OS/hypervisor to be installed on homelab.
• PCIe bandwidth and PSU wattage to support Nvidia’s GeForce RTX 3070/RTX 4070 GPU with an expansion card like Network Interface Card (NIC) for a possible 10 Gigabit upgrade in the future.
• 2 X 2TB HDD for NAS server.

Skip to My PC Build Specification »

CPU

You should make sure that your CPU has support for virtualization with enough cores and threads to host and deploy multiple services concurrently.

Depending on your goals you should also look for:

• Overclocking support
• ECC Memory support
• Integrated Graphics: If your CPU doesn’t have integrated graphics then you have to purchase a GPU to install or access the OS/hypervisor.

I wanted a CPU with more than or equal to 8 cores with integrated graphics, the options in my budget were: AMD Ryzen 7 7700 or Intel Core i5-13500. I didn’t have any overclocking requirements, otherwise, I would’ve gone with Intel Core i5-13600K.

I went with the i5-13500 as it has more cores and threads compared to Ryzen 7700.

Motherboard

The choice of your CPU limits the choices of your motherboard. First, you’ll have to select the motherboard chipset compatible with your CPU:

• AMD chipsets: B650, B650E, X670, X670E
• Intel 700 series chipsets: Z790, H770, B760, W790

Motherboard features you might want to consider for your homelab:

• Upgradability path with future CPU releases
• PCIe Bandwidth for expansion cards
• 2.5 Gigabit networking
• SATA ports for storage expansion
• IPMI support

I planned on getting MSI PRO B760M-A WIFI DDR4 but I got a better deal with MSI PRO Z690-P WIFI DDR4 so I went with that. It has 2.5 Gigabit networking and 6x SATA ports which provides me the option to add more drives in the future.

CPU Cooler

Most CPUs come with an air cooler. But if you are using a high-end CPU you might not be able to extract its complete performance using the default cooler.

Options are air cooling, custom water cooling, or an All-In-One (AIO) Liquid cooler. You can make your decision based on your CPU, cooling performance, noise level, and ease of installation.

I went with Corsair H100 RGB 240mm Liquid CPU Cooler due to its price and aesthetics.

Memory

At the time of writing the price of DDR5 memory is significantly higher than its MSRP so I opted for DDR4.

I went with 2 x Corsair VENGEANCE LPX 16GB DDR4 DRAM Memory Kit because its 3200MHz speed will be adequate for my workload.

Storage

Network Attached Storage (NAS) server is a common self-hosted service. With a NAS server, you have your cloud storage without privacy concerns or subscriptions. That’s why selecting the right storage device is important.

Along with drives for NAS you’ll need additional storage for the OS/Hypervisor running on your homelab.

Boot Drive

It’s ideal to store the OS/Hypervisor in a faster drive compared to the drives used for NAS, like NVMe or SATA Solid-State Drive (SSD).

I’m using Kingston NV2 PCIe 4.0 NVMe SSD as the boot disk for the Proxmox hypervisor.

NAS Drives

Although you can use any Hard Disk Drive (HDD) for your NAS server you might want to consider NAS-rated hard drives because they provide advantages like

• Optimization for read-and-write operations
• Relatively fewer vibrations compared to desktop hard drives
• Rated for 24/7 operation

If you are using two hard drives in mirrored mode then it is safer to source them from different vendors.

I opted for WD Blue PC Desktop Hard Drive 2TB and Seagate BarraCuda Hard Drives 2TB as storage drives. I’ll be running them in mirrored mode using TrueNAS.

Power Supply Unit (PSU)

Websites like OuterVision Power Supply Calculator could be used to calculate the power budget of your PC. It’s ideal to select a power supply with higher wattage to ensure upgradability.

PSUs are rated as White, Bronze, Silver, Gold, Platinum, and Titanium in the increasing order of their efficiency. An efficient PSU will draw less power and run quieter. Hence, reducing your power bill.

I opted for Corsair RM850 850 Watt 80 PLUS Gold Certified Fully Modular PSU which provides 850W of wattage and that is enough to run my homelab while also providing an option to add an Nvidia GeForce RTX 3070/RTX 4070 in the future.

PCIe Expansion Cards

If you are planning to use your homelab for video editing, animation, machine learning, gaming, or any similar workloads then adding a GPU can improve its performance significantly.

By adding a 10 Gigabit Network Interface Card (NIC) you can reduce the access time to your services.

Case

As long as all the components fit, any PC case could be used for homelab. You might want to consider its

• options for expansion
• cooling
• build quality
• cable management

I selected Corsair 4000D AIRFLOW Tempered Glass Mid-Tower ATX Case due to its airflow and ease of building.

Limitations of Using Custom PC Build for Self-Hosting

• While it is easy to build PCs by following guides and tutorials on the internet. It might still be very difficult for novice builders. Retailers do provide an option to assemble the components, sometimes at no extra cost.
• Current prices of some components are significantly higher than their MSRPs, even double in some cases.
• The upfront cost is higher compared to other options.

My PC Build Specification

The Result

Challenges of Self-Hosting

• Deploying and managing a homelab could be difficult for people who don’t work with computers. But it is possible with patience and research.
• Managing some services like an email server is difficult, especially when they integrate with other services.
• You have to manage and debug issues yourself. You’ll learn a lot, which might help you in your professional career.
• You have to provide technical support to everyone else using your services.
• You might lose out on some features or their quality provided by proprietary services. Like AI face recognition provided with Google Photos.
• You are responsible for the security of your data. Make sure that all of the services, OS, and hypervisors are up to date.
• For some applications, you might not have full autonomy to switch to a self-hosted alternative. For example, if your friends and family use WhatsApp then you can’t simply switch to a self-hosted alternative and expect others to do the same.

Thank you for taking the time to read this blog post! Have questions, feedback or want to discuss this topic? Feel free to reach out at blog@avni.sh.

If you found this content valuable and would like to stay updated with my latest posts, consider subscribing to my RSS Feed.

Resources to Learn About Homelab and Self-Hosting

Blogs

PiMyLifeUp
Pat’s Blog
noted.lol
homelabrat

Repositories

awesome-selfhosted
tteck/Proxmox
topics/homelab

Subreddits

r/selfhosted
r/homelab
r/homeautomation

Youtube Channels

NetworkChuck
TechnoTim
RaidOwl
Hardware Haven
Wolfgang’s Channel
Learn Linux TV
Lawrence Systems
AwesomeOpenSource

Resources

G Suite legacy free edition
Has anyone who bought a lifetime license for earlier version (5) gotten free update to v.1? Killed by Google
Sony Removing Some Purchased Films, Movies from Users’ Accounts
What is a Raspberry Pi?
Raspberry Pi OS
I Can Save You Money! – Raspberry Pi Alternatives
Intel NUC Mini PCs
Mini PCs – Powered by AMD Ryzen™ Embedded Processors
GTR
SER
SR
Why choose NAS drives over desktop drives for your NAS?
Power supply ratings explained: 80 Plus Platinum vs. Gold vs. Bronze vs. White
OuterVision® Power Supply Calculator
How to Choose a PC Case: 5 Features to Consider
Why you really DON’T want to self-host your own e-mail server
LastPass Hack: Engineer’s Failure to Update Plex Software Led to Massive Data Breach