GPU Configuration

GPU acceleration significantly improves inference speed for large models, and Menatic AI supports NVIDIA GPUs via the NVIDIA GPU Operator for Kubernetes.

Prerequisites

1. Nodes with NVIDIA GPUs
2. NVIDIA drivers installed on GPU nodes
3. NVIDIA GPU Operator deployed in the cluster

Enabling GPU for a Model

When adding or editing a local model:

1. Open the model configuration
2. Enable GPU Acceleration
3. Specify the number of GPUs to allocate (default: 1)

The platform adds a nvidia.com/gpu resource request to the Ollama pod, which causes Kubernetes to schedule it on a GPU-capable node.

Verifying GPU Detection

The dashboard’s GPU card shows detected GPU resources across all nodes. If your GPUs are not appearing:

# Check if GPU operator is running
kubectl get pods -n gpu-operator

# Check node GPU labels
kubectl describe node <node-name> | grep nvidia

# Verify GPU resource is reported
kubectl get nodes -o json | jq '.items[].status.capacity | select(."nvidia.com/gpu")'

GPU Memory Requirements

Model Size	Recommended VRAM
3B parameters	4 GB
7B parameters	8 GB
13B parameters	16 GB
30B parameters	24 GB
70B parameters	48–80 GB

Quantized models (q4, q5) require roughly half the VRAM of full precision (fp16) models.

Multi-GPU

For models larger than a single GPU’s VRAM, Ollama supports tensor parallelism across multiple GPUs. Set the GPU count to the number of GPUs to use; Ollama handles the distribution automatically.

CPU Fallback

If no GPU is available or GPU allocation fails, Ollama falls back to CPU inference automatically. CPU inference is functional but significantly slower for large models.