Local Models (Ollama)

Local models use Ollama to run inference entirely within your Kubernetes cluster, with no external API calls.

Deploying a Local Model

1. Navigate to AI Models in the sidebar
2. Click Add Model
3. Select Local (Ollama)
4. Enter the model name (e.g., llama3.2, mistral, qwen2.5:7b)
5. Click Deploy

The platform creates an Ollama StatefulSet in the apps namespace and pulls the model. Pull time depends on model size and your network speed — a 7B model is typically 4–8 GB.

Supported Models

Any model published in the Ollama library is supported. Popular choices:

Model	Size	Use Case
llama3.2	2B / 3B	Fast, general purpose
llama3.1:8b	8B	Balanced quality/speed
mistral	7B	Instruction following
qwen2.5:7b	7B	Multilingual, strong reasoning
deepseek-coder-v2	16B	Code generation
llama3.1:70b	70B	High quality (requires GPU)

Model Tags

Specify a tag to pin a specific version:

llama3.1:8b-instruct-q4_K_M

Quantized variants (q4, q5, q8) use less memory at some quality cost.

Checking Model Status

Model deployment status is shown in the AI Models table. A Running badge means the model is ready to accept inference requests. Starting means the pod is initializing or the model is being pulled.

Deleting a Model

Click the delete icon in the model row. This removes the Ollama deployment and stops billing for any cloud GPU resources.

Caution

Deleting a local model removes its deployment but does not delete the persistent volume. Model weights remain on disk until you manually delete the PVC.