Local GPU Inference Setup — CUDA, Nemotron & VRAM Requirements

Why Local Inference?

The trade-off: local models under 14B parameters are noticeably less capable than Claude Sonnet or GPT-4.1 for complex reasoning. For simple automation tasks (heartbeats, triage, summaries) they're excellent. For complex multi-step reasoning, route to a cloud model as a fallback.

Step 1 — Check GPU Compatibility

Local inference with NemoClaw requires an NVIDIA GPU with CUDA compute capability 7.0 or higher (Volta architecture or newer). Here's the VRAM requirement by use case:

Check your GPU's compute capability:

nvidia-smi --query-gpu=name,compute_cap,memory.total --format=csv
# Example output:
# NVIDIA GeForce RTX 4090, 8.9, 24564 MiB

Compute capability 8.9 (RTX 4090) is well above the 7.0 minimum. If nvidia-smi isn't found, proceed to Step 2.

Step 2 — Install NVIDIA Drivers

On Ubuntu 22.04 or 24.04 (the most common VPS and workstation OS for this):

# Add the NVIDIA driver PPA
sudo add-apt-repository ppa:graphics-drivers/ppa
sudo apt update

# Install the recommended driver (545 or 550 as of 2026)
sudo apt install nvidia-driver-550

# Or let Ubuntu pick the right driver:
sudo ubuntu-drivers autoinstall

# Reboot required after driver install
sudo reboot

After reboot, verify:

nvidia-smi
# Should show driver version, CUDA version, and your GPU name
# Example:
# Driver Version: 550.54.15   CUDA Version: 12.4

Step 3 — Install CUDA Toolkit

Download from developer.nvidia.com/cuda-downloads. Select Linux → x86_64 → Ubuntu → 22.04 → deb (network):

wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb
sudo dpkg -i cuda-keyring_1.1-1_all.deb
sudo apt update
sudo apt install cuda-toolkit-12-4

# Add CUDA to PATH
echo 'export PATH=/usr/local/cuda-12.4/bin:$PATH' >> ~/.bashrc
echo 'export LD_LIBRARY_PATH=/usr/local/cuda-12.4/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
source ~/.bashrc

# Verify
nvcc --version
# Should show: Cuda compilation tools, release 12.4

Step 4 — Install Ollama with GPU Support

Ollama is the easiest way to run open models with GPU acceleration. It auto-detects CUDA at install time:

curl -fsSL https://ollama.com/install.sh | sh

# Verify GPU is detected
ollama run llama3.2:3b "Hello" 2>&1 | head -5
# Look for "Using GPU" in the startup output

Confirm GPU usage directly:

# In one terminal, start a model
ollama run qwen2.5:7b "Describe yourself"

# In another terminal, watch GPU utilisation
watch -n 0.5 nvidia-smi
# GPU-Util should jump to 80-100% during inference

Recommended Models by VRAM

# 8 GB VRAM — fast, lightweight
ollama pull llama3.2:3b
ollama pull qwen2.5:7b          # needs 4-bit quant at 8 GB

# 16 GB VRAM — good quality
ollama pull qwen2.5:14b
ollama pull mistral-small:22b   # 4-bit fits 16 GB

# 24 GB VRAM — near-API quality
ollama pull qwen2.5:32b
ollama pull qwen2.5:72b         # 4-bit fits 24 GB

Step 5 — Nemotron via NVIDIA NIM

For NVIDIA's Nemotron models specifically, NVIDIA Inference Microservices (NIM) provides optimised containers with better performance than raw Ollama. NIM requires a free NVIDIA developer account:

# Authenticate with NGC (NVIDIA GPU Cloud)
docker login nvcr.io
# Username: $oauthtoken
# Password: <your NGC API key from build.nvidia.com>

# Pull and run Nemotron Mini (4.1B — fits in 8 GB VRAM)
docker run --gpus all -p 8000:8000 \
  nvcr.io/nvidia/nemotron-mini-4b-instruct:latest

# Or Nemotron-4-Mini-Instruct (8B — needs 16 GB)
docker run --gpus all -p 8000:8000 \
  nvcr.io/nvidia/nemotron-4-mini-instruct:latest

NIM serves an OpenAI-compatible API on port 8000. This means you can point NemoClaw at it using the OpenAI provider type with a local URL:

# Add NIM as a provider in OpenShell
openShell provider add \
  --name nemotron-local \
  --type openai-compatible \
  --base-url http://localhost:8000/v1 \
  --model nemotron-mini-4b-instruct

# Route inference through this provider
openShell inference route set --provider nemotron-local

Step 6 — Configure NemoClaw for Local Inference

Once Ollama or NIM is running on the host, configure NemoClaw to use it. Remember: this config lives inside the sandbox (after claw connect <sandbox-name>):

# Step 1: Connect to the sandbox
claw connect nemoclaw

# Step 2: Add the local policy rule so the sandbox can reach localhost
# (exit sandbox first, add rule, reload, re-enter)
exit
cat >> ~/.openShell/policies/includes/local-inference.yaml << 'EOF'
allow:
  - host: "localhost"
    ports: [11434]   # Ollama default port
    comment: "Local Ollama inference"
  - host: "127.0.0.1"
    ports: [11434, 8000]  # Ollama + NIM
    comment: "Local inference endpoints"
EOF
openShell policy reload

# Step 3: Re-enter sandbox and configure OpenClaw
claw connect nemoclaw

# Step 4: Add Ollama as a provider inside the sandbox config
openclaw config set agents.defaults.model.primary "ollama/qwen2.5:14b"
openclaw config set agents.defaults.models '{"ollama/qwen2.5:14b":{"alias":"Local Qwen 14B"},"anthropic/claude-haiku-4-5":{"alias":"Haiku (cloud fallback)"}}'

# Step 5: Restart the gateway
openclaw gateway restart

Test it:

# Inside the sandbox
openclaw run "What model are you running on?"
# Should respond mentioning qwen or the local model name

Performance Expectations

Numbers are approximate and vary by system RAM bandwidth, power mode, and temperature throttling.

Troubleshooting