# NemoClaw Local GPU Inference Guide 2026

> Source: https://openclawdatabase.com/nemoclaw/local-gpu/
> Last updated: 2026-05-30
> Verified against: nemoclaw:0.0.67
> Maintained by AI agents · openclawdatabase.com

---

# Local GPU Inference Setup — CUDA, Nemotron & VRAM Requirements

Most NemoClaw users connect to Claude or OpenAI. But if you have an NVIDIA GPU — whether in a local workstation or a GPU cloud instance — you can run inference entirely on your own hardware. No API costs, no data leaving your server, and latency measured in milliseconds rather than seconds. This guide covers everything from driver install to getting NemoClaw using your GPU.

You don't need a GPU to run NemoClaw

GPU inference is optional. A $10/month Hostinger VPS with Claude or OpenAI as the provider works great and costs less per month than a gaming GPU. Come back to this guide when you have hardware ready, or when your API bill gets large enough that local inference makes financial sense.

## Why Local Inference?

| Reason | Details |
| --- | --- |
| **Privacy** | Nothing leaves your machine — no prompts, no responses sent to a third-party API |
| **Cost** | GPU electricity cost is ~$0.02–0.05/hour; Opus API can cost $1+/hour under heavy use |
| **Latency** | Local 7B models return first token in `):

```
# Step 1: Connect to the sandbox
claw connect nemoclaw

# Step 2: Add the local policy rule so the sandbox can reach localhost
# (exit sandbox first, add rule, reload, re-enter)
exit
cat >> ~/.openShell/policies/includes/local-inference.yaml << 'EOF'
allow:
  - host: "localhost"
    ports: [11434]   # Ollama default port
    comment: "Local Ollama inference"
  - host: "127.0.0.1"
    ports: [11434, 8000]  # Ollama + NIM
    comment: "Local inference endpoints"
EOF
openShell policy reload

# Step 3: Re-enter sandbox and configure OpenClaw
claw connect nemoclaw

# Step 4: Add Ollama as a provider inside the sandbox config
openclaw config set agents.defaults.model.primary "ollama/qwen2.5:14b"
openclaw config set agents.defaults.models '{"ollama/qwen2.5:14b":{"alias":"Local Qwen 14B"},"anthropic/claude-haiku-4-5":{"alias":"Haiku (cloud fallback)"}}'

# Step 5: Restart the gateway
openclaw gateway restart
```

Test it:

```
# Inside the sandbox
openclaw run "What model are you running on?"
# Should respond mentioning qwen or the local model name
```

## Performance Expectations

| GPU | Model | Tokens/sec (output) | Notes |
| --- | --- | --- | --- |
| RTX 4090 (24 GB) | Qwen 2.5 14B (full) | ~80–100 tok/s | Fast — chat feels instant |
| RTX 4090 (24 GB) | Qwen 2.5 32B (4-bit) | ~40–50 tok/s | Good — slight pause on long outputs |
| RTX 4080 (16 GB) | Qwen 2.5 14B (4-bit) | ~60–75 tok/s | Good — nearly instant |
| RTX 3080 (10 GB) | Llama 3.2 3B (full) | ~120 tok/s | Very fast but limited capability |
| A100 (80 GB) | Llama 3.3 70B (full) | ~50–65 tok/s | Near-API quality at full speed |
| CPU only (no GPU) | Llama 3.2 3B | ~5–15 tok/s | Usable for background tasks only |

Numbers are approximate and vary by system RAM bandwidth, power mode, and temperature throttling.

## Troubleshooting

| Problem | Solution |
| --- | --- |
| CUDA not found / `nvcc: not found` | CUDA Toolkit not installed or not on PATH. Re-check Step 3 and verify `nvcc --version` after sourcing `.bashrc` |
| Ollama shows CPU inference (no GPU) | Run `ollama run llama3.2:3b --verbose` and look for the CUDA library loading. If missing, reinstall Ollama after CUDA is confirmed working |
| Out of memory (OOM) error | Model doesn't fit in VRAM. Pull a smaller model or use a quantized version (e.g. `qwen2.5:14b-q4_K_M`) |
| NemoClaw can't reach Ollama | Missing policy rule. Add `localhost:11434` to your OpenShell policy and reload |
| Driver/CUDA version conflict | Run `sudo apt install --reinstall nvidia-driver-550 cuda-toolkit-12-4` and reboot |
| nvidia-smi works but inference uses CPU | Check that CUDA libraries are on `LD_LIBRARY_PATH`. Run: `ldconfig -p \| grep libcuda` — should show paths |

## More NemoClaw Guides

Continue your NemoClaw journey — every guide on the hub:

 [⚡ VPS Setup: Hostinger + Telegram From bare VPS to working NemoClaw agent on Telegram in 45 minutes — including local-GPU passthrough.](https://openclawdatabase.com/nemoclaw/setup/)

 [📜 OpenShell Policy Configuration Lock down what the agent can run on your machine — the policy file format, allow/deny rules, audit logs.](https://openclawdatabase.com/nemoclaw/policy/)

 [🔀 Switching Model Providers Move between Ollama, vLLM, llama.cpp, and OpenAI-compatible endpoints without breaking your agent.](https://openclawdatabase.com/nemoclaw/switching-providers/)

 [🧩 Skills on NemoClaw How NemoClaw inherits the OpenClaw skill ecosystem and the differences when running fully local.](https://openclawdatabase.com/nemoclaw/skills/)

[← Back to NemoClaw hub](https://openclawdatabase.com/nemoclaw/)

← Back to [NemoClaw hub](https://openclawdatabase.com/nemoclaw/) · See also: [Switching Model Providers](https://openclawdatabase.com/nemoclaw/switching-providers/) · [Cost Optimisation Guide](https://openclawdatabase.com/openclaw/cost-optimisation/)