Local GPU Inference with NemoClaw — Ollama, vLLM, llama.cpp & NIM

Your options, and how well each is tested

GPU-backed vLLM and NIM need an NVIDIA GPU, the NVIDIA Container Toolkit and a healthy CDI spec. AMD/ROCm, Intel Arc and Apple Metal aren't supported for those paths.

Step 1: check the host

nvidia-smi                      # driver loaded, GPU and memory visible
nemoclaw host probe             # NemoClaw's own readiness check

If nvidia-smi isn't found, install your distribution's recommended NVIDIA driver (on Ubuntu, sudo ubuntu-drivers autoinstall) and reboot. For the Ollama path, that's all you need; Ollama ships its own CUDA libraries. For vLLM or NIM, also install the NVIDIA Container Toolkit and generate its CDI spec.

Step 2: onboard with Ollama

nemoclaw onboard     # choose the Ollama entry

• Ollama 0.32.9 or newer is required. Older versions can return tool calls as plain text, which breaks the agent. If yours is older, the menu shows an Upgrade Ollama entry: Homebrew on macOS, the official install script on Linux.
• Without sudo, the wizard installs Ollama under ~/.local instead of system-wide. Nothing then restarts it after a reboot; run ~/.local/bin/ollama serve yourself or add a service.
• Model choice by GPU memory. Each entry shows download size and approximate VRAM, and the wizard picks the largest model that fits your free memory:

NemoClaw pulls the model, loads it and tests a real tool call before continuing. If the model returns tool calls as text, onboarding stops rather than giving you an agent that can't use tools.

Context length: Hermes sandboxes need at least 64,000 tokens, and NemoClaw starts Ollama with that when it manages the daemon. If you run Ollama yourself with a smaller context, the wizard tells you the OLLAMA_CONTEXT_LENGTH to restart it with. For OpenClaw, a stock daemon below 16,384 tokens is raised to that in the agent config.

# scripted / unattended
NEMOCLAW_PROVIDER=ollama NEMOCLAW_MODEL=qwen3.5:9b \
  nemoclaw onboard --non-interactive --yes --yes-i-accept-third-party-software

Using an existing vLLM or llama.cpp server

• vLLM: a server on localhost:8000 (or $NEMOCLAW_VLLM_PORT) appears in the provider menu. NemoClaw uses its Chat Completions endpoint, because vLLM's Responses endpoint doesn't run the tool-call parser. Start vLLM with tool calling enabled for your model.
• llama.cpp (experimental): choose it explicitly. NemoClaw attaches only to an authenticated server on loopback port 8081 that passes its fingerprint check, and doesn't manage that server's lifecycle.
• Any other server (SGLang, TRT-LLM, a self-hosted NIM, LiteLLM): use the compatible endpoint route.

Inside the sandbox, the agent still talks to inference.local, and OpenShell forwards to your local server. To reach a host server from the sandbox policy side, use the local-inference preset.

Freeing GPU memory

Stopping or destroying an Ollama-backed sandbox tells Ollama to unload that sandbox's model. It stays loaded if another running sandbox uses it. Downloaded model files are kept. Changing a sandbox to a different model with nemoclaw onboard unloads the old one.

Should you go local?

• Yes for privacy-sensitive work, heavy background automation such as triage and summaries, or when API rate limits are the bottleneck.
• Maybe not for hard multi-step reasoning. A 9B–35B local model is well behind Claude Sonnet 5 or GPT-6 Sol on long tool chains. Many people run local by default and switch to a hosted model for hard jobs.
• Remember the shared route: all sandboxes on one OpenShell gateway use the same model. To run local and hosted models side by side, use separate gateways.

Official references: NemoClaw docs Local Inference and Platform Support. Why a 27B model can beat a much larger one on your tasks: small vs large models.