Nemotron 3.5 Lightning on vLLM: the Mamba flags for a local agent execution model

What the model actually is

• 30 billion parameters, 3 billion active per token — a mixture-of-experts design, so you get the capacity of a large model at the compute cost of a small one.
• Hybrid architecture — interleaved Mamba-2 layers with mixture-of-expert layers and some attention layers on top. This is the detail that changes how you serve it.
• Context window up to 1 million tokens.
• Open MWD 1.1 license — weights, training data and recipes are all open.
• Positioned as an execution-layer model for long-running agents, not a planner.

Serving it on vLLM — the flags that matter

The model is downloaded from Hugging Face and served with vLLM. Four things in the launch command are specific to this model, and each has a stated reason:

Mamba backend: flashinfer

Purpose: selects the kernel library that runs the Mamba-2 layers.

When to use: required for any hybrid Mamba model — this is not a pure transformer, so it needs its own backend. A standard vLLM invocation will not do the right thing here.

Mamba state dtype: float16

Purpose: stores the Mamba state in FP16 instead of FP32, halving the memory that state consumes.

When to use: when you want the memory saving — but pair it with the flag below, because the saving is a precision downgrade.

Mamba stochastic rounding (enabled)

Purpose: the accuracy fix for the FP16 downgrade above. It rounds randomly rather than always in one direction, so quantization error does not accumulate across long sequences.

When to use: whenever you set the Mamba state to FP16, and especially if you are running the long-context or long-horizon agent workloads this model is built for — accumulating rounding error over a long sequence is exactly the failure this prevents. Fahd's note is that Nvidia published this pairing rather than leaving people to discover the drift, which is worth crediting.

Tool-call parser + tool use enabled

Purpose: the rest of the command is the parser for the model's tool-call format plus enabling tool use.

When to use: always, for agent work — an execution-layer model that cannot emit parseable tool calls is not useful in a loop.

The agent test: an unhinted logic bug

The model is wired into Hermes Agent, which already lists it as available. The test target is a full-stack port-management app — FastAPI backend, SQLite database, JS frontend — that assigns arriving vessels to berths. Each berth has a maximum draft (how deep a ship can sit before it hits the bottom), and the clearance check is broken: a vessel drawing 9.8 m can be assigned to a berth rated 9.5 m, which the app should refuse.

The setup is deliberately hard: no hint is given — not how many bugs exist, not where to look. It runs in Hermes' autonomous mode, where Hermes sets a standing goal and a judge model decides after each turn whether the work is actually done.

Result: the model identified the bug in its first few minutes, correctly naming the draft comparison and the plus/minus sign error, then ran tests and checked the database. After a hard refresh the same assignment was correctly rejected. Fahd's assessment is that he has not seen larger models find that bug as quickly.

Two further tests: a single self-contained HTML file simulating a working rotisserie (continuous rotation, changing skin colour, heat shimmer, glow, flames, physics, controls, and a raw/golden/done/burnt state readout) which worked including mouse-drag rotation and speed controls; and a six-language public-address announcement testing for literal-translation failure, which passed.

Gotchas & Caveats

• Do not hand this model architectural problems. Nvidia's own benchmarks show Lightning losing coding and terminal benchmarks to Code Llama 3.6 and Gemma 4 at the same size — Code Llama is clearly ahead. What it wins is instruction-following and long context. Fahd's framing: it is not the model you give a hard architectural problem, it is the model you give a hundred small well-defined tasks and trust not to drift.
• It sits near the bottom of general intelligence indexes, well behind Opus 5 and Kimiko 3. That chart measures one axis; the entire argument for this model is the other one. Nvidia claims roughly 4× the output speed of similar-size models, because only 3B of the 30B parameters fire per token.
• The FP16 memory saving is not free — it is only safe with stochastic rounding on. Setting the dtype without the rounding flag is the mistake to avoid.
• The right question is not whether this beats a frontier model — it does not — but whether it is good enough to do the grunt work inside your agent loop at a fraction of the cost and latency.

Also shipped: NeMo Switchyard

Nvidia released NeMo Switchyard alongside the model, and Nous Research is listed as an integration partner for Hermes. The intended pattern is Lightning doing execution while a frontier model handles planning. Architecturally, the agent no longer talks to a model at all — it talks to Switchyard, which decides which model receives each request. The routing algorithms live in the library; an LLM client underneath makes the actual HTTP call and translates between provider APIs. Fahd flags this as deserving its own video, along with DeepSpark and DeepFlash which also shipped with the release.

Key Takeaways

• Nemotron 3.5 Lightning: 30B MoE / 3B active, hybrid Mamba-2 + MoE + attention, 1M context, fully open weights, data and recipes under Open MWD 1.1.
• Serving it on vLLM requires the flashinfer Mamba backend, an FP16 Mamba state dtype, stochastic rounding to compensate for it, and a tool-call parser with tool use enabled.
• Budget about 75 GB VRAM as shown; reduce the KV cache to bring that down.
• It found an unhinted logic bug in a FastAPI/SQLite app in minutes under Hermes autonomous mode with a judge model gating completion.
• Use it as an execution model under a frontier planner — that split is exactly what NeMo Switchyard exists to route.
• Nvidia's benchmark presentation is unusually honest about where the model loses, which makes the positioning credible.