Nemotron 3.5 Bolt 74B-A9B (24GB)
Vision-language MoE compressed from Nemotron 3.5 Super VL (120B-A12B) to 74B with 9B active, shipped with experts at native 2-bit so it fits in about 24 GB. 256K context. Built for local agents in llama.cpp.
Recommended quants
- Official 24GB GGUF (22.0 GB, experts IQ2_XXS)
nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUFAdd mmproj file for images. NVIDIA's own recipes target 28 GB / 32 GB+ GPUs, so 24 GB cards are tight once context is added. - LiLiCorr speculative draft
nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-LiLiCorr-GGUFUsed as a spec-decoding draft in NVIDIA's recipe. - Unsloth UD-Q2_K_XL 28.8 / UD-Q4_K_XL 50.2 / Q8_0 78.9 GB
unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUFHigher-precision options from the BF16 weights.
Chat template Works
Reasoning toggled via chat template (enable_thinking=True/False), inherited from Super 3.5 VL.
Fix: Pass enable_thinking=false for direct answers.
Sampling settings
| Text (official) | temp 1.0, top_p 0.95 |
|---|
Runtimes
| vLLM | Unverified | |
|---|---|---|
| llama.cpp | Works | Official target runtime; support added in llama.cpp PR #30520 (Oct 15). Update your build. |
| Ollama | Unverified | Library page nemotron-3.5-bolt exists. |
| LM Studio | Unverified | |
| MLX | Unverified |
Tool calling in agent tools
| Claude Code (local endpoint) | Unverified | |
|---|---|---|
| OpenCode | Unverified | |
| Cline | Unverified | |
| Aider | Unverified |
Known issues
Sources checked
- https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF
- https://huggingface.co/unsloth/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-GGUF
Spotted something wrong or a new fix? Corrections are reviewed before publishing. (Submission form coming soon.)