RunCards #1: Six models, and how to run them right

Hi, and welcome to the first issue. RunCards is one page per open model covering the quant to grab, the template fix you'll need, the sampling settings, and which runtimes actually work. Every card was checked recently, and anything we couldn't confirm is marked unverified.

Notable models this round

Nemotron 3.5 Bolt 74B-A9B (24GB) (NVIDIA, Oct 15)
A vision-language MoE with 9B active parameters and experts shipped at native 2-bit, so the official GGUF is 22.0 GB.
Run it right: Update llama.cpp, since support landed in PR #30520 on Oct 15. Expect a 24 GB card to be tight once you add context, because NVIDIA's own recipes target 28-32 GB+.

Qwen3.8-27B (Alibaba Qwen)
The newest dense Qwen, with reasoning_effort control.
Run it right: The official template defaults to xhigh reasoning, which can burn your token budget or return empty content. Set --chat-template-kwargs '{"reasoning_effort":"medium"}' or use froggeric's fixed templates (v22.x).

Qwen3.8-Flash-Next (125B-A6B preview) (Alibaba Qwen)
An experimental preview of the Qwen4 architecture with 262K native context. It thinks by default.
Run it right: Unsloth UD-Q4_K_XL (111.3 GB) fits 128 GB unified-memory machines only with small context. For direct answers, pass {"enable_thinking": false}.

Nemotron 3.5 Lightning 30B-A3B (NVIDIA, Aug 11)
A hybrid Mamba-2 + MoE model with 3B active and up to 1M context.
Run it right: On vLLM, NVIDIA changed the reasoning parser from nemotron_v3 to nano_v3. Use nano_v3 with --mamba-ssm-cache-dtype float32.

GLM-5.3-Flash (320B-A18B) (Z.ai, Aug 25)
The first natively multimodal GLM-5, MIT licensed.
Run it right: reasoning_effort defaults to max if you leave it out. Pass it explicitly for shorter answers. The Unsloth UD-Q2_K_XL / IQ2_XXS / IQ1_S quants (93-109 GB) are the only sizes near 128 GB.

KAT-Coder-V2.5-Dev (35B-A3B) (Kwaipilot, Jul 23)
An Apache-2.0 coding MoE with 262K context.
Run it right: On vLLM, --language-model-only is required because no vision weights ship. Add --reasoning-parser qwen3 --tool-call-parser qwen3_coder. On llama.cpp, bartowski's Q4_K_M is 21.4 GB.

What fits your box?

The hardware fit page covers 11 common setups, from 8 GB GPUs up to DGX Spark and Strix Halo, and shows which models and quant sizes fit each one. Check it before you start a 100 GB download.

Gotcha of the week: Laguna XS 2.1 on a Mac

Laguna XS 2.1 (poolside, 33B-A3B) is pitched as running on a 36 GB Mac. On Apple Silicon, though, Ollama and llama.cpp can return empty output because of a Metal f16 overflow, which stays until llama.cpp PR #25389 ships in your build. The fix: use mlx-lm, which poolside recommends.

Bonus Laguna gotcha: on vLLM, use a build with PR #47311, or the poolside_v1 tool parser will silently drop some tool calls.

Know a fix we missed? Reply or write to [email protected].

See you next issue,
RunCards