GB300 NVL72 FP8: MTP vs Off Speculative Decoding

MTP posts 10224 tok/s/chip for $0.06 per million tokens at 93 tok/s/user on Qwen 3.5 397B-A17B (GB300 NVL72 FP8); Off posts 2800 tok/s/chip for $0.23. MTP is 265% cheaper per token; MTP delivers 265% more tok/s/chip. Draft-token acceptance rates determine whether speculative decoding helps or hurts at a given concurrency level.

Throughput at 131 tok/s/user on Qwen 3.5 397B-A17B (GB300 NVL72 FP8): MTP hits 5267 tok/s/chip, Off hits 1529. Per-million costs land at $0.12 and $0.42 respectively. MTP is 244% cheaper per token; MTP delivers 244% more tok/s/chip. Speculative decoding trades extra compute on draft tokens for fewer decoding steps — the payoff depends on sequence length and batch size.

Toward the upper edge of the 55–206 tok/s/user interactivity band, at 169 tok/s/user on Qwen 3.5 397B-A17B (GB300 NVL72 FP8): MTP runs 3181 tok/s/chip at $0.20/M tokens, Off runs 728 at $0.88/M. MTP is 337% cheaper per token; MTP delivers 337% more tok/s/chip. Gains from speculative decoding vary by workload; short-output prompts tend to benefit less. (Numbers reflect this URL's pinned 8k/1k · fp8 workload — changing sequence or model updates both the table and chart; the table stays pinned to this page's precision, so precision toggles in the controls affect the chart only.)