RTX PRO 6000 FP4: MTP vs Off Speculative Decoding

MTP posts 1372 tok/s/chip for $0.14 per million tokens at 29 tok/s/user on Qwen 3.5 397B-A17B (RTX PRO 6000 FP4); Off posts 974 tok/s/chip for $0.19. MTP is 41% cheaper per token; MTP delivers 41% more tok/s/chip. Draft-token acceptance rates determine whether speculative decoding helps or hurts at a given concurrency level.

Throughput at 46 tok/s/user on Qwen 3.5 397B-A17B (RTX PRO 6000 FP4): MTP hits 1082 tok/s/chip, Off hits 579. Per-million costs land at $0.17 and $0.33 respectively. MTP is 87% cheaper per token; MTP delivers 87% more tok/s/chip. Speculative decoding trades extra compute on draft tokens for fewer decoding steps — the payoff depends on sequence length and batch size.

Toward the upper edge of the 13–79 tok/s/user interactivity band, at 63 tok/s/user on Qwen 3.5 397B-A17B (RTX PRO 6000 FP4): MTP runs 916 tok/s/chip at $0.21/M tokens, Off runs 316 at $0.60/M. MTP is 190% cheaper per token; MTP delivers 190% more tok/s/chip. Gains from speculative decoding vary by workload; short-output prompts tend to benefit less. (Numbers reflect this URL's pinned 8k/1k · fp4 workload — changing sequence or model updates both the table and chart; the table stays pinned to this page's precision, so precision toggles in the controls affect the chart only.)