MI355X FP4: MTP vs Off Speculative Decoding

MTP posts 5202 tok/s/chip for $0.08 per million tokens at 21 tok/s/user on DeepSeekv4 Pro 0813 1.6T (MI355X FP4); Off posts 4487 tok/s/chip for $0.09. MTP is 16% cheaper per token; MTP delivers 16% more tok/s/chip. Draft-token acceptance rates determine whether speculative decoding helps or hurts at a given concurrency level.

Throughput at 39 tok/s/user on DeepSeekv4 Pro 0813 1.6T (MI355X FP4): MTP hits 3789 tok/s/chip, Off hits 1330. Per-million costs land at $0.11 and $0.31 respectively. MTP is 185% cheaper per token; MTP delivers 185% more tok/s/chip. Speculative decoding trades extra compute on draft tokens for fewer decoding steps — the payoff depends on sequence length and batch size.

Toward the upper edge of the 4–75 tok/s/user interactivity band, at 57 tok/s/user on DeepSeekv4 Pro 0813 1.6T (MI355X FP4): MTP runs 2198 tok/s/chip at $0.19/M tokens, Off runs 593 at $0.70/M. MTP is 270% cheaper per token; MTP delivers 270% more tok/s/chip. Gains from speculative decoding vary by workload; short-output prompts tend to benefit less. (Numbers reflect this URL's pinned 8k/1k · fp4 workload — changing sequence or model updates both the table and chart; the table stays pinned to this page's precision, so precision toggles in the controls affect the chart only.)