What is Speculative decoding?

Speeding up inference by having a small model draft several tokens ahead and a large model verify them together, accepting the correct ones at once.

Field: AI Hardware & Software Stack

Questions that mention Speculative decoding (1)

  1. Which describes the idea of speculative decoding for speeding up LLM inference?

Related terms

AI compiler AI inference All-reduce Arithmetic intensity Attention mechanism Backpropagation Batch size Computational graph Data parallelism FlashAttention

Back to the glossary