AI Hardware & Software Stack
Which describes the idea of speculative decoding for speeding up LLM inference?
- AGuessing answers randomly and outputting them directly
- BA small model drafts several tokens ahead, the large model verifies them together in one pass, and the correct ones are accepted at once
- CUsing the large model with half its layers removed
Show answer and explanation
Answer: B. A small model drafts several tokens ahead, the large model verifies them together in one pass, and the correct ones are accepted at once
Decode is memory-bound, so the large model spends nearly the same time reading weights whether it computes one token or several. So a light, small model predicts the next few tokens, the large model checks them all in a single forward pass, and the correct ones are accepted at once. If the predictions are often right, the large model is invoked fewer times, speeding up output without changing its quality.
- A. Drafts are always verified by the large model.
- B. Correct. The output can match the large model alone.
- C. The large model is used intact.