What is Prefill?
The first stage of LLM inference, processing all input prompt tokens together; it tends to be compute-bound. The following token-by-token generation is called decode.
Field: AI Hardware & Software Stack
Prefill in the explanations
In prefill, hundreds or thousands of input tokens are computed together, so weights are reused across many tokens, the work becomes matrix-matrix products and it is compute-bound.
LLM inference splits into a stage that processes the input …For compute-bound prefill the gain is small, and quantization that performs the computation itself in low precision helps more.
In LLM inference, why does weight-only 4-bit quantization (…
Questions that mention Prefill (2)
- LLM inference splits into a stage that processes the input prompt together (prefill) and a stage that generates output one token at a time (decode). With small batches, what mainly limits each stage?
- In LLM inference, why does weight-only 4-bit quantization (W4A16, activations kept in FP16) especially speed up decode?
Related terms
AI compiler AI inference All-reduce Arithmetic intensity Attention mechanism Backpropagation Batch size Computational graph Data parallelism FlashAttention