LLM inference splits into a stage that processes the input prompt together (prefill) and a stage that generates output one token at a time (decode). With small batches, what mainly limits each stage?

Answer: B. Prefill is limited by compute, decode by memory bandwidth

In prefill, hundreds or thousands of input tokens are computed together, so weights are reused across many tokens, the work becomes matrix-matrix products and it is compute-bound. In decode, every new token requires reading all the weights for just one token's worth of computation, so intensity is low and it is memory-bound. Decode speed is therefore roughly memory bandwidth divided by model size.

• A. It is the other way around.
• B. Correct. Decode reads all weights for every token.
• C. Weights sit in memory during inference.