AI Hardware & Software Stack

Why does increasing the batch size (the number of inputs processed together) tend to improve inference efficiency?

  1. AMore inputs reduce the computation per input
  2. BA larger batch reduces the number of weights
  3. CWeights read once serve several inputs, raising arithmetic intensity and keeping the arithmetic units busy
Show answer and explanation

Answer: C. Weights read once serve several inputs, raising arithmetic intensity and keeping the arithmetic units busy

With batch size B, multiplying the weight matrix by B vectors becomes a product of the weight matrix and a B-wide matrix. Weight reads stay the same while operations grow B-fold, so arithmetic intensity rises roughly B-fold and the workload moves from memory-bound toward compute-bound. Server inference uses GPUs efficiently by batching requests from many users together.

  • A. Computation per input does not change.
  • B. The number of weights does not change.
  • C. Correct. Matrix-vector products become matrix-matrix products.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. What is a drawback of making the inference batch size too large?
  2. LLM inference splits into a stage that processes the input prompt together (prefill) and a stage that generates output one token at a time (decode). With small batches, what mainly limits each stage?
  3. What is the role of the KV cache in LLM inference?
  4. When LLM inference handles long contexts or large batches, what tends to become a problem?
  5. A model with about 14 GB of FP16 weights decodes at batch size 1 on a GPU with 2 TB/s memory bandwidth. What is closest to the lower bound on time per generated token?