AI Hardware & Software Stack

Why does a fully connected layer (a weight-matrix-by-vector product) tend to be memory-bound with a single input (batch size 1)?

  1. AThe vector has too many elements to fit in the arithmetic units
  2. BGPUs cannot compute matrix-vector products
  3. CEach weight read from memory is used in only one multiply-add, so arithmetic intensity is very low
Show answer and explanation

Answer: C. Each weight read from memory is used in only one multiply-add, so arithmetic intensity is very low

In a matrix-vector product, each element of the weight matrix is read once and used for a single multiply-add with one input element. In FP16 that is two operations per 2 bytes, an arithmetic intensity of about 1. Reading the weights takes most of the time while the arithmetic units mostly wait. Grouping inputs so the same weights serve several of them (a larger batch) raises arithmetic intensity.

  • A. Element count is not the reason.
  • B. It can be computed, but memory waits dominate.
  • C. Correct. Weights cannot be reused.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. Why does increasing the batch size (the number of inputs processed together) tend to improve inference efficiency?
  2. What is a drawback of making the inference batch size too large?
  3. LLM inference splits into a stage that processes the input prompt together (prefill) and a stage that generates output one token at a time (decode). With small batches, what mainly limits each stage?
  4. What is the role of the KV cache in LLM inference?
  5. When LLM inference handles long contexts or large batches, what tends to become a problem?