AI Hardware & Software Stack

What is the role of the KV cache in LLM inference?

  1. AIt stores frequent questions and answers to reply instantly to repeats
  2. BIt places all model weights in the CPU cache
  3. CIt stores the attention keys and values of already processed tokens so they need not be recomputed for each new token
Show answer and explanation

Answer: C. It stores the attention keys and values of already processed tokens so they need not be recomputed for each new token

In attention, each new token refers to the keys and values of all earlier tokens. Those keys and values do not change, so storing them instead of recomputing means only the new token's share must be computed. That store is the KV cache. It cuts computation dramatically, but the amount of stored keys and values grows in proportion to sequence length and batch size.

  • A. It does not store answers.
  • B. It is unrelated to CPU caches.
  • C. Correct. It trades memory for a large reduction in computation.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. When LLM inference handles long contexts or large batches, what tends to become a problem?
  2. A model with about 14 GB of FP16 weights decodes at batch size 1 on a GPU with 2 TB/s memory bandwidth. What is closest to the lower bound on time per generated token?
  3. How does the computation of Transformer self-attention grow with input sequence length n?
  4. What is the main idea behind fast attention implementations such as FlashAttention?
  5. What is the main purpose of operator fusion in AI computation?