AI Hardware & Software Stack
When LLM inference handles long contexts or large batches, what tends to become a problem?
- AThe number of weights grows with context length
- BKV cache memory grows with context length and batch size and can even exceed the weights
- CMore tokens always make the model's answers shorter
Show answer and explanation
Answer: B. KV cache memory grows with context length and batch size and can even exceed the weights
KV cache size is the number of layers × key and value dimensions × number of tokens × batch size × bytes per element. With contexts of tens of thousands of tokens or many simultaneous requests, it reaches tens of GB and can exceed weight memory. Techniques include storing the KV cache in lower precision, sharing keys and values across attention heads and managing memory in small blocks.
- A. The number of weights is fixed.
- B. Correct. Techniques such as KV cache compression and quantization are used.
- C. It has nothing to do with answer length.