What is KV cache?
Memory in LLM inference holding the attention keys and values of already processed tokens. It avoids recomputation but grows with context length and batch size.
Field: AI Hardware & Software Stack
KV cache in the explanations
That store is the KV cache.
What is the role of the KV cache in LLM inference?Inference also needs memory for intermediate results and, for large language models, the KV cache.
If a 7-billion-parameter (7B) model's weights are held in F…Memory for intermediate results and the KV cache also grows with batch size.
What is a drawback of making the inference batch size too l…
Questions that mention KV cache (4)
- What is the role of the KV cache in LLM inference?
- If a 7-billion-parameter (7B) model's weights are held in FP16 (2 bytes), roughly how much memory do the weights take?
- What is a drawback of making the inference batch size too large?
- When LLM inference handles long contexts or large batches, what tends to become a problem?
Related terms
Batch size Throughput AI compiler AI inference All-reduce Arithmetic intensity Attention mechanism Backpropagation Computational graph Data parallelism