What is Batch size?
The number of inputs processed together. Larger batches reuse weights for efficiency but increase latency and memory.
Field: AI Hardware & Software Stack
Batch size in the explanations
With batch size B, multiplying the weight matrix by B vectors becomes a product of the weight matrix and a B-wide matrix.
Why does increasing the batch size (the number of inputs pr…Memory for intermediate results and the KV cache also grows with batch size.
What is a drawback of making the inference batch size too l…It cuts computation dramatically, but the amount of stored keys and values grows in proportion to sequence length and batch size.
What is the role of the KV cache in LLM inference?
Questions that mention Batch size (7)
- Why does a fully connected layer (a weight-matrix-by-vector product) tend to be memory-bound with a single input (batch size 1)?
- Why does increasing the batch size (the number of inputs processed together) tend to improve inference efficiency?
- What is a drawback of making the inference batch size too large?
- A model with about 14 GB of FP16 weights decodes at batch size 1 on a GPU with 2 TB/s memory bandwidth. What is closest to the lower bound on time per generated token?
- What is the role of the KV cache in LLM inference?
- When LLM inference handles long contexts or large batches, what tends to become a problem?
- Which describes gradient checkpointing (activation recomputation) used in training?
Related terms
KV cache Arithmetic intensity Throughput GPU Gradient checkpointing Backpropagation AI compiler AI inference All-reduce Attention mechanism