AI Hardware & Software Stack

What is a drawback of making the inference batch size too large?

  1. AWaiting to gather inputs and the longer processing per batch delay each request's response, and memory use increases
  2. BWeight values change and answers become inaccurate
  3. CArithmetic intensity drops and efficiency worsens
Show answer and explanation

Answer: A. Waiting to gather inputs and the longer processing per batch delay each request's response, and memory use increases

A larger batch raises total throughput, but waiting for other inputs to arrive and the longer computation per batch increase the time until each request returns (latency). Memory for intermediate results and the KV cache also grows with batch size. Interactive applications increase the batch only as far as their response-time targets allow.

  • A. Correct. Throughput must be balanced against response time.
  • B. Weights do not change.
  • C. Arithmetic intensity actually rises.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. LLM inference splits into a stage that processes the input prompt together (prefill) and a stage that generates output one token at a time (decode). With small batches, what mainly limits each stage?
  2. What is the role of the KV cache in LLM inference?
  3. When LLM inference handles long contexts or large batches, what tends to become a problem?
  4. A model with about 14 GB of FP16 weights decodes at batch size 1 on a GPU with 2 TB/s memory bandwidth. What is closest to the lower bound on time per generated token?
  5. How does the computation of Transformer self-attention grow with input sequence length n?