AI Hardware & Software Stack

On an accelerator with 100 TFLOPS peak and 2 TB/s memory bandwidth, a workload with arithmetic intensity 10 FLOP/byte runs. What performance does the Roofline model predict?

  1. A100 TFLOPS (compute-bound)
  2. B20 TFLOPS (memory-bound)
  3. C10 TFLOPS (memory-bound)
Show answer and explanation

Answer: B. 20 TFLOPS (memory-bound)

In the Roofline model, attainable performance is the smaller of the peak and bandwidth × arithmetic intensity. 2 TB/s × 10 FLOP/byte = 20 TFLOPS, below the 100 peak, so this workload is memory-bound with a 20 TFLOPS ceiling. Peak ÷ bandwidth = 50 FLOP/byte is the boundary (the ridge point); only workloads with higher intensity can reach peak performance.

  • A. With intensity below 50, peak is out of reach.
  • B. Correct. 2 TB/s × 10 = 20 TFLOPS.
  • C. That assumes 1 TB/s of bandwidth.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. Why does a fully connected layer (a weight-matrix-by-vector product) tend to be memory-bound with a single input (batch size 1)?
  2. Why does increasing the batch size (the number of inputs processed together) tend to improve inference efficiency?
  3. What is a drawback of making the inference batch size too large?
  4. LLM inference splits into a stage that processes the input prompt together (prefill) and a stage that generates output one token at a time (decode). With small batches, what mainly limits each stage?
  5. What is the role of the KV cache in LLM inference?