AI Hardware & Software Stack

Which describes MFU (Model FLOPs Utilization), a metric of large-scale training efficiency?

  1. AThe percentage of hardware peak performance achieved, computed from the operations the model actually requires
  2. BThe fraction of model weights that are non-zero
  3. CThe fraction of GPU memory in use
Show answer and explanation

Answer: A. The percentage of hardware peak performance achieved, computed from the operations the model actually requires

MFU derives the rate of useful computation from tokens processed per second and the operations the model requires, then divides it by the GPUs' peak performance. Communication waits, memory-bound operations, pipeline bubbles and recomputation all lower it. Because it shows how well the hardware is used, it guides improvements to training configuration and software.

  • A. Correct. In large-scale training, 40-60% is considered good.
  • B. It is unrelated to sparsity.
  • C. It is not memory utilization.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. For a Transformer with N parameters trained on D tokens, which approximation is commonly used for the number of training operations?
  2. For an LLM with N parameters, roughly how many operations does generating one token take in inference (excluding attention)?
  3. When computing softmax exp(xi)/Σexp(xj), why is the maximum often subtracted from each xi first?
  4. Which describes the idea of speculative decoding for speeding up LLM inference?
  5. Why are lookups in the large embedding tables used by recommendation models and similar hard for AI hardware?