AI Hardware & Software Stack

For a Transformer with N parameters trained on D tokens, which approximation is commonly used for the number of training operations?

  1. AAbout 2ND
  2. BAbout N + D
  3. CAbout 6ND
Show answer and explanation

Answer: C. About 6ND

A forward pass for one token needs a multiply and an add per parameter, about 2N operations. Backpropagation takes about twice that, 4N, so training costs about 6N per token and about 6ND for D tokens. Training a 7-billion-parameter model on 1 trillion tokens, for example, takes about 4.2 × 10²² operations. Dividing this by the GPUs' effective performance gives a rough training time.

  • A. 2ND is the forward pass (inference) alone.
  • B. Operations scale multiplicatively.
  • C. Correct. 2ND forward plus 4ND backward.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. For an LLM with N parameters, roughly how many operations does generating one token take in inference (excluding attention)?
  2. When computing softmax exp(xi)/Σexp(xj), why is the maximum often subtracted from each xi first?
  3. Which describes the idea of speculative decoding for speeding up LLM inference?
  4. Why are lookups in the large embedding tables used by recommendation models and similar hard for AI hardware?
  5. Which orders the AI software stack from the layer closest to users down to the layer closest to hardware?