AI Hardware & Software Stack

For an LLM with N parameters, roughly how many operations does generating one token take in inference (excluding attention)?

  1. AAbout 2N
  2. BAbout 6N
  3. CAbout N²
Show answer and explanation

Answer: A. About 2N

The forward pass multiplies each weight by an input and adds the result once, about 2N operations per token: about 14 billion for a 7-billion-parameter model. That is small for a modern GPU, but decode must read 14 GB of weights in the meantime, so memory bandwidth, not computation, determines the time.

  • A. Correct. One multiply and one add per parameter.
  • B. 6N is per token in training.
  • C. It does not scale with N squared.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. When computing softmax exp(xi)/Σexp(xj), why is the maximum often subtracted from each xi first?
  2. Which describes the idea of speculative decoding for speeding up LLM inference?
  3. Why are lookups in the large embedding tables used by recommendation models and similar hard for AI hardware?
  4. Which orders the AI software stack from the layer closest to users down to the layer closest to hardware?
  5. What is the difference between training and inference for a neural network?