AI Hardware & Software Stack

Learn about AI hardware and software with multiple-choice questions: inference versus training, quantization, arithmetic intensity and the roofline model, systolic arrays, operator fusion and tiling.

—Your accuracy
0 / 50Questions answered
Practice this topicAll questions in this topic, in random order Weak questionsRetry only the questions you last got wrong

AI Hardware & Software Stack: key points

Most of the computation in neural networks is large matrix multiply-accumulate. AI hardware therefore lines up many multiply-accumulate units and is designed to reuse data as much as possible to reduce traffic to memory. Training requires far more computation and memory than inference, and also needs higher numerical precision.

Performance is not set by the number of compute units alone. Arithmetic intensity, the amount of computation per byte of data, together with the roofline model, shows whether a workload is limited by compute or by memory bandwidth. This topic also covers how AI compilers and inference runtimes unlock hardware performance through quantization, operator fusion and tiling.

All AI Hardware & Software Stack questions (50)

  1. What is the difference between training and inference for a neural network?
  2. Which is NOT a reason training needs much more memory than inference for the same model?
  3. A 7-billion-parameter (7B) model is trained with weights, gradients and Adam's two states all held in FP32 (4 bytes). Excluding activations, roughly how much memory is needed?
  4. If a 7-billion-parameter (7B) model's weights are held in FP16 (2 bytes), roughly how much memory do the weights take?
  5. If the 7B model (about 14 GB in FP16) is quantized to 4-bit integers (INT4), roughly how much memory do the weights take?
  6. How many floating-point operations (counting multiplies and adds separately) does multiplying an M × K matrix by a K × N matrix take, roughly?
  7. For the product of two N × N FP16 (2-byte) matrices, assuming each input is read once and the result written once, roughly what is the arithmetic intensity (operations per byte)?
  8. On an accelerator with 100 TFLOPS peak and 2 TB/s memory bandwidth, a workload with arithmetic intensity 10 FLOP/byte runs. What performance does the Roofline model predict?
  9. Why does a fully connected layer (a weight-matrix-by-vector product) tend to be memory-bound with a single input (batch size 1)?
  10. Why does increasing the batch size (the number of inputs processed together) tend to improve inference efficiency?
  11. What is a drawback of making the inference batch size too large?
  12. LLM inference splits into a stage that processes the input prompt together (prefill) and a stage that generates output one token at a time (decode). With small batches, what mainly limits each stage?
  13. What is the role of the KV cache in LLM inference?
  14. When LLM inference handles long contexts or large batches, what tends to become a problem?
  15. A model with about 14 GB of FP16 weights decodes at batch size 1 on a GPU with 2 TB/s memory bandwidth. What is closest to the lower bound on time per generated token?
  16. How does the computation of Transformer self-attention grow with input sequence length n?
  17. What is the main idea behind fast attention implementations such as FlashAttention?
  18. What is the main purpose of operator fusion in AI computation?
  19. A matrix product followed by a bias add and a ReLU is fused from three separate kernels into one. What mainly decreases?
  20. What do AI compilers such as XLA, TVM or PyTorch's compile feature mainly do?
  21. Which describes the computational graph used by AI frameworks and compilers?
  22. Which describes ONNX?
  23. Why can inference runtimes such as TensorRT or ONNX Runtime run faster than using the training framework directly for inference?
  24. Why do element-wise operations such as ReLU or layer normalization tend to take a lot of time relative to their operation count in AI workloads?
  25. Why are convolutions in CNNs sometimes converted into matrix multiplications, for example with im2col?
  26. What is the difference between post-training quantization (PTQ) and quantization-aware training (QAT)?
  27. In post-training quantization of activations to INT8, what is the purpose of calibration?
  28. In quantizing large language models, a few channels with large outliers can hurt accuracy. What is a common countermeasure?
  29. In LLM inference, why does weight-only 4-bit quantization (W4A16, activations kept in FP16) especially speed up decode?
  30. Which statement about pruning neural networks is correct?
  31. Which describes the 2:4 structured sparsity supported by GPU matrix units?
  32. Which describes knowledge distillation?
  33. Which describes data parallelism in multi-GPU training?
  34. Which describes tensor parallelism, which splits a large layer's matrix multiplication too big for one GPU across several GPUs?
  35. In pipeline parallelism, where a model's layers are split across GPUs in sequence, why is a batch divided into micro-batches?
  36. What does the all-reduce communication used in distributed training do?
  37. Which describes gradient checkpointing (activation recomputation) used in training?
  38. What is the main reason systolic arrays in AI accelerators compute matrix products efficiently?
  39. Why do some AI accelerators include large on-chip SRAM (tens to hundreds of MB)?
  40. Which describes the "weight-stationary" dataflow used in AI accelerators?
  41. Which is NOT an advantage of running AI inference on the device itself (at the edge), such as a smartphone, rather than on cloud servers?
  42. Why are the NPUs in smartphone SoCs more power-efficient than CPUs for AI inference?
  43. What should be kept in mind about AI accelerator performance figures quoted as "XX TOPS"?
  44. Which describes MFU (Model FLOPs Utilization), a metric of large-scale training efficiency?
  45. For a Transformer with N parameters trained on D tokens, which approximation is commonly used for the number of training operations?
  46. For an LLM with N parameters, roughly how many operations does generating one token take in inference (excluding attention)?
  47. When computing softmax exp(xi)/Σexp(xj), why is the maximum often subtracted from each xi first?
  48. Which describes the idea of speculative decoding for speeding up LLM inference?
  49. Why are lookups in the large embedding tables used by recommendation models and similar hard for AI hardware?
  50. Which orders the AI software stack from the layer closest to users down to the layer closest to hardware?

Other topics in Computer systems