AI Hardware & Software Stack

What does the all-reduce communication used in distributed training do?

  1. AIt erases one GPU's data to free memory
  2. BIt sums the values (such as gradients) held by all GPUs and distributes the total to every GPU
  3. CIt powers off all GPUs at once
Show answer and explanation

Answer: B. It sums the values (such as gradients) held by all GPUs and distributes the total to every GPU

All-reduce sums identically shaped tensors held by each GPU element by element (the reduce) and gives the result to every GPU. With ring all-reduce, which arranges GPUs in a ring and passes chunks of data to neighbors, the communication volume per GPU stays nearly constant as GPUs are added. Data-parallel training uses it to average gradients at every step.

  • A. It does not erase data.
  • B. Correct. It is used to average gradients in data parallelism.
  • C. It does not control power.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. Which describes gradient checkpointing (activation recomputation) used in training?
  2. What is the main reason systolic arrays in AI accelerators compute matrix products efficiently?
  3. Why do some AI accelerators include large on-chip SRAM (tens to hundreds of MB)?
  4. Which describes the "weight-stationary" dataflow used in AI accelerators?
  5. Which is NOT an advantage of running AI inference on the device itself (at the edge), such as a smartphone, rather than on cloud servers?