What is All-reduce?
Communication that sums values held by all GPUs element by element and gives the result to every GPU, used to average gradients in data-parallel training.
Field: AI Hardware & Software Stack
All-reduce in the explanations
All-reduce sums identically shaped tensors held by each GPU element by element (the reduce) and gives the result to every GPU.
What does the all-reduce communication used in distributed …Their gradients differ, so an all-reduce communication sums and averages gradients across GPUs, and everyone updates weights with the same gradient, keeping the replicas identical.
Which describes data parallelism in multi-GPU training?
Questions that mention All-reduce (2)
Related terms
GPU Data parallelism AI compiler AI inference Arithmetic intensity Attention mechanism Backpropagation Batch size Computational graph FlashAttention