AI Hardware & Software Stack
Which describes data parallelism in multi-GPU training?
- AEach GPU holds a full model replica, computes on its share of the data, and gradients are averaged across GPUs before updating weights
- BModel layers are split so each GPU handles different layers
- COne GPU computes while the others stand by
Show answer and explanation
Answer: A. Each GPU holds a full model replica, computes on its share of the data, and gradients are averaged across GPUs before updating weights
In data parallelism, every GPU holds the same model, the training batch is divided among them, and each performs forward and backward passes. Their gradients differ, so an all-reduce communication sums and averages gradients across GPUs, and everyone updates weights with the same gradient, keeping the replicas identical. It is simple and effective when the model fits on one GPU, but communication grows heavier as GPUs are added.
- A. Correct. It is the most basic parallel approach.
- B. That is pipeline parallelism.
- C. That is not parallel at all.