AI Hardware & Software Stack
Which describes tensor parallelism, which splits a large layer's matrix multiplication too big for one GPU across several GPUs?
- AEach GPU holds a full model replica and the data is split
- BThe weight matrix is split by rows or columns across GPUs, each computes a partial product, and results are combined through communication
- CEach layer is placed on a different GPU and processed in turn
Show answer and explanation
Answer: B. The weight matrix is split by rows or columns across GPUs, each computes a partial product, and results are combined through communication
In tensor parallelism, the weight matrix might be split column-wise into four parts held by four GPUs, each computing the product of the same input with its own columns. Combining results by concatenation or summation requires communication in every layer, so it is generally used among GPUs in the same server joined by very fast links. Combined with data and pipeline parallelism, it is used to train enormous models.
- A. That is data parallelism.
- B. Correct. It splits computation within a layer.
- C. That is pipeline parallelism.