What is Tensor parallelism?
Splitting one layer's weight matrix by rows or columns across GPUs, computing partial products and combining them through communication.
Field: AI Hardware & Software Stack
Tensor parallelism in the explanations
In tensor parallelism, the weight matrix might be split column-wise into four parts held by four GPUs, each computing the product of the same input with its own columns.
Which describes tensor parallelism, which splits a large la…
Questions that mention Tensor parallelism (1)
Related terms
GPU Pipeline parallelism AI compiler AI inference All-reduce Arithmetic intensity Attention mechanism Backpropagation Batch size Computational graph