What is Pipeline parallelism?

Splitting a model's layers across GPUs in sequence, streaming micro-batches to reduce the time GPUs sit idle (the bubble).

Field: AI Hardware & Software Stack

Pipeline parallelism in the explanations

Combined with data and pipeline parallelism, it is used to train enormous models.
Which describes tensor parallelism, which splits a large la…

Questions that mention Pipeline parallelism (2)

In pipeline parallelism, where a model's layers are split across GPUs in sequence, why is a batch divided into micro-batches?
Which describes tensor parallelism, which splits a large layer's matrix multiplication too big for one GPU across several GPUs?

Related terms

GPU Tensor parallelism AI compiler AI inference All-reduce Arithmetic intensity Attention mechanism Backpropagation Batch size Computational graph

Back to the glossary