AI Hardware & Software Stack
In pipeline parallelism, where a model's layers are split across GPUs in sequence, why is a batch divided into micro-batches?
- AMicro-batches reduce the number of computations
- BTo reduce the idle time (bubble) in which later GPUs wait for earlier ones, keeping several GPUs busy at once
- CMicro-batches eliminate inter-GPU communication
Show answer and explanation
Answer: B. To reduce the idle time (bubble) in which later GPUs wait for earlier ones, keeping several GPUs busy at once
If a whole batch flows through at once, while the first GPU computes, the second and later GPUs can do nothing but wait. Splitting the batch into micro-batches lets the second GPU compute micro-batch 1 while the first computes micro-batch 2, and so on, overlapping the work. Some waiting (the bubble) remains at the start and end, so the number of micro-batches is made large enough.
- A. The computation count is unchanged.
- B. Correct. It is like an assembly line.
- C. Communication between stages is still needed.