AI Hardware & Software Stack
Why are lookups in the large embedding tables used by recommendation models and similar hard for AI hardware?
- ATable lookups require enormous numbers of multiplications
- BEmbedding tables are small and always fit on chip
- CThere is almost no computation, and scattered reads from a huge table leave matrix units unused, so speed depends on memory capacity, bandwidth and latency
Show answer and explanation
Answer: C. There is almost no computation, and scattered reads from a huge table leave matrix units unused, so speed depends on memory capacity, bandwidth and latency
In recommendation systems, tables holding a vector per user or item ID can reach hundreds of GB, and each input reads and sums scattered rows. There is almost no computation, and the irregular reads defeat coalescing and caching, very different from the matrix-product work accelerators excel at. Tables are sharded across the memory of many GPUs or CPUs to secure enough capacity and bandwidth.
- A. Lookups are mostly reads with few multiplications.
- B. They can reach hundreds of GB and do not fit on chip.
- C. Correct. The memory system, not the arithmetic units, is key.