GPU Programming

A program that launches a very small kernel thousands of times in a row is slow. What is the most suitable improvement?

  1. AReduce the number of launches, for example by fusing several small kernels into one
  2. BInsert delays between kernel launches
  3. CUse one thread per block
Show answer and explanation

Answer: A. Reduce the number of launches, for example by fusing several small kernels into one

Each kernel launch costs a fixed few microseconds of CPU-side processing and GPU setup. If the kernel body is shorter than that, launch overhead dominates. Fusing several operations into one kernel reduces launches and also avoids writing intermediate results to memory and reading them back. Mechanisms that record a sequence of launches and submit them at once, such as CUDA Graphs, are also used.

  • A. Correct. It reduces fixed per-launch overhead.
  • B. Adding delays makes it slower still.
  • C. That throws away the GPU's parallelism.
Practice this topic Back to topic

More GPU Programming questions

  1. Which describes CUDA unified memory?
  2. When a GPU performs element-wise addition of two large arrays (c[i] = a[i] + b[i]), what mainly determines performance?
  3. Why is double-precision (FP64) performance on many consumer gaming GPUs much lower than single-precision (FP32), for example one thirty-second?
  4. What is a characteristic of the tensor cores (matrix units) in recent GPUs?
  5. In GPU mixed-precision computation, why are multiplications often done in a low precision such as FP16 while accumulation is done in FP32?