AI Hardware & Software Stack

Which is NOT an advantage of running AI inference on the device itself (at the edge), such as a smartphone, rather than on cloud servers?

  1. AAny model, however large, can run without limits on the device
  2. BResponses are fast without waiting for the network and work offline
  3. CData stays on the device, protecting privacy
Show answer and explanation

Answer: A. Any model, however large, can run without limits on the device

Edge inference avoids network delay, works without connectivity, keeps photos, audio and other data on the device and avoids server costs. Devices, however, have tight limits on memory, battery and heat, so large models cannot run as is. Quantization, distillation into smaller models and power-efficient dedicated circuits such as NPUs make them run within these constraints.

  • A. Correct. Devices are tightly constrained in memory and power.
  • B. This is an edge advantage.
  • C. This is an edge advantage.
Practice this topic Back to topic

More AI Hardware & Software Stack questions

  1. Why are the NPUs in smartphone SoCs more power-efficient than CPUs for AI inference?
  2. What should be kept in mind about AI accelerator performance figures quoted as "XX TOPS"?
  3. Which describes MFU (Model FLOPs Utilization), a metric of large-scale training efficiency?
  4. For a Transformer with N parameters trained on D tokens, which approximation is commonly used for the number of training operations?
  5. For an LLM with N parameters, roughly how many operations does generating one token take in inference (excluding attention)?