English
Menu

Information Theory and Coding Basics

A source emits the symbols A, B, C and D independently with probabilities 1/2, 1/4, 1/8 and 1/8. Which of the following is the entropy (average amount of information per symbol) of this source?

  1. A1.75 bits
  2. B2 bits
  3. C2.25 bits
Show answer and explanation

Answer: A. 1.75 bits

Entropy is the average of the amount of information −log₂ p of each symbol weighted by its probability: H = Σ(−p log₂ p). The symbols carry 1, 2, 3 and 3 bits of information, so H = 1/2 × 1 + 1/4 × 2 + 1/8 × 3 + 1/8 × 3 = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 bits. If the four symbols were equally likely it would be 2 bits, but a skewed distribution lowers it accordingly.

  • A. Correct. H = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 bits.
  • B. 2 bits is log₂ 4, the value (the maximum) when the four symbols are equally likely.
  • C. 2.25 bits is the plain average of the amounts of information 1, 2, 3 and 3, without weighting them by probability.
Practice this topic Back to topic

Books on Information Theory and Coding Basics Ad

As an Amazon Associate I earn from qualifying purchases.

More Information Theory and Coding Basics questions

  1. Which of the following correctly pairs the maximum entropy of a source that emits 8 kinds of symbols with the condition under which it is reached?
  2. A binary source emits 0 with probability 0.9 and 1 with probability 0.1, independently. Take log₂ 0.9 ≈ −0.152 and log₂ 0.1 ≈ −3.32. Which of the following is closest to the entropy of this source?
  3. A source that emits 8 kinds of symbols has an entropy of 2.4 bits. Defining redundancy as 1 − (entropy ÷ maximum entropy), which of the following is the redundancy of this source?
  4. For two random variables X and Y, which of the following correctly describes the conditional entropy H(X|Y) and the mutual information I(X;Y)?
  5. Two random variables X (value 0 or 1) and Y (value a, b or c) have the joint probabilities P(0, a) = P(0, b) = P(1, b) = P(1, c) = 1/4, and every other combination has probability 0. Which of the following is the mutual information I(X;Y)?