English
Menu

Information Theory and Coding Basics

Which of the following correctly pairs the maximum entropy of a source that emits 8 kinds of symbols with the condition under which it is reached?

  1. A3 bits; when one of the symbols has probability 1
  2. B3 bits; when all 8 symbols have probability 1/8
  3. C8 bits; when all 8 symbols have probability 1/8
Show answer and explanation

Answer: B. 3 bits; when all 8 symbols have probability 1/8

The entropy of a source emitting n kinds of symbols is greatest when every symbol occurs with the same probability 1/n, and its value is then log₂ n. For n = 8 this is log₂ 8 = 3 bits. The average information per symbol is greatest when the next symbol is hardest to predict. Conversely, when one symbol occurs with probability 1, we know what will come, so the entropy is 0.

  • A. When one symbol occurs with probability 1, the entropy is at its minimum of 0 bits.
  • B. Correct. With equal probabilities it reaches the maximum of log₂ 8 = 3 bits.
  • C. 8 is the number of kinds of symbols. The maximum entropy is its logarithm, log₂ 8 = 3 bits.
Practice this topic Back to topic

Books on Information Theory and Coding Basics Ad

As an Amazon Associate I earn from qualifying purchases.

More Information Theory and Coding Basics questions

  1. A binary source emits 0 with probability 0.9 and 1 with probability 0.1, independently. Take log₂ 0.9 ≈ −0.152 and log₂ 0.1 ≈ −3.32. Which of the following is closest to the entropy of this source?
  2. A source that emits 8 kinds of symbols has an entropy of 2.4 bits. Defining redundancy as 1 − (entropy ÷ maximum entropy), which of the following is the redundancy of this source?
  3. For two random variables X and Y, which of the following correctly describes the conditional entropy H(X|Y) and the mutual information I(X;Y)?
  4. Two random variables X (value 0 or 1) and Y (value a, b or c) have the joint probabilities P(0, a) = P(0, b) = P(1, b) = P(1, c) = 1/4, and every other combination has probability 0. Which of the following is the mutual information I(X;Y)?
  5. The 36 characters made up of the 26 capital letters and the 10 digits are to be distinguished by bit strings all of the same length (a fixed-length code). Which of the following is the minimum number of bits needed per character?