English
Menu

Information Theory and Coding Basics

For two random variables X and Y, which of the following correctly describes the conditional entropy H(X|Y) and the mutual information I(X;Y)?

  1. AKnowing the value of Y can make H(X|Y), the uncertainty about X, larger than H(X) on average
  2. BThe mutual information I(X;Y) is greatest when X and Y are independent, and its value is then H(X) + H(Y)
  3. CI(X;Y) = H(X) − H(X|Y) is the reduction in uncertainty about X from knowing Y, and is 0 if they are independent
Show answer and explanation

Answer: C. I(X;Y) = H(X) − H(X|Y) is the reduction in uncertainty about X from knowing Y, and is 0 if they are independent

H(X|Y) is the average uncertainty about X that remains after the value of Y is known; knowing Y never increases the average uncertainty, so H(X|Y) ≤ H(X). The reduction is the mutual information I(X;Y) = H(X) − H(X|Y) = H(X) + H(Y) − H(X, Y), and if X and Y are independent it is 0, because Y tells us nothing about X. In communication, with X the symbol sent and Y the symbol received, I(X;Y) is the amount of information actually conveyed per transmission.

  • A. It does not increase on average. H(X|Y) ≤ H(X), with equality when X and Y are independent.
  • B. When they are independent, Y tells us nothing about X, so the mutual information is at its minimum of 0.
  • C. Correct. The reduction in uncertainty from knowing Y is the information Y gives about X.
Practice this topic Back to topic

Books on Information Theory and Coding Basics Ad

As an Amazon Associate I earn from qualifying purchases.

More Information Theory and Coding Basics questions

  1. Two random variables X (value 0 or 1) and Y (value a, b or c) have the joint probabilities P(0, a) = P(0, b) = P(1, b) = P(1, c) = 1/4, and every other combination has probability 0. Which of the following is the mutual information I(X;Y)?
  2. The 36 characters made up of the 26 capital letters and the 10 digits are to be distinguished by bit strings all of the same length (a fixed-length code). Which of the following is the minimum number of bits needed per character?
  3. Which of the following correctly describes a variable-length code compared with a fixed-length code?
  4. Which of the following sets of codewords assigned to four symbols is a prefix code (a code in which no codeword is the beginning of another codeword)?
  5. A prefix code assigns the codewords A = 0, B = 10, C = 110 and D = 111 to the symbols A, B, C and D. Which of the following is the result of decoding the bit string 110100111 from the start?