English
Menu

Information Theory and Coding Basics

Two random variables X (value 0 or 1) and Y (value a, b or c) have the joint probabilities P(0, a) = P(0, b) = P(1, b) = P(1, c) = 1/4, and every other combination has probability 0. Which of the following is the mutual information I(X;Y)?

  1. A0.5 bits
  2. B1 bit
  3. C1.5 bits
Show answer and explanation

Answer: A. 0.5 bits

X is 0 or 1 with probability 1/2 each, so H(X) = 1 bit; Y is a with 1/4, b with 1/2 and c with 1/4, so H(Y) = 1.5 bits. The four pairs each have probability 1/4, so H(X, Y) = 2 bits, and I(X;Y) = H(X) + H(Y) − H(X, Y) = 1 + 1.5 − 2 = 0.5 bits. If Y is a or c the value of X is known, and if it is b it is not, so Y gives part of the information about X.

  • A. Correct. 1 + 1.5 − 2 = 0.5 bits.
  • B. 1 bit is H(Y|X) = H(X, Y) − H(X) = 2 − 1, the uncertainty about Y that remains once X is known.
  • C. 1.5 bits is H(Y), the entropy of Y itself. It is not the part shared between X and Y.
Practice this topic Back to topic

Books on Information Theory and Coding Basics Ad

As an Amazon Associate I earn from qualifying purchases.

More Information Theory and Coding Basics questions

  1. The 36 characters made up of the 26 capital letters and the 10 digits are to be distinguished by bit strings all of the same length (a fixed-length code). Which of the following is the minimum number of bits needed per character?
  2. Which of the following correctly describes a variable-length code compared with a fixed-length code?
  3. Which of the following sets of codewords assigned to four symbols is a prefix code (a code in which no codeword is the beginning of another codeword)?
  4. A prefix code assigns the codewords A = 0, B = 10, C = 110 and D = 111 to the symbols A, B, C and D. Which of the following is the result of decoding the bit string 110100111 from the start?
  5. A binary prefix code is to be built by assigning codewords of the following lengths (numbers of bits) to the four symbols A to D. Which of the following sets of code lengths makes this possible?