English
Menu

What is Mutual information?

A quantity expressing how much information two random variables X and Y carry about each other. It is I(X;Y) = H(X) − H(X|Y) = H(X) + H(Y) − H(X, Y), the average reduction in the uncertainty (entropy) of X from knowing Y. It is never negative and is 0 when X and Y are independent. Swapping X and Y gives the same value. With X the input of a channel and Y its output, it is the amount of information conveyed per transmission, and its maximum over the input probability distributions is the channel capacity.

Japanese name: 相互情報量

Mutual information appears in 3 practice questions across 1 topic, including Information Theory and Coding Basics.

Field: Information Theory and Coding Basics

Try it: questions about Mutual information

For two random variables X and Y, which of the following correctly describes the conditional entropy H(X|Y) and the mutual information I(X;Y)?

  1. AKnowing the value of Y can make H(X|Y), the uncertainty about X, larger than H(X) on average
  2. BThe mutual information I(X;Y) is greatest when X and Y are independent, and its value is then H(X) + H(Y)
  3. CI(X;Y) = H(X) − H(X|Y) is the reduction in uncertainty about X from knowing Y, and is 0 if they are independent
Show answer and explanation

Answer: C. I(X;Y) = H(X) − H(X|Y) is the reduction in uncertainty about X from knowing Y, and is 0 if they are independent

H(X|Y) is the average uncertainty about X that remains after the value of Y is known; knowing Y never increases the average uncertainty, so H(X|Y) ≤ H(X). The reduction is the mutual information I(X;Y) = H(X) − H(X|Y) = H(X) + H(Y) − H(X, Y), and if X and Y are independent it is 0, because Y tells us nothing about X. In communication, with X the symbol sent and Y the symbol received, I(X;Y) is the amount of information actually conveyed per transmission.

Two random variables X (value 0 or 1) and Y (value a, b or c) have the joint probabilities P(0, a) = P(0, b) = P(1, b) = P(1, c) = 1/4, and every other combination has probability 0. Which of the following is the mutual information I(X;Y)?

  1. A0.5 bits
  2. B1 bit
  3. C1.5 bits
Show answer and explanation

Answer: A. 0.5 bits

X is 0 or 1 with probability 1/2 each, so H(X) = 1 bit; Y is a with 1/4, b with 1/2 and c with 1/4, so H(Y) = 1.5 bits. The four pairs each have probability 1/4, so H(X, Y) = 2 bits, and I(X;Y) = H(X) + H(Y) − H(X, Y) = 1 + 1.5 − 2 = 0.5 bits. If Y is a or c the value of X is known, and if it is b it is not, so Y gives part of the information about X.

Mutual information in the explanations

Questions that mention Mutual information (3)

  1. For two random variables X and Y, which of the following correctly describes the conditional entropy H(X|Y) and the mutual information I(X;Y)?
  2. Two random variables X (value 0 or 1) and Y (value a, b or c) have the joint probabilities P(0, a) = P(0, b) = P(1, b) = P(1, c) = 1/4, and every other combination has probability 0. Which of the following is the mutual information I(X;Y)?
  3. In a binary symmetric channel, 0s and 1s are sent and each bit flips independently with probability 0.1. With the binary entropy function H(p) = −p log₂ p − (1 − p) log₂(1 − p), H(0.1) ≈ 0.469. Which of the following is closest to the channel capacity of this channel per transmission?

Books on Information Theory and Coding Basics Ad

As an Amazon Associate I earn from qualifying purchases.

Related terms

Information entropy Self-information Binary symmetric channel Shannon–Hartley theorem Average code length Forward error correction (FEC) Hamming code Hamming distance Huffman coding Kraft inequality

Back to the glossary