English
Menu

Information Theory and Coding Basics

A source that emits 8 kinds of symbols has an entropy of 2.4 bits. Defining redundancy as 1 − (entropy ÷ maximum entropy), which of the following is the redundancy of this source?

  1. A20%
  2. B25%
  3. C80%
Show answer and explanation

Answer: A. 20%

The maximum entropy for 8 kinds of symbols is log₂ 8 = 3 bits, so the redundancy is 1 − 2.4 ÷ 3 = 1 − 0.8 = 0.2, that is, 20%. Redundancy expresses how much is wasted, because of skewed symbol probabilities and the like, compared with the maximum amount of information. The more redundant a source, the more room there is to shrink it with lossless compression.

  • A. Correct. 1 − 2.4 ÷ 3 = 0.2.
  • B. 25% is (3 − 2.4) ÷ 2.4, which divides the difference by the entropy instead of by the maximum entropy.
  • C. 80% is 2.4 ÷ 3, the ratio of the entropy to the maximum entropy. Redundancy is 1 minus this ratio.
Practice this topic Back to topic

Books on Information Theory and Coding Basics Ad

As an Amazon Associate I earn from qualifying purchases.

More Information Theory and Coding Basics questions

  1. For two random variables X and Y, which of the following correctly describes the conditional entropy H(X|Y) and the mutual information I(X;Y)?
  2. Two random variables X (value 0 or 1) and Y (value a, b or c) have the joint probabilities P(0, a) = P(0, b) = P(1, b) = P(1, c) = 1/4, and every other combination has probability 0. Which of the following is the mutual information I(X;Y)?
  3. The 36 characters made up of the 26 capital letters and the 10 digits are to be distinguished by bit strings all of the same length (a fixed-length code). Which of the following is the minimum number of bits needed per character?
  4. Which of the following correctly describes a variable-length code compared with a fixed-length code?
  5. Which of the following sets of codewords assigned to four symbols is a prefix code (a code in which no codeword is the beginning of another codeword)?