A binary source emits 0 with probability 0.9 and 1 with probability 0.1, independently. Take log₂ 0.9 ≈ −0.152 and log₂ 0.1 ≈ −3.32. Which of the following is closest to the entropy of this source?

Answer: C. About 0.47 bits

H = −0.9 × log₂ 0.9 − 0.1 × log₂ 0.1 ≈ 0.9 × 0.152 + 0.1 × 3.32 ≈ 0.137 + 0.332 ≈ 0.47 bits. If 0 and 1 were equally likely it would be 1 bit, but here a 0 is almost predictable, so the average information per symbol is less than half of that. A skewed source like this can, with clever coding, be represented with fewer than 1 bit per symbol.

• A. About 0.14 is only the term for 0, −0.9 × log₂ 0.9; the term for 1 has not been added.
• B. About 0.33 is only the term for 1, −0.1 × log₂ 0.1; the term for 0 has not been added.
• C. Correct. Adding the two terms gives about 0.47 bits.