Information Theory and Coding Basics
Which of the following correctly describes a property of the amount of information (−log₂ p on being told that an event of probability p has occurred)?
- AThe amount of information on being told that an event of probability 1, which always occurs, has occurred is 1 bit
- BThe amount of information on being told that two independent events have both occurred is the sum of their individual amounts of information
- CThe more probable an event, the more information we receive on being told that it has occurred
Show answer and explanation
Answer: B. The amount of information on being told that two independent events have both occurred is the sum of their individual amounts of information
Two independent events both occur with probability p₁ × p₂, so the amount of information is −log₂(p₁ × p₂) = (−log₂ p₁) + (−log₂ p₂), the sum of the individual amounts. Information is defined with a logarithm precisely to give it this additive property. Being told that a certain event has occurred tells us nothing new, so its amount of information is −log₂ 1 = 0.
- A. The amount of information of a certain event is −log₂ 1 = 0 bits. 1 bit is the amount of information of an event of probability 1/2.
- B. Correct. Taking the logarithm turns the product of probabilities into a sum.
- C. It is the other way round. The less likely an event, the more information we receive on being told that it has occurred.
Continue: 9 more Information Theory and Coding Basics questions