English
Menu

What is Information entropy?

The average amount of information per symbol emitted by a source. With pᵢ the probability of each symbol, it is H = −Σ pᵢ log₂ pᵢ (bits), and it expresses how hard it is to predict what comes next. It reaches its maximum log₂ n when all n kinds of symbols are equally likely, and is 0 when one symbol always occurs. It gives the lower bound on the average number of bits needed per symbol in lossless coding. Shannon introduced it in 1948, and it was named after thermodynamic entropy because the formulas look alike. Also called Shannon entropy.

Japanese name: 情報エントロピー

Information entropy appears in 11 practice questions across 2 topics, including Information Theory and Coding Basics, Machine Learning Basics.

Field: Information Theory and Coding Basics

Worked examples

A coin with heads and tails at 1/2 each has 1 bit, and a die whose faces 1 to 6 are equally likely has log₂ 6 ≈ 2.58 bits. A biased coin with heads at 0.9 and tails at 0.1 has about 0.47 bits, smaller because the result is almost predictable. A symbol of probability 0 is treated as contributing 0 × log₂ 0 = 0.

The limit of coding

For a source that emits symbols independently, no lossless code can make the average code length per symbol smaller than the entropy (the source coding theorem). The average code length of a Huffman code is at least the entropy and less than the entropy + 1 bit, and coding symbols together brings the length per symbol arbitrarily close to the entropy.

Related quantities

For two random variables we consider the joint entropy H(X, Y), the conditional entropy H(X|Y), which expresses the uncertainty about X that remains once Y is known, and their difference, the mutual information I(X;Y) = H(X) − H(X|Y). The cross-entropy used in machine learning measures the difference between the true distribution and the predicted one, and it becomes smaller the better the prediction.

Try it: questions about Information entropy

A source emits the symbols A, B, C and D independently with probabilities 1/2, 1/4, 1/8 and 1/8. Which of the following is the entropy (average amount of information per symbol) of this source?

  1. A1.75 bits
  2. B2 bits
  3. C2.25 bits
Show answer and explanation

Answer: A. 1.75 bits

Entropy is the average of the amount of information −log₂ p of each symbol weighted by its probability: H = Σ(−p log₂ p). The symbols carry 1, 2, 3 and 3 bits of information, so H = 1/2 × 1 + 1/4 × 2 + 1/8 × 3 + 1/8 × 3 = 0.5 + 0.5 + 0.375 + 0.375 = 1.75 bits. If the four symbols were equally likely it would be 2 bits, but a skewed distribution lowers it accordingly.

Which of the following correctly pairs the maximum entropy of a source that emits 8 kinds of symbols with the condition under which it is reached?

  1. A3 bits; when one of the symbols has probability 1
  2. B3 bits; when all 8 symbols have probability 1/8
  3. C8 bits; when all 8 symbols have probability 1/8
Show answer and explanation

Answer: B. 3 bits; when all 8 symbols have probability 1/8

The entropy of a source emitting n kinds of symbols is greatest when every symbol occurs with the same probability 1/n, and its value is then log₂ n. For n = 8 this is log₂ 8 = 3 bits. The average information per symbol is greatest when the next symbol is hardest to predict. Conversely, when one symbol occurs with probability 1, we know what will come, so the entropy is 0.

FAQ

Is a source with higher entropy a better source?

It is not a matter of better or worse but of how unpredictable the source is. Data with high entropy are hard to compress, and for things that must not be predicted, such as cryptographic keys and random numbers, high entropy is required. Conversely, highly redundant data such as text shrink well when compressed.

Is it the same as entropy in physics?

The formulas look very much alike, and it is closely related to entropy in statistical mechanics, but entropy in physics is tied to heat and temperature and has different units. Information theory treats entropy as a quantity determined by the probability distribution alone.

Information entropy in the explanations

Questions that mention Information entropy (11)

  1. A source emits the symbols A, B, C and D independently with probabilities 1/2, 1/4, 1/8 and 1/8. Which of the following is the entropy (average amount of information per symbol) of this source?
  2. Which of the following correctly pairs the maximum entropy of a source that emits 8 kinds of symbols with the condition under which it is reached?
  3. A binary source emits 0 with probability 0.9 and 1 with probability 0.1, independently. Take log₂ 0.9 ≈ −0.152 and log₂ 0.1 ≈ −3.32. Which of the following is closest to the entropy of this source?
  4. A source that emits 8 kinds of symbols has an entropy of 2.4 bits. Defining redundancy as 1 − (entropy ÷ maximum entropy), which of the following is the redundancy of this source?
  5. For two random variables X and Y, which of the following correctly describes the conditional entropy H(X|Y) and the mutual information I(X;Y)?
  6. A source emits the symbols A, B and C independently with probabilities 0.6, 0.3 and 0.1, and its entropy is about 1.30 bits. Which of the following is closest to the efficiency (entropy ÷ average code length) when this source is encoded with a Huffman code?
  7. Which of the following correctly describes the relationship between lossless compression and the entropy of a source?
  8. In a binary symmetric channel, 0s and 1s are sent and each bit flips independently with probability 0.1. With the binary entropy function H(p) = −p log₂ p − (1 − p) log₂(1 − p), H(0.1) ≈ 0.469. Which of the following is closest to the channel capacity of this channel per transmission?
  9. Let p be the probability that a model outputs for the correct class. Which of the following describes the cross-entropy loss −log p used to train classifiers?
  10. A binary source emits 0 with probability 0.8 and 1 with probability 0.2, independently. Symbols are taken two at a time, each of the four pairs 00, 01, 10 and 11 is treated as one symbol, and they are encoded with a Huffman code. Which of the following is the average code length per original symbol?
  11. A node of a decision tree holds 6 data points of class A and 4 of class B. What is the Gini impurity of this node?

Books on Information Theory and Coding Basics Ad

As an Amazon Associate I earn from qualifying purchases.

Related terms

Self-information Average code length Lossless compression Mutual information Huffman coding Binary symmetric channel Shannon–Hartley theorem Forward error correction (FEC) Hamming code Hamming distance

Back to the glossary