Information Theory and Coding Basics

Which of the following correctly states Shannon's channel coding theorem?

  1. AIf the transmission rate is below the channel capacity, suitable coding can make the error rate arbitrarily small
  2. BWith strong enough error-correcting codes, the error rate can be made arbitrarily small even at rates above the channel capacity
  3. COn a noisy channel, the only way to bring the error rate close to 0 is to bring the transmission rate close to 0 as well
Show answer and explanation

Answer: A. If the transmission rate is below the channel capacity, suitable coding can make the error rate arbitrarily small

The channel coding theorem shows that at any rate below the channel capacity C, the error rate can be made arbitrarily small by using sufficiently long codes, and that this is impossible at rates above C. The theorem shows that such codes exist but does not give a concrete way to construct them. Today, codes whose performance approaches this limit (the Shannon limit), such as LDPC codes and polar codes, are in practical use.

  • A. Correct. Below the channel capacity, codes exist that make errors arbitrarily rare.
  • B. At rates above the channel capacity, no code of any kind can make the error rate small.
  • C. With methods that simply repeat the same bits, errors do decrease only if the rate is lowered, but the theorem shows that errors can be reduced while keeping a fixed rate below the capacity.
Practice this topic Back to topic

Books on Information Theory and Coding Basics Ad

As an Amazon Associate I earn from qualifying purchases.

More Information Theory and Coding Basics questions

  1. To convert a Gray code into an ordinary binary number, the most significant bit is used as it is, and each lower bit is found as the exclusive OR of the binary bit already found one position higher and the Gray code bit in the same position. Which of the following is the 4-bit Gray code 1101 converted into binary?
  2. Let the amount of information obtained on being told that an event of probability p has occurred be −log₂ p bits. Which of the following is the amount of information obtained on being told that an event of probability 1/8 has occurred?
  3. Which of the following correctly describes a property of the amount of information (−log₂ p on being told that an event of probability p has occurred)?
  4. A source emits the symbols A, B, C and D independently with probabilities 1/2, 1/4, 1/8 and 1/8. Which of the following is the entropy (average amount of information per symbol) of this source?
  5. Which of the following correctly pairs the maximum entropy of a source that emits 8 kinds of symbols with the condition under which it is reached?