English
Menu

What is Bayes' theorem?

The relation that reverses the direction of a conditional probability: P(A | B) = P(B | A) × P(A) ÷ P(B). It is used to update the prior probability P(A) of a cause A into the posterior probability P(A | B) on the basis of an observed result B. It is used for calculating the positive predictive value of a test, for spam filtering (naive Bayes) and for estimating the causes of failures. It also shows the effect of the base rate: when the cause is rare, many positives are false even for an accurate test.

Japanese name: ベイズの定理

Bayes' theorem appears in 2 practice questions across 2 topics, including Probability, Statistics and Queueing, Machine Learning Basics.

Field: Probability, Statistics and Queueing

What the formula means

Bayes' theorem is written P(A | B) = P(B | A) × P(A) ÷ P(B). P(A) is the probability of cause A before seeing the result (the prior probability), P(B | A) is the probability of observing result B when cause A is present (the likelihood), and P(A | B) is the probability of cause A after seeing result B (the posterior probability). The denominator P(B) is the probability of result B with and without the cause combined, calculated as P(B) = P(B | A) × P(A) + P(B | not A) × P(not A).

Working through a test example

Consider a test that judges 99% of defective parts positive and wrongly judges 1% of good parts positive, when 0.1% of parts are defective. Out of 1 million parts, 990 of the 1,000 defective ones test positive, and 9,990 of the 999,000 good ones also test positive. Only 990 ÷ (990 + 9,990) ≈ 9% of the positives are really defective. Replacing probabilities with concrete counts in a table like this gives the right answer even without remembering the formula.

Where it is used

Spam filtering has long used naive Bayes classification, which finds the probability that an email is spam from the probabilities of the words it contains. In equipment fault diagnosis, the probabilities of possible causes are updated from the observed symptoms. Using each posterior probability as the next prior whenever new data arrive improves the estimate step by step.

Try it: questions about Bayes' theorem

In a population of chips, 1% have a latent defect. A screening test judges 95% of the defective chips positive and wrongly judges 4% of the good chips positive. Which of the following is closest to the probability that a chip judged positive is actually defective?

  1. AAbout 4.9%
  2. BAbout 19%
  3. CAbout 95%
Show answer and explanation

Answer: B. About 19%

Considering 10,000 chips, 95 of the 100 defective chips test positive, and 396 of the 9,900 good chips also test positive. Of the 491 positives, only 95 are actually defective, so 95 ÷ 491 ≈ 0.19, or about 19%. Written with Bayes' theorem, P(defective | positive) = 0.01 × 0.95 ÷ (0.01 × 0.95 + 0.99 × 0.04) ≈ 0.19. When defects are rare, many good chips are mixed in with the positives even if the rate of wrongly judging them positive is small. This is called the effect of the base rate (prior probability).

20% of all emails are spam. A certain word appears in 50% of spam emails and in 5% of non-spam emails. Which is closest to the probability that an email containing this word is spam?

  1. A0.10
  2. B0.50
  3. C0.71
Show answer and explanation

Answer: C. 0.71

By Bayes' theorem, P(spam | word) = P(word | spam) × P(spam) ÷ P(word). The numerator is 0.5 × 0.2 = 0.10, and the probability of containing the word is P(word) = 0.5 × 0.2 + 0.05 × 0.8 = 0.14, so the answer is 0.10 ÷ 0.14 ≈ 0.71. Naive Bayes classification performs this calculation by assuming that several words (features) are mutually independent within each class and multiplying their probabilities, and it has long been used for spam detection.

FAQ

Why are many positives false even for an accurate test?

When the cause (a defect or disease) is very rare among the things tested, the number of good items wrongly judged positive exceeds the number of true positives. For example, if good items outnumber defective ones 1,000 to 1, even a 1% error rate gives about ten times as many positives as there are defective items. How far a positive result can be trusted depends not only on the accuracy of the test but also on the original proportion (the base rate).

How does it differ from conditional probability?

A conditional probability is the quantity itself, "the probability that A occurs when B has occurred", while Bayes' theorem is a formula expressing the relationship between P(A | B) and P(B | A), with the direction reversed. It is used when the known accuracy of a test is P(positive | defective) but what you want is P(defective | positive).

Bayes' theorem in the explanations

Questions that mention Bayes' theorem (2)

  1. In a population of chips, 1% have a latent defect. A screening test judges 95% of the defective chips positive and wrongly judges 4% of the good chips positive. Which of the following is closest to the probability that a chip judged positive is actually defective?
  2. 20% of all emails are spam. A certain word appears in 50% of spam emails and in 5% of non-spam emails. Which is closest to the probability that an email containing this word is spam?

Books on Probability, Statistics and Queueing Ad

As an Amazon Associate I earn from qualifying purchases.

Related terms

Binomial distribution Conditional probability Confidence interval Correlation coefficient Expected value Exponential distribution Hypothesis testing Median Normal distribution Permutations and combinations

Back to the glossary