English
Menu

What is Hypothesis testing?

A statistical method for judging, on the basis of data, whether to reject a hypothesis such as "there is no difference". The hypothesis to be refuted is set up as the null hypothesis, and the probability of obtaining a result equal to or more extreme than the one observed, assuming it is true (the p-value), is calculated. If the p-value is smaller than a significance level decided in advance (such as 5%), the null hypothesis is rejected. There are type I errors, rejecting a true null hypothesis, and type II errors, failing to reject a false one.

Japanese name: 仮説検定

Hypothesis testing appears in 2 practice questions across 1 topic, including Probability, Statistics and Queueing.

Field: Probability, Statistics and Queueing

Steps of a test

First, a null hypothesis such as "the yield does not change under the new conditions" and an alternative hypothesis, "it changes", are set up. Next, the significance level α (such as 5% or 1%) is decided before looking at the data. The test statistic and p-value are calculated from the data; if the p-value is smaller than α, the null hypothesis is rejected and the difference is judged "significant", and otherwise it is not rejected.

Two kinds of error

A type I error is judging that there is a difference when there really is none (a false positive), and its probability is the significance level α. A type II error is missing a difference that really exists (a false negative); if its probability is β, the probability 1 − β of correctly detecting the difference is called the power. For the same sample size, making α smaller makes β larger, so the sample must be enlarged to make both small.

Cautions in reading the results

Failing to reject the null hypothesis does not prove that "there is no difference"; the sample may simply have been too small to give enough power. Conversely, with a very large sample even a difference too small to matter in practice becomes significant, so the size of the difference should be examined as well. Note also that repeating many tests makes it more likely that some will be significant by chance.

Try it: questions about Hypothesis testing

Which of the following correctly describes the meaning of the p-value obtained in a hypothesis test?

  1. AThe probability, calculated from the observed data, that the null hypothesis itself is true
  2. BIf the p-value is 0.03, the probability that the alternative hypothesis is true is 97%
  3. CThe probability, assuming the null hypothesis is true, of obtaining the observed result or a more extreme one
Show answer and explanation

Answer: C. The probability, assuming the null hypothesis is true, of obtaining the observed result or a more extreme one

A hypothesis test sets up a null hypothesis such as "there is no difference" and uses the p-value to express how unlikely a result like the actual data would be if it were true. If the p-value is smaller than a significance level decided in advance (for example 0.05), the null hypothesis is rejected and a difference is judged to exist. The p-value is not the probability that the null or the alternative hypothesis itself is true.

Whether a product lot meets its specification is tested using a sample taken from it. If the null hypothesis is "this lot meets the specification", which of the following is a type I error?

  1. AJudging a lot that meets the specification as not meeting it and rejecting it
  2. BJudging a lot that does not meet the specification as meeting it and accepting it
  3. CThe instrument measuring the samples is broken, so the measured values themselves are wrong
Show answer and explanation

Answer: A. Judging a lot that meets the specification as not meeting it and rejecting it

A type I error is rejecting the null hypothesis when it is true, here rejecting a good lot (producer's risk). Its probability is the significance level α, decided in advance as 5% or similar. A type II error is failing to reject the null hypothesis when it is false, here accepting a bad lot (consumer's risk). For the same sample size, making α smaller makes type II errors more likely.

FAQ

If the p-value is larger than 0.05, can we say there is no difference?

No. It only means that "these data did not provide enough evidence to judge that there is a difference". To examine a difference, plan the sample size so that the necessary power is obtained, or show the range of the size of the difference with a confidence interval.

How is the significance level decided?

5% is often used, but it is not a rule. If wrongly concluding that there is a difference would be costly, a smaller level such as 1% is used; if missing a difference would be costly, more weight is put on power. In either case, the level is decided before the test, not chosen after looking at the data to suit the result.

Hypothesis testing in the explanations

Questions that mention Hypothesis testing (2)

  1. Which of the following correctly describes the meaning of the p-value obtained in a hypothesis test?
  2. Whether a product lot meets its specification is tested using a sample taken from it. If the null hypothesis is "this lot meets the specification", which of the following is a type I error?

Books on Probability, Statistics and Queueing Ad

As an Amazon Associate I earn from qualifying purchases.

Related terms

Bayes' theorem Binomial distribution Conditional probability Confidence interval Correlation coefficient Expected value Exponential distribution Median Normal distribution Permutations and combinations

Back to the glossary