English
Menu

What is Overfitting?

A state in which a model memorizes fine details and chance noise in the training data, so that its error on the training data is small but its error on new data (generalization error) is large. It tends to occur when a complex model with many parameters is trained on little data. It is reduced by adding data, simplifying the model, regularization, dropout, early stopping and similar methods.

Japanese name: 過学習

Overfitting appears in 3 practice questions across 1 topic, including Machine Learning Basics.

Field: Machine Learning Basics

How to recognize it

Overfitting is recognized by comparing performance with a validation set kept separate from the training data. A learning curve, which plots the training and validation losses against the number of training steps, shows the training loss continuing to fall while the validation loss starts to rise from some point. A widening gap of this kind is overfitting. Conversely, when both losses stay large and do not fall, the model is underfitting: it is too simple, or otherwise unable to capture the trend.

Causes

When the model has too much freedom (number of parameters, depth of a decision tree and so on) compared with the amount of data, it can fit not only the real trend but also the noise that happens to be in that data. For example, a 9th-degree polynomial fitted to 10 points with distinct x values can pass exactly through every point, but it swings widely between the points and predicts new points badly. Tuning hyperparameters again and again while looking at the same data also inflates apparent performance and causes overfitting.

How to reduce it

The most reliable remedy is more training data. When data cannot be added, data augmentation, such as rotating or flipping images, increases the apparent amount of data. On the model side, methods include simplifying the model by reducing parameters, regularization that adds a penalty on weight size to the loss, dropout that randomly disables neurons during training, and early stopping that ends training when the validation loss starts to rise. Ensemble learning, which averages the predictions of several models, also reduces the variability of predictions.

Try it: questions about Overfitting

Which of the following is seen when overfitting is occurring?

  1. AThe error on the training data is small, but the error on the validation data is large
  2. BThe error stays large and does not fall on either the training data or the validation data
  3. CAs training proceeds, the errors on the training data and on the validation data both keep falling in the same way
Show answer and explanation

Answer: A. The error on the training data is small, but the error on the validation data is large

Overfitting is a state in which the model memorizes fine details and noise in the training data, so its performance on new data (generalization performance) drops. A small error on the training data together with a large error on the validation data is the telltale sign. A large error even on the training data indicates underfitting, in which the model fails to capture the trend, for example because it is too simple. Remedies include adding data, simplifying the model, regularization, dropout, and early stopping, which ends training when the validation error starts to rise.

Which of the following correctly describes regularization and related methods for reducing overfitting?

  1. AL1 regularization adds the sum of the absolute values of the weights to the loss, and tends to make some weights exactly zero
  2. BL2 regularization adds the sum of the absolute values of the weights to the loss, and strongly narrows down features by setting unneeded weights to zero
  3. CDropout randomly disables neurons every time, in both training and inference, to give the output some variability
Show answer and explanation

Answer: A. L1 regularization adds the sum of the absolute values of the weights to the loss, and tends to make some weights exactly zero

Regularization adds a penalty depending on the size of the weights to the loss function, keeping the weights from growing too large and the model simple. L1 regularization adds the sum of the absolute values of the weights and tends to make some weights exactly zero, so it narrows down the features used (lasso regression). L2 regularization adds the sum of the squared weights and shrinks the weights overall, but they rarely become exactly zero (ridge regression, weight decay). Dropout randomly disables neurons during training only, so that the network does not rely too much on particular neurons; at inference all neurons are used.

FAQ

How do overfitting and underfitting differ?

Overfitting is fitting the training data so closely that the error on new data becomes large, while the error on the training data is small. Underfitting is a state in which the error is large even on the training data, because the model is too simple or has not been trained enough. In terms of bias and variance, underfitting corresponds to high bias and overfitting to high variance.

If accuracy on the training data is 100%, is the model overfitting?

That alone does not decide it. If the model also performs well enough on validation and test data, there is no problem. Suspect overfitting when there is a large gap between performance on the training data and on the validation data.

Overfitting in the explanations

Questions that mention Overfitting (3)

  1. Which of the following is seen when overfitting is occurring?
  2. Which of the following correctly describes regularization and related methods for reducing overfitting?
  3. Which of the following is correct when a model's error is considered in terms of bias and variance?

Books on Machine Learning Basics Ad

As an Amazon Associate I earn from qualifying purchases.

Semiconductor, Electronics and Computing Calculators

Confusion matrix calculatorCompute accuracy, precision, recall, F1, specificity, MCC and other metrics from TP, FN, FP and TN

Related terms

Regularization Cross-validation Activation function Confusion matrix Convolutional neural network (CNN) Gradient descent k-means clustering Precision and recall Principal component analysis (PCA) Recurrent neural network (RNN)

Back to the glossary