English
Menu

What is Regularization?

A method of reducing overfitting by adding to the loss function a penalty term that depends on the size of the weights, keeping the weights from growing too large. L1 regularization, which adds the sum of absolute weight values, tends to drive some weights to exactly zero, while L2 regularization (weight decay), which adds the sum of squared weights, shrinks the weights overall. The penalty strength is a hyperparameter chosen by cross-validation or similar means. In a broader sense, dropout, early stopping and other techniques for improving generalization are also collectively called regularization.

Japanese name: 正則化

Regularization appears in 2 practice questions across 1 topic, including Machine Learning Basics.

Field: Machine Learning Basics

Try it: questions about Regularization

Which of the following correctly describes regularization and related methods for reducing overfitting?

  1. AL1 regularization adds the sum of the absolute values of the weights to the loss, and tends to make some weights exactly zero
  2. BL2 regularization adds the sum of the absolute values of the weights to the loss, and strongly narrows down features by setting unneeded weights to zero
  3. CDropout randomly disables neurons every time, in both training and inference, to give the output some variability
Show answer and explanation

Answer: A. L1 regularization adds the sum of the absolute values of the weights to the loss, and tends to make some weights exactly zero

Regularization adds a penalty depending on the size of the weights to the loss function, keeping the weights from growing too large and the model simple. L1 regularization adds the sum of the absolute values of the weights and tends to make some weights exactly zero, so it narrows down the features used (lasso regression). L2 regularization adds the sum of the squared weights and shrinks the weights overall, but they rarely become exactly zero (ridge regression, weight decay). Dropout randomly disables neurons during training only, so that the network does not rely too much on particular neurons; at inference all neurons are used.

Regularization in the explanations

Questions that mention Regularization (2)

  1. Which of the following correctly describes regularization and related methods for reducing overfitting?
  2. Which of the following is seen when overfitting is occurring?

Books on Machine Learning Basics Ad

As an Amazon Associate I earn from qualifying purchases.

Related terms

Overfitting Activation function Confusion matrix Convolutional neural network (CNN) Cross-validation Gradient descent k-means clustering Precision and recall Principal component analysis (PCA) Recurrent neural network (RNN)

Back to the glossary