English
Menu

What is Gradient descent?

An optimization method that computes the gradient of the loss function (the partial derivative with respect to each parameter) and repeatedly moves the parameters a little in the direction opposite to the gradient, searching for parameters that make the loss small. The update is written w ← w − η∇L, where η is called the learning rate. Stochastic gradient descent (SGD), which uses gradients computed on part of the data (a mini-batch), and its refinements such as momentum and Adam are widely used to train neural networks.

Japanese name: 勾配降下法

Gradient descent appears in 1 practice question across 1 topic, including Machine Learning Basics.

Field: Machine Learning Basics

The update rule and the learning rate

Viewing the loss L as a function of a weight w, the gradient ∂L/∂w gives the direction and size of the increase in L when w is increased slightly. So w is moved against the gradient: w ← w − η × ∂L/∂w. For example, with L = (w − 3)² and a start at w = 0, the gradient is 2 × (0 − 3) = −6, so with η = 0.1 the weight moves to 0.6. In this example, η = 0.1 shrinks the remaining distance to the minimum at w = 3 by a factor of 0.8 at every update, whereas η = 1.1 overshoots the minimum each time and the distance grows by a factor of 1.2, so the iteration diverges.

Batch, mini-batch and stochastic gradient descent

Computing the exact gradient of the loss over the whole training set before each update is called batch gradient descent, but with a lot of data each update takes a long time. Stochastic gradient descent instead repeats updates using gradients computed on only a part of the data (a mini-batch). The gradient becomes noisy, but many more updates can be made, and the mini-batch can be sized so that a GPU computes it efficiently in parallel. One complete pass through the training data is called an epoch.

Improved methods

Plain SGD makes slow progress on losses shaped like a long, narrow valley, bouncing back and forth across the valley. Momentum carries over the direction of past updates like inertia and damps this oscillation. Adam and RMSProp automatically adjust the step size of each parameter from the history of its gradient magnitudes. Warm-up, which raises the learning rate gradually at the start of training, and schedules that lower the learning rate as training proceeds are also common.

Try it: questions about Gradient descent

In gradient descent, the gradient of the loss ∂L/∂w at weight w = 2.0 is 0.5 and the learning rate is 0.1. What is the weight after one update?

  1. A1.5
  2. B1.95
  3. C2.05
Show answer and explanation

Answer: B. 1.95

Gradient descent moves the weight in the direction that decreases the loss, that is, opposite to the gradient, updating w ← w − η × ∂L/∂w (η is the learning rate). Here w = 2.0 − 0.1 × 0.5 = 1.95. A learning rate that is too large overshoots the valley of the loss and oscillates or diverges, while one that is too small makes training progress very slowly. In practice, stochastic gradient descent (SGD), which repeats updates with gradients computed on part of the data (a mini-batch), and improved versions such as Adam are used.

FAQ

How does gradient descent differ from backpropagation?

Backpropagation is a method for efficiently computing the gradient of the loss with respect to every weight of a neural network. Gradient descent is a method that uses those gradients to update the weights. Training repeats the cycle of computing the loss with a forward pass, computing the gradients with backpropagation and updating the weights with gradient descent.

Does gradient descent find the best solution?

If the loss function is convex (has a single valley), a suitably chosen learning rate brings it close to the minimum. The loss of a neural network is not convex, so what is found is often a local solution, but experience shows that networks with many parameters still often reach performance that is good enough in practice.

Gradient descent in the explanations

Questions that mention Gradient descent (1)

  1. In gradient descent, the gradient of the loss ∂L/∂w at weight w = 2.0 is 0.5 and the learning rate is 0.1. What is the weight after one update?

Books on Machine Learning Basics Ad

As an Amazon Associate I earn from qualifying purchases.

Related terms

Activation function Confusion matrix Convolutional neural network (CNN) Cross-validation k-means clustering Overfitting Precision and recall Principal component analysis (PCA) Recurrent neural network (RNN) Regularization

Back to the glossary