Machine Learning Basics
In gradient descent, the gradient of the loss ∂L/∂w at weight w = 2.0 is 0.5 and the learning rate is 0.1. What is the weight after one update?
- A1.5
- B1.95
- C2.05
Show answer and explanation
Answer: B. 1.95
Gradient descent moves the weight in the direction that decreases the loss, that is, opposite to the gradient, updating w ← w − η × ∂L/∂w (η is the learning rate). Here w = 2.0 − 0.1 × 0.5 = 1.95. A learning rate that is too large overshoots the valley of the loss and oscillates or diverges, while one that is too small makes training progress very slowly. In practice, stochastic gradient descent (SGD), which repeats updates with gradients computed on part of the data (a mini-batch), and improved versions such as Adam are used.
- A. This subtracts the gradient 0.5 directly, forgetting to multiply by the learning rate.
- B. Correct. 2.0 − 0.1 × 0.5 = 1.95.
- C. This adds the gradient, moving the weight in the direction that increases the loss.