Machine Learning Basics
In a neural network with many stacked layers, which of the following is the reason that using the sigmoid function as the activation function makes the layers near the input hard to train?
- AThe derivative of the sigmoid function is at most 0.25, so the gradient shrinks each time it passes through a layer in backpropagation and becomes almost 0 in the layers near the input
- BThe output of the sigmoid function is not confined to the range 0 to 1, so values grow at each layer and the computation overflows
- CThe sigmoid function has points where it cannot be differentiated, so some layers cannot obtain gradients in backpropagation
Show answer and explanation
Answer: A. The derivative of the sigmoid function is at most 0.25, so the gradient shrinks each time it passes through a layer in backpropagation and becomes almost 0 in the layers near the input
In backpropagation, the derivatives of the activation functions of successive layers are multiplied into the gradient one after another. The derivative of the sigmoid function, σ(z)(1 − σ(z)), is at most 0.25, so after 10 layers, for example, the gradient becomes as small as 0.25¹⁰ ≈ 9.5 × 10⁻⁷. This is called the vanishing gradient problem. Countermeasures include using ReLU, whose derivative is 1 over the positive range, adding residual (skip) connections that carry signals past layers, choosing the initial weights carefully, and using batch normalization.
- A. Correct. The repeated multiplication by derivatives smaller than 1 makes the gradient vanish.
- B. The output of the sigmoid function is confined to the range 0 to 1, so values do not overflow.
- C. The sigmoid function can be differentiated everywhere.