English
Menu

What is Activation function?

The function applied in each neuron of a neural network to the weighted sum of its inputs (plus a bias). Using a nonlinear function lets a network with stacked layers represent complex relationships. ReLU is common in hidden layers, while output layers often use the sigmoid function for two-class classification, the softmax function for multi-class classification and the identity function (no change) for regression.

Japanese name: 活性化関数

Activation function appears in 4 practice questions across 2 topics, including Machine Learning Basics, AI Hardware & Software Stack.

Field: Machine Learning Basics

Why a nonlinear function is needed

Without an activation function (that is, with the identity function), each layer is only a matrix multiplication plus a vector addition. Any number of stacked linear transformations can be combined into a single linear transformation, so however deep the network, it can only represent a linear function of its inputs. Inserting nonlinear activation functions lets it represent relationships that cannot be separated by a single straight line, such as XOR, and far more complex ones.

Common functions

The sigmoid function σ(z) = 1 ÷ (1 + e^(−z)) keeps its output between 0 and 1, so the output can be treated as a probability. tanh keeps its output between −1 and 1, centered on 0. ReLU(z) = max(0, z) is a simple function that passes positive inputs unchanged and outputs 0 for inputs of 0 or less; it is cheap to compute and its gradient does not easily vanish, so it is the standard choice for hidden layers of deep networks. Smooth variants of ReLU such as GELU are also widely used.

Relation to vanishing gradients

For inputs of large magnitude, the sigmoid function and tanh saturate at a constant output and their derivatives become almost 0. The derivative of the sigmoid function is at most 0.25, so as layers are stacked the gradient shrinks multiplicatively during backpropagation and the layers near the input learn poorly (vanishing gradients). ReLU has a derivative of 1 over the positive range, so its gradient does not easily shrink. On the other hand, a neuron whose input is always negative keeps a gradient of 0 and stops learning, which is why variants such as Leaky ReLU give a small slope to the negative range as well.

Try it: questions about Activation function

In a neural network with many stacked layers, which of the following is the reason that using the sigmoid function as the activation function makes the layers near the input hard to train?

  1. AThe derivative of the sigmoid function is at most 0.25, so the gradient shrinks each time it passes through a layer in backpropagation and becomes almost 0 in the layers near the input
  2. BThe output of the sigmoid function is not confined to the range 0 to 1, so values grow at each layer and the computation overflows
  3. CThe sigmoid function has points where it cannot be differentiated, so some layers cannot obtain gradients in backpropagation
Show answer and explanation

Answer: A. The derivative of the sigmoid function is at most 0.25, so the gradient shrinks each time it passes through a layer in backpropagation and becomes almost 0 in the layers near the input

In backpropagation, the derivatives of the activation functions of successive layers are multiplied into the gradient one after another. The derivative of the sigmoid function, σ(z)(1 − σ(z)), is at most 0.25, so after 10 layers, for example, the gradient becomes as small as 0.25¹⁰ ≈ 9.5 × 10⁻⁷. This is called the vanishing gradient problem. Countermeasures include using ReLU, whose derivative is 1 over the positive range, adding residual (skip) connections that carry signals past layers, choosing the initial weights carefully, and using batch normalization.

FAQ

Is the softmax function an activation function?

It is treated as one of the activation functions used in the output layer for multi-class classification. It differs from the others in that, instead of acting on each neuron separately, it uses all the values of the output layer to turn them into probabilities that sum to 1.

Is it related to "activation" in semiconductor manufacturing?

No. The activation function of a neural network takes its name from the way it imitates a nerve cell becoming excited (firing). Activation in semiconductor manufacturing refers to heat-treating implanted dopants so that they occupy lattice sites in the crystal and become able to supply carriers.

Activation function in the explanations

Questions that mention Activation function (4)

  1. In a neural network with many stacked layers, which of the following is the reason that using the sigmoid function as the activation function makes the layers near the input hard to train?
  2. What is the main purpose of operator fusion in AI computation?
  3. Which logic function cannot be represented by a simple perceptron consisting only of input and output layers (a single neuron that outputs 0 or 1 with a threshold)?
  4. A small neural network with inputs x₁ = 1 and x₂ = 2 has two hidden neurons h₁ = ReLU(x₁ + x₂ − 1) and h₂ = ReLU(x₁ − 2x₂ + 1), and output y = 2h₁ + 3h₂ + 1. What is the value of y?

Books on Machine Learning Basics Ad

As an Amazon Associate I earn from qualifying purchases.

Semiconductor, Electronics and Computing Calculators

CNN output size calculatorStack convolution, pooling and fully connected layers and get each layer's output size, parameter count and multiply-accumulate count

Related terms

Backpropagation Operator fusion Confusion matrix Convolutional neural network (CNN) Cross-validation Gradient descent k-means clustering Overfitting Precision and recall Principal component analysis (PCA)

Back to the glossary