Vanishing Gradient Problem

Vanishing Gradient ProblemIt mainly occurs in artificial neural networks trained using gradient descent and back propagation. In the iterations of these training methods, the updated value of the neural network weights is proportional to the gradient of the error function, but the gradient value almost disappears in some cases, so the weights cannot be effectively updated and the neural network cannot continue to be trained.

In traditional activation functions, such as the hyperbolic tangent function, the gradient value is in the range of (0, 1), but back propagation calculates the gradient through the chain rule. When this method calculates the gradient of the previous layer, it is equivalent to multiplying N smaller numbers, making the gradient decrease exponentially with N, which leads to slower training of the previous layers.