Regularization
RegularizationIt is the process of introducing additional information to solve a problem or prevent overfitting. It is a common method in mathematics, statistics, and computer science, especially in machine learning and inverse problems.
Common regularization methods
- L2 regularization: The sum of the squares of all parameters w, divided by the sample size n of the training set, λ is the regularization term coefficient, which is used to weigh the proportion of the regularization term and the C0 term, and there is also a coefficient of 1/2. The purpose of adding regularization terms is to reduce the sum of squares of parameters. The most commonly used regularization method in machine learning is to impose L2 norm constraints on weights;
- L1 regularization: The sum of the absolute values of all weights w is multiplied by λ/n. L1 regularization adds a regularization term to the objective function to reduce the absolute value of the parameters and is often used in feature selection.
- Dropout: L1 and L2 regularization are achieved by modifying the cost function, while Dropout is achieved by modifying the neural network itself. It is a technique used when training neural networks.
- Drop Connect: A generalization of Dropout, another regularization strategy to reduce overfitting of algorithms. Drop Connect usually sets a randomly selected subset of the network architecture weights to 0.
- Early stopping: Limiting the number of training iterations required for the model to minimize the cost function, which is often used to prevent over-expressed models from having poor generalization performance during training.