L2 Regularization

$L_{2}$ regularization, also known as weight decay, is a regularization technique applied to the weights of neural networks. By adding an $L_{2}$ norm penalty term of the weights to the loss function, a new loss function is formed: $L_{new}(w) = L_{original}(w) + \lambda w^Tw$, where $\lambda$ controls the strength of the penalty, encouraging the weight values to be smaller. This method helps prevent overfitting and improves generalization. Weight decay can be directly incorporated into the weight update rule, rather than being defined implicitly through the objective function alone.