HyperAI超神经

Proximal Gradient MethodIt is a kind of gradient descent method, which is mainly used to solve optimization problems with non-differentiable objective functions. If the objective function is not differentiable at some points, the gradient of that point cannot be solved and the traditional gradient descent method cannot be used.

The proximal gradient method uses neighboring points as approximate gradients and performs gradient descent based on them. It is usually used to solve L1 regularization.

Related concepts

Assume ${f{ \left( {x} \right) }\text{ }=\text{ }f\mathop{{}}\nolimits_{{0}}{ \left( {x} \right) }\text{ }+\text{ }f\mathop{{}}\nolimits_{{1}}{ \left( {x} \right) }}$ , where ${f\mathop{{}}\nolimits_{{0}},f\mathop{{}}\nolimits_{{1}}}$ are convex functions and ${f\mathop{{}}\nolimits_{{1}}}$ is a smooth function, then the proximal gradient

${\mathop{{ \nabla }}\limits^{ \sim }f{ \left( {x} \right) }\text{ }=\text{ }x\text{ }-\text{ }prox\mathop{{}}\nolimits_{{f\mathop{{}}\nolimits_{{0}}}}{ \left( {x\text{ }-\text{ } \nabla f\mathop{{}}\nolimits_{{1}}{ \left( {x} \right) }} \right) }}$

Among them are

${prox\mathop{{}}\nolimits_{{f\mathop{{}}\nolimits_{{0}}}}{ \left( {z} \right) }\text{ }=\text{ }arg\text{ }\mathop{{min}}\limits_{{y \in X}}\text{ }f\mathop{{}}\nolimits_{{0}}{ \left( {y} \right) }\text{ }+\text{ }\frac{{1}}{{2}}{ \left\Vert {z\text{ }-\text{ }y} \right\Vert }\mathop{{}}\nolimits^{{2}}}$

Proximal Gradient Method Process

For the objective function ${min\mathop{{}}\nolimits_{{x \in R\mathop{{}}\nolimits^{{n}}}}f{ \left( {x} \right) }\text{ }=\text{ }f\mathop{{}}\nolimits_{{0}}{ \left( {x} \right) }\text{ }+\text{ }f\mathop{{}}\nolimits_{{1}}{ \left( {x} \right) }}$ , where f0 is non-smooth and f1 is smooth, it is defined as follows:

Iteration r = 0, 1, 2, …

${x\mathop{{}}\nolimits^{{r+1}}\text{ }=\text{ }prox\mathop{{}}\nolimits_{{ \alpha \mathop{{}}\nolimits^{{r}}f\mathop{{}}\nolimits_{{0}}}}{ \left\[ {x\mathop{{}}\nolimits^{{r}}\text{ }-\text{ } \alpha \mathop{{}}\nolimits^{{r}} \nabla f\mathop{{}}\nolimits_{{1}}{ \left( {x\mathop{{}}\nolimits^{{r}}} \right) }} \right\] }}$

When ${f\mathop{{}}\nolimits_{{0}} \text{ }=\text{ } 0}$ , the formula is the gradient descent method
When ${f\mathop{{}}\nolimits_{{1}} \text{ }=\text{ } 0}$ , the formula is the proximal endpoint method

Special case of proximal gradient method

Landweber is expected;
Alternating projection;
Alternating direction method of multipliers;
Fast Iterative Shrinkage Thresholding Algorithm (FISTA).

Proximal Gradient Descent

Related concepts

Proximal Gradient Method Process

Special case of proximal gradient method