A Unified Algebraic Perspective on Lipschitz Neural Networks

Important research efforts have focused on the design and training of neuralnetworks with a controlled Lipschitz constant. The goal is to increase andsometimes guarantee the robustness against adversarial attacks. Recentpromising techniques draw inspirations from different backgrounds to design1-Lipschitz neural networks, just to name a few: convex potential layers derivefrom the discretization of continuous dynamical systems,Almost-Orthogonal-Layer proposes a tailored method for matrix rescaling.However, it is today important to consider the recent and promisingcontributions in the field under a common theoretical lens to better design newand improved layers. This paper introduces a novel algebraic perspectiveunifying various types of 1-Lipschitz neural networks, including the onespreviously mentioned, along with methods based on orthogonality and spectralmethods. Interestingly, we show that many existing techniques can be derivedand generalized via finding analytical solutions of a common semidefiniteprogramming (SDP) condition. We also prove that AOL biases the scaled weight tothe ones which are close to the set of orthogonal matrices in a certainmathematical manner. Moreover, our algebraic condition, combined with theGershgorin circle theorem, readily leads to new and diverse parameterizationsfor 1-Lipschitz network layers. Our approach, called SDP-based Lipschitz Layers(SLL), allows us to design non-trivial yet efficient generalization of convexpotential layers. Finally, the comprehensive set of experiments on imageclassification shows that SLLs outperform previous approaches on certifiedrobust accuracy. Code is available athttps://github.com/araujoalexandre/Lipschitz-SLL-Networks.