Using geometry and physics to explain how deep neural networks learn features, researchers from the University of Basel and the University of Science and Technology of China have developed a novel theoretical framework inspired by mechanical systems. By modeling deep neural networks (DNNs) as spring-block chains—a system used in geophysics to study earthquakes and material deformation—they uncovered a "phase diagram" that reveals how DNNs separate data across layers during training. This approach draws a striking parallel between the behavior of neural network layers and the movement of blocks connected by springs sliding on a rough surface. Just as springs extend under force, neural network layers simplify and separate data, with nonlinearity in the network mimicking friction. Adding noise during training, akin to shaking the system, reduces friction and leads to more uniform data separation across layers—a phenomenon similar to "acoustic lubrication" in engineering. The team discovered that the "law of data separation," where each layer improves data separation by a consistent amount in well-trained networks, emerges naturally from this physical model. This behavior breaks down under certain hyperparameter choices, highlighting the importance of balancing nonlinearity, noise, and learning rate. Their spring-block theory not only explains this pattern but also provides a simple, intuitive way to understand complex DNN dynamics—using familiar concepts from physics rather than abstract mathematical abstractions. The model successfully predicts data separation curves during training, which correlate with a network’s ability to generalize to unseen data. Beyond explanation, the framework offers practical potential: by manipulating noise and nonlinearity, researchers could steer training to improve generalization, potentially accelerating the training of large models like transformers. The theory also opens the door to diagnostic tools that analyze internal "load distribution" in neural networks, identifying overworked or underused layers—much like stress maps in structural engineering. These insights could help detect overfitting or redundancy, guiding model improvements. With a path toward a first-principles explanation and a powerful alternative to scaling laws, this physics-inspired approach marks a significant step toward demystifying deep learning and enhancing its reliability and efficiency.

Researchers from the University of Basel and the University of Science and Technology of China have developed a novel theoretical framework to explain how deep neural networks (DNNs) learn features during training, using principles from geometry and physics. Their work, published in Physical Review Letters, draws an unexpected parallel between DNNs and mechanical systems like spring-block chains and folding rulers. At the heart of their discovery is a "law of data separation," observed in well-trained neural networks: each layer progressively separates data from different classes—such as images of cats versus dogs—by roughly the same amount. This consistent improvement in separation across layers is not just a coincidence but a key to why DNNs generalize well to new data. The team realized that this behavior closely resembles how a chain of blocks connected by springs behaves when pulled across a rough surface. In this analogy, the springs represent the linear transformations in neural network layers, while the friction between blocks and the surface models the nonlinearity introduced by activation functions. As the system is pulled, the blocks separate layer by layer—mirroring how DNNs refine their internal representations. Adding noise during training—akin to shaking or vibrating the spring-block system—reduces friction temporarily, allowing the system to redistribute separation more evenly across layers. This phenomenon, similar to "acoustic lubrication" in engineering or stick-slip dynamics in geophysics, helps explain why noise can improve training stability and generalization in DNNs. The researchers found that this spring-block model accurately predicts data separation curves during training. The shape of these curves, derived from simple mechanical principles, correlates strongly with a network’s ability to perform well on unseen data. This offers a powerful, intuitive way to understand and potentially control DNN behavior. Importantly, this approach takes a top-down, phenomenological perspective rather than relying on first-principles derivations from massive, complex networks. Instead of analyzing billions of parameters, the model uses just a few variables—like nonlinearity and noise—to capture the collective behavior of deep networks. The theory also opens the door to practical improvements. By understanding how to manipulate the shape of the data separation curve, researchers could design better training strategies, especially for large models like transformers used in large language models. It could also lead to diagnostic tools that identify "overloaded" layers—those prone to overfitting—or underused ones, signaling redundancy. In essence, the study shows that deep learning’s complex, high-dimensional processes can be understood through simple physical analogies. What was once seen as a black box of billions of parameters becomes a system governed by familiar laws of motion, force, and energy. As the researchers note, this framework may soon serve as a bridge between intuition and computation—allowing scientists to leverage everyday mechanical intuition to guide the design and optimization of AI systems. Their work represents a significant step toward a deeper, more unified understanding of how artificial neural networks learn and generalize.