Differentiable Programming
Differentiable Programming is a programming paradigm in which digital computer programs can be fully differentiated through automatic differentiation. This allows gradient-based optimization of parameters in the program, which is usually done through gradient descent, as well as other learning methods based on higher-order derivative information. Differentiable programming has been widely used in various fields, especially in scientific computing and machine learning.
Methods of Differentiable Programming
Most differentiable programming frameworks work by building graphs that contain the control flow and data structures in a program, and generally fall into two categories of approaches:
- Static, compiled graph-based approach: For example, TensorFlow, Theano, and MXNet. They tend to allow good compiler optimizations and easier scaling to large systems, but their static nature limits interactivity and the types of programs that can be easily created (such as those involving loops or recursion), and makes it more difficult for users to reason about their programs effectively. A proof-of-concept compiler toolchain called Myia uses a subset of Python as a front end and supports higher-order functions, recursion, and higher-order derivatives.
- Operator overloading, dynamic graph-based approach: For example, PyTorch and NumPy's autograd package. Their dynamic and interactive nature makes it easier to write and reason about most programs.
A limitation of early approaches is that they can only differentiate code written in a way that fits into a framework, limiting their interoperability with other programs. Newer approaches address this problem by constructing graphs based on the language's grammar, or IR, allowing differentiation of arbitrary code.
Differentiable programming has made significant progress in various fields beyond its traditional applications. For example, in healthcare and life sciences, it is used for deep learning in biophysics-based modeling of molecular mechanisms. This involves leveraging differentiable programming in areas such as protein structure prediction and drug discovery. These applications demonstrate the potential of differentiable programming to facilitate significant advances in understanding complex biological systems and improving healthcare solutions.