Algorithmic Language Models with Neurally Compiled Libraries Key tasks such as reasoning and planning are fundamentally algorithmic in nature. To tackle these tasks effectively, it's essential to implement genuine reasoning or planning algorithms, rather than taking shortcuts. Unfortunately, large language models (LLMs) generally lack robust algorithmic capabilities. This shortfall is primarily due to limitations in neural network optimization algorithms, optimization data, optimization objectives, and the architecture's expressive capacity. To address this issue, our paper introduces a method for augmenting LLMs with fundamental operations and complex differentiable program libraries. This approach ensures that common algorithms do not have to be learned from scratch. We have built upon the LLaMA3 transformer architecture by incorporating memory, registers, basic operations, and adaptive recursion. We also define a process for directly compiling algorithms into a differentiable base library, which can be used locally and provide gradients for optimization. In our initial research, we investigated the feasibility of combining LLaMA3 with a differentiable computer. Specifically, we fine-tuned a small transformer on simple algorithmic tasks that require variable computational depth. The results of this study show promising potential for enhancing the algorithmic capabilities of LLMs without the need for extensive retraining or complex modifications. This method could significantly improve the performance of LLMs in tasks that demand precise and systematic reasoning, making them more reliable in real-world applications. By leveraging the power of differentiable libraries, these models can better handle complex tasks that require iterative and structured computation, such as those involving logical deductions or detailed planning. Our approach not only simplifies the learning process but also opens up new avenues for integrating advanced algorithms into LLMs. This could pave the way for more sophisticated and versatile AI systems, capable of tackling a broader range of challenges in fields such as natural language processing, robotics, and automated decision-making.