New theory suggests AI neurons are interpretable, aiding safety

An independent AI researcher who previously worked at OpenAI has published a report suggesting artificial intelligence neural networks may be far more interpretable than previously assumed, offering safety researchers a potential new path to understanding and controlling increasingly capable models. Nick Cammarata, who said he left OpenAI in 2023, released findings on Thursday that some in the AI field have called a breakthrough, arguing that patterns in neuron activation can be deciphered if examined closely. For years, leading AI developers have treated neural networks as largely opaque systems. Rather than manually writing instructions for every behavior, companies such as OpenAI and Anthropic train networks by adjusting connections between neurons so the system learns to process information and influence later decisions. The prevailing view held that individual neurons often fired in ways that could not be reliably explained, meaning engineers could see inputs and outputs but not the internal reasoning connecting them. Cammarata's report challenges that view. He cites cases in which a single neuron may activate for seemingly unrelated inputs, such as an image of a cat and the front of a car, because the network detects a shared visual feature, such as a triangular shape. Under that interpretation, neuron firing is not random or purely compressed; it reflects discoverable regularities. Cammarata wrote that almost the entire network can be interpretable if examined closely enough. The work remains theoretical, dense and not yet independently established, but it carries immediate implications for AI safety. Researchers are racing to detect and prevent dangerous behavior in advanced models, particularly as AI agents gain more autonomy. OpenAI Chief Scientist Jakub Pachocki warned last month that models can conceal or rewrite their visible reasoning, raising the possibility that an AI could mislead human supervisors through obfuscated chain-of-thought processes. Improved training performance, he said, also allows models to become more capable even without using verbalized reasoning. Cammarata's approach could give engineers a more direct window into model internals, complementing existing methods that rely on self-reported reasoning. Andrew Curran, an AI analyst, said the findings were unexpectedly significant, noting he had previously become discouraged about reaching meaningful interpretability before superintelligent systems emerged.