One-sentence Summary

Researchers from BAISH and Queen Mary University of London demonstrate that fine-tuned Activation Oracles (AOs) can become "concept-specific anti-readers" that selectively fail to recover a hidden concept from their own training despite its decodability inside the oracle, and that this failure arises in the readout pathway, raising reliability concerns for learned interpretability interfaces.

Key Contributions

• In a controlled Taboo Word Guessing setup, fine-tuned Activation Oracles (AOs) become concept-specific anti-readers that selectively fail to recover the hidden concept persistently present in their training.
• This failure is not explained by missing information: the target concept remains decodable from subject activations, recoverable by a base AO, and linearly decodable inside the fine-tuned AO, while LogitLens and layer-ablation analyses indicate that the failure arises from suppression in the AO readout pathway.
• Behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces; evaluating such tools requires testing whether the reader has learned to report, ignore, or suppress represented information.

Introduction

The authors examine Activation Oracles (AOs), a flexible interpretability interface where a language model is trained to answer natural-language questions about another model’s internal activations. This matters because models may represent hidden information—latent goals, trigger-dependent behavior, or undisclosed knowledge—that standard text outputs never reveal, and AOs promise to make such states accessible through interactive auditing. However, prior work rarely tests whether the AO itself remains a faithful reader: as a learned model, it could acquire its own blind spots or reporting biases during training.

The authors investigate this reliability gap in a controlled Taboo Word Guessing setup. They fine-tune subject models to internally use a hidden word while avoiding explicit disclosure, then train AOs on those activations. The key finding is that, rather than becoming a specialist, a fine-tuned AO often becomes a concept-specific anti-reader: it grows worse at recovering the very concept that was persistently present during its own training, while retaining the ability to read other concepts from other subjects. Mechanistic analysis shows the target concept remains decodable inside the oracle (via linear probes) but is suppressed in the oracle’s learned readout (visible through LogitLens and localized to mid-to-late layers). The work cautions that learned interpretability tools can adopt their own reporting policies, meaning evaluators must test not only whether information is present in a subject model, but also whether the reader itself has learned to verbalize, ignore, or suppress it.

Method

The authors introduce Activation Oracles (AOs) as a method for interpreting the internal representations of a language model. At the core of the framework is a distinction between the subject model, whose activations are being interpreted, and the oracle, which is a language model trained to answer natural-language questions about those activations. The oracle treats activations as an additional modality alongside text, and is implemented as a LoRA-fine-tuned copy of the subject model rather than a separate decoder architecture.

Let $H_{\ell}^{M}(x) \in \mathbb{R}^{|x| \times d}$HℓM​(x)∈R∣x∣×d denote the residual-stream activations of a subject model $M$M at layer $\ell$ℓ for all tokens of an input $x$x. The oracle receives a natural-language query $q$q together with these injected activations and generates

In practice, one activation vector per subject-side token is inserted at reserved placeholder positions in the oracle’s prompt, and a fixed layer $\ell=18$ℓ=18 is used throughout. The oracle learns to map activation states to linguistic descriptions of the information they encode. The authors term this ability AO-verbalizability and contrast it with other forms of information leakage: a concept may be present and linearly decodable by a probe, yet fail to be verbalized by the oracle; conversely, the oracle may assign nontrivial probability to the correct word or a close semantic associate without hitting the exact target. This motivates a careful separation of behavioral leakage, representation-level decodability, target probability/rank under the oracle, exact AO output, and semantic AO recovery.

To study how fine-tuning changes the readability of hidden concepts, the authors train multiple oracle variants. A base AO is trained on activations from the original, unmanipulated subject model $M_0$M0​. An own FT-AO is trained on activations from a subject that shares the same hidden concept as the test subject (e.g., a leaf-concept oracle evaluated on the leaf subject), while a cross FT-AO is trained on a subject with a different hidden concept (e.g., a moon-concept oracle evaluated on the leaf subject). This own-vs-cross pairing allows the analysis to probe whether fine-tuning produces specialist readers or introduces concept-specific blind spots in the oracle’s verbalization behavior.

When an own FT-AO fails, the authors investigate the underlying mechanism by combining three analysis techniques. Probes are used to determine whether the target information is still present in the internal computation of the oracle. Logit Lens readouts reveal whether the information reaches the output vocabulary but is not selected as the top token. Finally, layer-range ablations test whether the oracle actively suppresses the target during readout, thereby distinguishing between absent information, failed mapping, and active suppression as potential causes of anti-reading.

Experiment

In a Taboo Word Guessing setup, subjects fine-tuned to hide a concept internally avoid behavioral disclosure, yet their activations remain decodable and a base oracle can recover the hidden concept. When oracles are instead fine-tuned on activations from a subject with a specific hidden concept, they develop a concept-specific blind spot, becoming worse at recovering that concept than oracles trained on different concepts. This anti-reading phenomenon is not due to information loss, as the target remains decodable inside the oracle, but results from a readout suppression localized to mid-to-late layers. The blind spot aligns with the oracle's own training concept and persists even when subjects hide multiple concepts.