arxiv:2607.23379

When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles

Published on Jul 25

· Submitted by

Tatiana Gaintseva on Aug 10

Queen Mary University of London

Upvote

Authors:

Tobias Bersia ,

Abstract

Activation Oracles can become concept-specific anti-readers that selectively fail to recover internally represented hidden concepts, revealing reliability concerns for learned interpretability interfaces.

Generated by thinkingmachines/Inkling-Small

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.

View arXiv page View PDF Add to collection

Community

Atmyre

Paper submitter 1 day ago

•

edited 1 day ago

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model’s internal activations. They turn activation analysis into a chat interface: an auditor asks “what is this hidden state about?” and reads the answer. If you want to audit a model that hides something (a backdoor, a concealed goal, latent knowledge), it’s natural to think about training the oracle on that model’s own activations so it becomes a specialist reader of it.

We show that the approach actually backfires. Across all five concepts we tested, an AO fine-tuned on a subject that hides concept c becomes selectively worse at recovering c, while it still reads other concepts fine. For cooperative subjects, exact recovery of the hidden concept drops from 94% for a base AO to 35% for the oracle trained on that subject, even though an oracle trained on a different hidden concept still reaches 71%. We call this concept-specific anti-reading.

We also checked that the information is not gone, only the readout is. The concept is still recoverable by a base AO, still linearly decodable inside the fine-tuned oracle, and still present at the oracle’s mid layers. The failure is in how the oracle maps that representation to an output word. Ablating the oracle’s mid-to-late layers restores recovery, and the failing oracle is usually confident rather than uncertain.