Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques
paperAuthors
Credibility Rating
Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: arXiv
Introduces InterpBench, a benchmark of semi-synthetic transformers with known circuits for validating mechanistic interpretability methods, addressing a critical gap in evaluating neural network interpretability techniques essential for AI safety.
Paper Details
Metadata
Abstract
Mechanistic interpretability methods aim to identify the algorithm a neural network implements, but it is difficult to validate such methods when the true algorithm is unknown. This work presents InterpBench, a collection of semi-synthetic yet realistic transformers with known circuits for evaluating these techniques. We train simple neural networks using a stricter version of Interchange Intervention Training (IIT) which we call Strict IIT (SIIT). Like the original, SIIT trains neural networks by aligning their internal computation with a desired high-level causal model, but it also prevents non-circuit nodes from affecting the model's output. We evaluate SIIT on sparse transformers produced by the Tracr tool and find that SIIT models maintain Tracr's original circuit while being more realistic. SIIT can also train transformers with larger circuits, like Indirect Object Identification (IOI). Finally, we use our benchmark to evaluate existing circuit discovery techniques.
Summary
This paper introduces InterpBench, a benchmark of semi-synthetic transformers with known internal circuits designed to evaluate mechanistic interpretability methods. The authors develop Strict Interchange Intervention Training (SIIT), an improved training technique that aligns neural network computations with specified causal models while preventing non-circuit components from influencing outputs. They demonstrate that SIIT can produce realistic transformers with known circuits—including complex ones like Indirect Object Identification—and use this benchmark to evaluate existing circuit discovery techniques, addressing a key validation challenge in mechanistic interpretability research.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Interpretability | Research Area | 66.0 |
Cached Content Preview
HTML conversions [sometimes display errors](https://info.dev.arxiv.org/about/accessibility_html_error_messages.html) due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.
- failed: csvsimple-ł\_\_csvsim\_package\_expl\_tl.sty
Authors: achieve the best HTML results from your LaTeX submissions by following these [best practices](https://info.arxiv.org/help/submit_latex_best_practices.html).
[License: CC BY 4.0](https://info.arxiv.org/help/license/index.html#licenses-available)
arXiv:2407.14494v3 \[cs.LG\] 11 Oct 2025
# InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques
Report issue for preceding element
Rohan Gupta
cybershiptrooper@gmail.com
&Iván Arcuschin11footnotemark: 1
University of Buenos Aires
iarcuschin@dc.uba.ar
Equal contributions.Thomas Kwa
kwathomas0@gmail.com
&Adrià Garriga-Alonso
FAR AI
adria@far.ai
Report issue for preceding element
###### Abstract
Report issue for preceding element
Mechanistic interpretability methods aim to identify the algorithm a neural network implements, but it is difficult to validate such methods when the true algorithm is unknown. This work presents InterpBench, a collection of semi-synthetic yet realistic transformers with known circuits for evaluating these techniques. We train simple neural networks using a stricter version of Interchange Intervention Training (IIT) which we call Strict IIT (SIIT). Like the original, SIIT trains neural networks by aligning their internal computation with a desired high-level causal model, but it also prevents non-circuit nodes from affecting the model’s output.
We evaluate SIIT on sparse transformers produced by the Tracr tool and find that SIIT models maintain Tracr’s original circuit while being more realistic. SIIT can also train transformers with larger circuits, like Indirect Object Identification (IOI). Finally, we use our benchmark to evaluate existing circuit discovery techniques.
Report issue for preceding element
## 1 Introduction
Report issue for preceding element
The field of mechanistic interpretability (MI) aims to reverse-engineer the algorithm implemented by a neural network
\[ [14](https://arxiv.org/html/2407.14494v3#bib.bib14 "")\]. The current MI paradigm holds that the neural network (NN) represents concepts as
_features_, which may have their dedicated subspace \[ [31](https://arxiv.org/html/2407.14494v3#bib.bib31 ""), [8](https://arxiv.org/html/2407.14494v3#bib.bib8 "")\] or be in _superposition_ with
other features
\[ [32](https://arxiv.org/html/2407.14494v3#bib.bib32 ""), [15](https://arxiv.org/html/2407.14494v3#bib.bib15 ""), [16](https://arxiv.org/html/2407.14494v3#bib.bib16 "")\]. The NN
arrives at its output by composing many _circuits_, which are subcomponents that implement particular funct
... (truncated, 98 KB total)25d0620e3c6a2ea4 | Stable ID: N2RmN2NmOD