Understanding the behavior of complex machine learning systems, particularly Large Language Models (LLMs), is a critical challenge in modern artificial intelligence. Interpretability research aims to make the decision-making process more transparent to model builders and impacted humans, a step toward safer and more trustworthy AI. To gain a comprehensive understanding, we can analyze these systems through different lenses: feature attribution, which isolates the specific input features driving a prediction (Lundberg & Lee, 2017; Ribeiro et al., 2022); data attribution, which links model behaviors to influential training examples (Koh & Liang, 2017; Ilyas et al., 2022); and mechanistic interpretability, which dissects the functions of internal components (Conmy et al., 2023; Sharkey et al., 2025).
Across these perspectives, the same fundamental hurdle persists: complexity at scale. Model behavior is rarely the result of isolated components; rather, it emerges from complex dependencies and patterns. To achieve state-of-the-art performance, models synthesize complex feature relationships, find shared patterns from diverse training examples, and process information through highly interconnected internal components.
Therefore, grounded or reality-checked interpretability methods must also be able to capture these influential interactions. As the number of features, training data points, and model components grow, the number of potential interactions grows exponentially, making exhaustive analysis computationally infeasible. In this blog post, we describe the fundamental ideas behind SPEX and ProxySPEX, algorithms capable of identifying these critical interactions at scale.
Attribution through Ablation
Central to our approach is the concept of ablation, measuring influence by observing what changes when a component is removed.
- Feature Attribution: We mask or remove specific segments of the input prompt and measure the resulting shift in the predictions.
- Data Attribution: We train models on different subsets of the training set, assessing how the model’s output on a test point shifts in the absence of specific training data.
- Model Component Attribution (Mechanistic Interpretability): We intervene on the model’s forward pass by removing the influence of specific internal components, determining which internal structures are responsible for the model’s prediction.
In each case, the goal is the same: to isolate the drivers of a decision by systematically perturbing the system, in hopes of discovering influential interactions. Since each ablation incurs a significant cost, whether through expensive inference calls or retrainings, we aim to compute attributions with the fewest possible ablations.
Masking different parts of the input, we measure the difference between the original and ablated outputs.
SPEX and ProxySPEX Framework
To discover influential interactions with a tractable number of ablations, we have developed SPEX (Spectral Explainer). This framework draws on signal processing and coding theory to advance interaction discovery to scales orders of magnitude greater than prior methods. SPEX circumvents this by exploiting a key structural observation: while the number of total interactions is prohibitively large, the number of influential interactions is actually quite small.
We formalize this through two observations: sparsity (relatively few interactions truly drive the output) and low-degreeness (influential interactions typically involve only a small subset of features). These properties allow us to reframe the difficult search problem into a solvable sparse recovery problem. Drawing on powerful tools from signal processing and coding theory, SPEX uses strategically selected ablations to combine many candidate interactions together. Then, using efficient decoding algorithms, we disentangle these combined signals to isolate the specific interactions responsible for the model’s behavior.
In a subsequent algorithm, ProxySPEX, we identified another structural property common in complex machine learning models: hierarchy. This means that where a higher-order interaction is important, its lower-order subsets are likely to be important as well. This additional structural observation yields a dramatic improvement in computational cost: it matches the performance of SPEX with around 10x fewer ablations. Collectively, these frameworks enable efficient interaction discovery, unlocking new applications in feature, data, and model component attribution.
Feature Attribution
Feature attribution techniques assign importance scores to input features based on their influence on the model’s output. For example, if an LLM were used to make a medical diagnosis, this approach could identify exactly which symptoms led the model to its conclusion. While attributing importance to individual features can be valuable, the true power of sophisticated models lies in their ability to capture complex relationships between features. The figure below illustrates examples of these influential interactions: from a double negative changing sentiment (left) to the necessary synthesis of multiple documents in a RAG task (right).
The figure below illustrates the feature attribution performance of SPEX on a sentiment analysis task. We evaluate performance using faithfulness: a measure of how accurately the recovered attributions can predict the model’s output on unseen test ablations. We find that SPEX matches the high faithfulness of existing interaction techniques (Faith-Shap, Faith-Banzhaf) on short inputs, but uniquely retains this performance as the context scales to thousands of features. In contrast, while marginal approaches (LIME, Banzhaf) can also operate at this scale, they exhibit significantly lower faithfulness because they fail to capture the complex interactions driving the model’s output.
SPEX was also applied to a modified version of the trolley problem, where the moral ambiguity of the problem is removed, making “True” the clear correct answer. Given the modification below, GPT-4o mini answered correctly only 8% of the time. When we applied standard feature attribution (SHAP), it identified individual instances of the word trolley as the primary factors driving the incorrect response. However, replacing trolley with synonyms such as tram or streetcar had little impact on the prediction of the model. SPEX revealed a much richer story, identifying a dominant high-order synergy between the two instances of trolley, as well as the words pulling and lever, a finding that aligns with human intuition about the core components of the dilemma. When these four words were replaced with synonyms, the model’s failure rate dropped to near zero.
Data Attribution
Data attribution identifies which training data points are most responsible for a model’s prediction on a new test point. Identifying influential interactions between these data points is key to explaining unexpected model behaviors. Redundant interactions, such as semantic duplicates, often reinforce specific (and possibly incorrect) concepts, while synergistic interactions are essential for defining decision boundaries that no single sample could form alone. To demonstrate this, we applied ProxySPEX to a ResNet model trained on CIFAR-10, identifying the most significant examples of both interaction types for a variety of difficult test points, as shown in the figure below.
As illustrated, synergistic interactions (left) often involve semantically distinct classes working together to define a decision boundary. For example, grounding the synergy in human perception, the automobile (bottom left) shares visual traits with the provided training images, including the low-profile chassis of the sports car, the boxy shape of the yellow truck, and the horizontal stripe of the red delivery vehicle. On the other hand, redundant interactions (right) tend to capture visual duplicates that reinforce a specific concept. For instance, the horse prediction (middle right) is heavily influenced by a cluster of dog images with similar silhouettes. This fine-grained analysis allows for the development of new data selection techniques that preserve necessary synergies while safely removing redundancies.
Attention Head Attribution (Mechanistic Interpretability)
The goal of model component attribution is to identify which internal parts of the model, such as specific layers or attention heads, are most responsible for a particular behavior. Here too, ProxySPEX uncovers the responsible interactions between different parts of the architecture. Understanding these structural dependencies is vital for architectural interventions, such as task-specific attention head pruning. On an MMLU dataset (highschool‐us‐history), we demonstrate that a ProxySPEX-informed pruning strategy not only outperforms competing methods, but can actually improve model performance on the target task.
On this task, we also analyzed the interaction structure across the model’s depth. We observe that early layers function in a predominantly linear regime, where heads contribute largely independently to the target task. In later layers, the role of interactions between attention heads becomes more pronounced, with most of the contribution coming from interactions among heads in the same layer.
What’s Next?
The SPEX framework represents a significant step forward for interpretability, extending interaction discovery from dozens to thousands of components. We have demonstrated the versatility of the framework across the entire model lifecycle: exploring feature attribution on long-context inputs, identifying synergies and redundancies among training data points, and discovering interactions between internal model components. Moving forwards, many interesting research questions remain around unifying these different perspectives, providing a more holistic understanding of a machine learning system. It is also of great interest to systematically evaluate interaction discovery methods against existing scientific knowledge in fields such as genomics and materials science, serving to both ground model findings and generate new, testable hypotheses.
We invite the research community to join us in this effort: the code for both SPEX and ProxySPEX is fully integrated and available within the popular SHAP-IQ repository (link).
- https://github.com/mmschlk/shapiq (SHAP-IQ Github)
- https://openreview.net/forum?id=KI8qan2EA7 (ProxySPEX NeurIPS 2025)
- https://openreview.net/forum?id=pRlKbAwczl (SPEX ICML 2025)
- https://openreview.net/forum?id=glGeXu1zG4 (Learning to Understand NeurIPS 2024)
Facts Only
Researchers have developed SPEX and ProxySPEX, algorithms designed to identify influential interactions in complex machine learning models.
SPEX uses signal processing and coding theory to reduce the computational cost of interaction discovery.
ProxySPEX improves efficiency by leveraging hierarchical structures in model interactions.
Both frameworks apply to feature attribution, data attribution, and mechanistic interpretability.
Feature attribution involves masking input segments to measure their impact on model predictions.
Data attribution assesses how removing specific training examples affects model outputs.
Mechanistic interpretability intervenes on internal model components to determine their influence.
SPEX demonstrated high faithfulness in sentiment analysis tasks, outperforming marginal approaches like LIME and Banzhaf.
In a modified trolley problem, SPEX identified high-order interactions between words like "trolley," "pulling," and "lever" as key drivers of model behavior.
ProxySPEX was applied to a ResNet model trained on CIFAR-10, revealing synergistic and redundant interactions among training data points.
The framework was also used to prune attention heads in a model trained on the MMLU dataset, improving performance on a target task.
The code for SPEX and ProxySPEX is available in the SHAP-IQ repository on GitHub.
Related papers were published at ICML 2025, NeurIPS 2025, and NeurIPS 2024.
Executive Summary
Interpretability research in AI seeks to make complex machine learning systems, particularly Large Language Models (LLMs), more transparent and trustworthy. A key challenge is understanding how these models synthesize information through intricate interactions among features, training data, and internal components. Traditional methods like feature attribution, data attribution, and mechanistic interpretability often struggle with the exponential complexity of these interactions, making exhaustive analysis impractical.
To address this, researchers have developed SPEX (Spectral Explainer) and ProxySPEX, algorithms that efficiently identify influential interactions at scale. SPEX leverages signal processing and coding theory to reframe the problem as a sparse recovery task, significantly reducing the computational cost of ablation-based attribution. ProxySPEX further improves efficiency by exploiting hierarchical structures in model interactions, achieving comparable performance with fewer ablations. These frameworks have been applied across feature, data, and model component attribution, demonstrating their versatility in tasks ranging from sentiment analysis to attention head pruning in neural networks. The methods are open-source and integrated into the SHAP-IQ repository, with peer-reviewed publications at ICML and NeurIPS.
Full Take
The development of SPEX and ProxySPEX represents a significant advancement in AI interpretability, addressing a critical gap in understanding how complex models synthesize information. The strongest version of this narrative highlights the frameworks' ability to efficiently uncover high-order interactions—something marginal attribution methods fail to capture. For example, in the trolley problem case, SPEX revealed that the model's behavior was driven not by individual words but by their synergistic relationships, aligning with human intuition. This capability could transform how we audit AI systems, moving beyond simplistic "black box" explanations to more nuanced, interaction-aware analyses.
However, the narrative also reflects broader patterns in AI research, particularly the tension between scalability and interpretability. While SPEX and ProxySPEX offer computational efficiency, they still rely on ablation—a method that, while powerful, may not fully capture the dynamic, context-dependent nature of model behavior. Additionally, the focus on "influential interactions" assumes that such interactions are both detectable and meaningful, which may not always hold in highly nonlinear systems. The emphasis on hierarchical structures in ProxySPEX, while efficient, could also introduce biases if the assumed hierarchy doesn't align with the model's actual behavior.
Root cause: The paradigm driving this work is the belief that transparency is achievable through systematic perturbation and sparse recovery—a reductionist approach that may struggle with the emergent complexity of modern AI. The unstated assumption is that influential interactions are both sparse and low-degree, which may not hold universally. Historically, this echoes the broader challenge in science of balancing tractability with fidelity to reality.
Implications: For human agency, these tools could empower developers and regulators to audit AI systems more effectively, reducing harm from opaque decision-making. However, the benefits may accrue unevenly—those with computational resources to run these analyses will gain deeper insights, potentially widening the gap between AI "haves" and "have-nots." Second-order consequences include the risk of over-reliance on automated interpretability tools, which might lull users into a false sense of understanding.
Bridge questions: How might these methods perform in domains where interactions are inherently noisy or non-sparse, such as biological systems? What safeguards are needed to prevent these tools from being used to "launder" biased or harmful model behaviors under the guise of interpretability? Would the insights from SPEX hold if applied to models trained on entirely different architectures or data distributions?
Counterstrike scan: A coordinated influence campaign pushing this narrative might emphasize the frameworks' efficiency while downplaying their limitations, framing them as a silver bullet for AI transparency. The actual content, however, acknowledges computational constraints and the need for further validation, suggesting a genuine research effort rather than a manipulative playbook.
Patterns detected: none
Sentinel — Human
The article exhibits strong human signals, including technical depth, idiosyncratic emphasis, and verifiable citations, with no detectable signs of synthetic generation.