Identifying Interactions at Scale for LLMs

--> Understanding the behavior of complex machine learning systems, particularly Large Language Models (LLMs), is a critical challenge in modern artificial intelligence. Interpretability research aims to make the decision-making process more transparent to model builders and impacted humans, a step toward safer and more trustworthy AI. To gain a comprehensive understanding, we can analyze these systems through different lenses: feature attribution , which isolates the specific input features driving a prediction ( Lundberg & Lee, 2017 ; Ribeiro et al., 2022 ); data attribution , which links model behaviors to influential training examples ( Koh & Liang, 2017 ; Ilyas et al., 2022 ); and mechanistic interpretability , which dissects the functions of internal components ( Conmy et al., 2023 ; Sharkey et al., 2025 ). Across these perspectives, the same fundamental hurdle persists: complexity at scale . Model behavior is rarely the result of isolated components; rather, it emerges from complex dependencies and patterns. To achieve state-of-the-art performance, models synthesize complex feature relationships, find shared patterns from diverse training examples, and process information through highly interconnected internal components. Therefore, grounded or reality-checked interpretability methods must also be able to capture these influential interactions . As the number of features, training data points, and model components grow, the number of potential interactions grows exponentially, making exhaustive analysis computationally infeasible. In this blog post, we describe the fundamental ideas behind SPEX and ProxySPEX , algorithms capable of identifying these critical interactions at scale. Attribution through Ablation Central to our approach is the concept of ablation , measuring influence by observing what changes when a component is removed. Feature Attribution: We mask or remove specific segments of the input prompt and measure the resulting shift in the prediction