litlas
Log inCreate account

Copy Suppression: Comprehensively Understanding an Attention Head

Authors: Callum McDougall, Arthur Conmy, Cody Rushing, Thomas McGrath, Neel Nanda - arXiv 2023 cited by 50

We present a single attention head in GPT-2 Small that has one main role across the entire training distribution. If components in earlier layers predict a certain token, and this token appears earlier in the context, the head suppresses it: we call this copy suppression. Attention Head 10.7 (L10H7) suppresses naive copying behavior which improves overall model calibration. This explains why multiple prior works studying certain narrow tasks found negative heads that systematically favored the wrong answer. We uncover the mechanism that the Negative Heads use for copy suppression with weights-based evidence and are able to explain 76.9% of the impact of L10H7 in GPT-2 Small. To the best of our knowledge, this is the most comprehensive description of the complete role of a component in a language model to date. One major effect of copy suppression is its role in self-repair. Self-repair refers to how ablating crucial model components results in downstream neural network parts compensating for this ablation. Copy suppression leads to self-repair: if an initial overconfident copier is ablated, then there is nothing to suppress. We show that self-repair is implemented by several mechanisms, one of which is copy suppression, which explains 39% of the behavior in a narrow task. Interactive visualisations of the copy suppression phenomena may be seen at our web app https://copy-suppression.streamlit.app/

A MultiscaleVisualization of…A Multiscale Visualization of Attention in the Transformer ModelCompositionalExplanations of NeuronsCompositional Explanations of NeuronsAn InterpretabilityIllusion for BERTAn Interpretability Illusion for BERTAnalyzing Transformersin Embedding SpaceAnalyzing Transformers in Embedding SpaceTowards AutomatedCircuit Discovery for…Towards Automated Circuit Discovery for Mechanistic InterpretabilityHow to use and interpretactivation patchingHow to use and interpret activation patchingThe RemarkableRobustness of LLMs…The Remarkable Robustness of LLMs: Stages of Inference?A Practical Review ofMechanistic…A Practical Review of Mechanistic Interpretability for Transformer-Based Language ModelsUniversal Neurons inGPT2 Language ModelsUniversal Neurons in GPT2 Language ModelsAttention Heads of LargeLanguage Models: A…Attention Heads of Large Language Models: A SurveyUniversal Response andEmergence of Induction…Universal Response and Emergence of Induction in LLMsLoFiT: LocalizedFine-tuning on LLM…LoFiT: Localized Fine-tuning on LLM RepresentationsIs This the Subspace YouAre Looking for? An…Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation PatchingA Primer on the InnerWorkings of…A Primer on the Inner Workings of Transformer-based Language ModelsArithmetic WithoutAlgorithms: Language…Arithmetic Without Algorithms: Language Models Solve Math With a Bag of HeuristicsAttention heads of largelanguage modelsAttention heads of large language modelsCopy Suppression:Comprehensively…Copy Suppression: Comprehensively Understanding an Attention HeadEarlier referencesFocus paperCiting papersOlderNewer

Click a node to pin it, click the empty canvas to go back to this paper, or hover to preview. Open a node’s page from its title.