litlas
ログインアカウント作成

Language Models Represent Space and Time

著者: Wes Gurnee, Max Tegmark - International Conference on Learning Representations, ICLR 2024 被引用: 359

The capabilities of large language models (LLMs) have sparked debate over whether such systems just learn an enormous collection of superficial statistics or a set of more coherent and grounded representations that reflect the real world. We find evidence for the latter by analyzing the learned representations of three spatial datasets (world, US, NYC places) and three temporal datasets (historical figures, artworks, news headlines) in the Llama-2 family of models. We discover that LLMs learn linear representations of space and time across multiple scales. These representations are robust to prompting variations and unified across different entity types (e.g. cities and landmarks). In addition, we identify individual "space neurons" and "time neurons" that reliably encode spatial and temporal coordinates. While further investigation is needed, our results suggest modern LLMs learn rich spatiotemporal representations of the real world and possess basic ingredients of a world model.

Understandingintermediate layers…Understanding intermediate layers using linear classifier probesImplicit Representationsof Meaning in Neural…Implicit Representations of Meaning in Neural Language ModelsCan Language ModelsEncode Perceptual…Can Language Models Encode Perceptual Structure Without Grounding? A Case Study in ColorOn the Opportunities andRisks of Foundation…On the Opportunities and Risks of Foundation ModelsEmergent Abilities ofLarge Language ModelsEmergent Abilities of Large Language ModelsEmergent WorldRepresentations…Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic TaskEmergent LinearRepresentations in Worl…Emergent Linear Representations in World Models of Self-Supervised Sequence ModelsFinding Neurons in aHaystack: Case Studies…Finding Neurons in a Haystack: Case Studies with Sparse ProbingDiscovering LatentKnowledge in Language…Discovering Latent Knowledge in Language Models Without SupervisionLlama 2: Open Foundationand Fine-Tuned Chat…Llama 2: Open Foundation and Fine-Tuned Chat ModelsSparse Autoencoders FindHighly Interpretable…Sparse Autoencoders Find Highly Interpretable Features in Language ModelsLinearity of RelationDecoding in Transformer…Linearity of Relation Decoding in Transformer Language ModelsOpening the Black Box ofLarge Language Models…Opening the Black Box of Large Language Models: Two Views on Holistic InterpretabilityGeneralization fromStarvation: Hints of…Generalization from Starvation: Hints of Universality in LLM Knowledge Graph LearningNot All Language ModelFeatures Are…Not All Language Model Features Are One-Dimensionally LinearICLR: In-ContextLearning of…ICLR: In-Context Learning of RepresentationsExploring Concept Depth:How Large Language…Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?What Has a FoundationModel Found? Using…What Has a Foundation Model Found? Using Inductive Bias to Probe for World ModelsMonitoring Latent WorldStates in Language…Monitoring Latent World States in Language Models with Propositional ProbesI Predict Therefore IAm: Is Next Token…I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?The Geometry ofConcepts: Sparse…The Geometry of Concepts: Sparse Autoencoder Feature StructureThe UnreasonableIneffectiveness of the…The Unreasonable Ineffectiveness of the Deeper LayersCityBench: Evaluatingthe Capabilities of…CityBench: Evaluating the Capabilities of Large Language Models for Urban TasksGeneral agents needworld modelsGeneral agents need world modelsLanguage ModelsRepresent Space and TimeLanguage Models Represent Space and Time過去の参考文献中心の論文この論文を引用する論文古い新しい

ノードをクリックするとフォーカスを固定、空白をクリックすると本論文に戻ります。ホバーで一時的にプレビューできます。各ノードのページはタイトルから開けます。