litlas
Log inCreate account

Relating transformers to models and neural representations of the hippocampal formation

Authors: James C. R. Whittington, Joseph Warren, Tim E. J. Behrens - International Conference on Learning Representations, ICLR 2022 cited by 115

Many deep neural network architectures loosely based on brain networks have recently been shown to replicate neural firing patterns observed in the brain. One of the most exciting and promising novel architectures, the Transformer neural network, was developed without the brain in mind. In this work, we show that transformers, when equipped with recurrent position encodings, replicate the precisely tuned spatial representations of the hippocampal formation; most notably place and grid cells. Furthermore, we show that this result is no surprise since it is closely related to current hippocampal models from neuroscience. We additionally show the transformer version offers dramatic performance gains over the neuroscience version. This work continues to bind computations of artificial and brain networks, offers a novel understanding of the hippocampal-cortical interaction, and suggests how wider cortical areas may perform complex tasks beyond current neuroscience models such as language comprehension.

Grid Cells, Place Cells,and Geodesic…Grid Cells, Place Cells, and Geodesic Generalization for Spatial Reinforcement LearningDeep Supervised, but NotUnsupervised, Models Ma…Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical RepresentationHuman-level controlthrough deep…Human-level control through deep reinforcement learningDeep Residual Learningfor Image RecognitionDeep Residual Learning for Image RecognitionDense Associative Memoryfor Pattern RecognitionDense Associative Memory for Pattern RecognitionUsing Fast Weights toAttend to the Recent…Using Fast Weights to Attend to the Recent PastVector-based navigationusing grid-like…Vector-based navigation using grid-like representations in artificial agentsEmergence of grid-likerepresentations by…Emergence of grid-like representations by training recurrent neural networks to perform spatial localizationAn Image is Worth 16x16Words: Transformers for…An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleHippocampal SpatialMapping As Fast Graph…Hippocampal Spatial Mapping As Fast Graph LearningSystem Identification ofNeural Systems: If We…System Identification of Neural Systems: If We Got It Right, Would We Know?Actionable NeuralRepresentations: Grid…Actionable Neural Representations: Grid Cells from Minimal ConstraintsThe neuroconnectionistresearch programmeThe neuroconnectionist research programmeCoherently remappingtoroidal cells but not…Coherently remapping toroidal cells but not Grid cells are responsible for path integration in virtual agentsShort-term Hebbianlearning can implement…Short-term Hebbian learning can implement transformer-like attentionReplay and compositionalcomputationReplay and compositional computationTransformer Meets RemoteSensing Video Detection…Transformer Meets Remote Sensing Video Detection and Tracking: A Comprehensive SurveyDeep neural networksarchitectures from the…Deep neural networks architectures from the perspective of manifold learningNeuroformer: Multimodaland Multitask Generativ…Neuroformer: Multimodal and Multitask Generative Pretraining for Brain DataA tale of twoalgorithms: Structured…A tale of two algorithms: Structured slots explain prefrontal sequence memory and are unified with hippocampal cognitive mapsLinking In-contextLearning in Transformer…Linking In-context Learning in Transformers to Human Episodic MemoryOn Conformal Isometry ofGrid Cells: Learning…On Conformal Isometry of Grid Cells: Learning Distance-Preserving Position EmbeddingRelating transformers tomodels and neural…Relating transformers to models and neural representations of the hippocampal formationEarlier referencesFocus paperCiting papersOlderNewer

Click a node to pin it, click the empty canvas to go back to this paper, or hover to preview. Open a node’s page from its title.