litlas
Log inCreate account

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Authors: Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby - International Conference on Learning Representations, ICLR 2021 cited by 67,110

While the Transformer architecture has become the de-facto standard for natural language processing tasks, its applications to computer vision remain limited. In vision, attention is either applied in conjunction with convolutional networks, or used to replace certain components of convolutional networks while keeping their overall structure in place. We show that this reliance on CNNs is not necessary and a pure transformer applied directly to sequences of image patches can perform very well on image classification tasks. When pre-trained on large amounts of data and transferred to multiple mid-sized or small image recognition benchmarks (ImageNet, CIFAR-100, VTAB, etc.), Vision Transformer (ViT) attains excellent results compared to state-of-the-art convolutional networks while requiring substantially fewer computational resources to train.

ImageNet: A large-scalehierarchical image…ImageNet: A large-scale hierarchical image databaseImageNet Classificationwith Deep Convolutional…ImageNet Classification with Deep Convolutional Neural NetworksDeep Residual Learningfor Image RecognitionDeep Residual Learning for Image RecognitionWeight Normalization: ASimple…Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural NetworksAttention Is All YouNeedAttention Is All You NeedBERT: Pre-training ofDeep Bidirectional…BERT: Pre-training of Deep Bidirectional Transformers for Language UnderstandingLanguage Models areFew-Shot LearnersLanguage Models are Few-Shot LearnersAxial-DeepLab:Stand-Alone…Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic SegmentationGenerative PretrainingFrom PixelsGenerative Pretraining From PixelsOn Robustness andTransferability of…On Robustness and Transferability of Convolutional Neural NetworksGShard: Scaling GiantModels with Conditional…GShard: Scaling Giant Models with Conditional Computation and Automatic ShardingLearning Multiple Layersof Features from Tiny…Learning Multiple Layers of Features from Tiny ImagesSelf-Supervised Learningfor Videos: A SurveySelf-Supervised Learning for Videos: A SurveyGraph ConvolutionalNetworks based on…Graph Convolutional Networks based on Manifold Learning for Semi-Supervised Image ClassificationDeep learning-basedmethods for anomaly…Deep learning-based methods for anomaly detection in video surveillance: a reviewSimViT: Exploring aSimple Vision…SimViT: Exploring a Simple Vision Transformer with Sliding WindowsA Comprehensive Surveyof Dataset DistillationA Comprehensive Survey of Dataset DistillationZero-shot ReferringImage Segmentation with…Zero-shot Referring Image Segmentation with Global-Local Context FeaturesUDepth: Fast MonocularDepth Estimation for…UDepth: Fast Monocular Depth Estimation for Visually-guided Underwater RobotsMicron-BERT: BERT-BasedFacial Micro-Expression…Micron-BERT: BERT-Based Facial Micro-Expression RecognitionOne Fits Many: ClassConfusion Loss for…One Fits Many: Class Confusion Loss for Versatile Domain AdaptationControlTraj:Controllable Trajectory…ControlTraj: Controllable Trajectory Generation with Topology-Constrained Diffusion ModelContrastiveForward-Forward: A…Contrastive Forward-Forward: A Training Algorithm of Vision TransformerLLaVA-PruMerge: AdaptiveToken Reduction for…LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal ModelsAn Image is Worth 16x16Words: Transformers for…An Image is Worth 16x16 Words: Transformers for Image Recognition at ScaleEarlier referencesFocus paperCiting papersOlderNewer

Click a node to pin it, click the empty canvas to go back to this paper, or hover to preview. Open a node’s page from its title.