litlas
ログインアカウント作成

VeRA: Vector-based Random Matrix Adaptation

著者: Dawid Jan Kopiczko, Tijmen Blankevoort, Yuki M. Asano - International Conference on Learning Representations, ICLR 2024 被引用: 346

Low-rank adapation (LoRA) is a popular method that reduces the number of trainable parameters when finetuning large language models, but still faces acute storage challenges when scaling to even larger models or deploying numerous per-user or per-task adapted models. In this work, we present Vector-based Random Matrix Adaptation (VeRA), which significantly reduces the number of trainable parameters compared to LoRA, yet maintains the same performance. It achieves this by using a single pair of low-rank matrices shared across all layers and learning small scaling vectors instead. We demonstrate its effectiveness on the GLUE and E2E benchmarks, image classification tasks, and show its application in instruction-tuning of 7B and 13B language models.

ImageNet: A large-scalehierarchical image…ImageNet: A large-scale hierarchical image databaseDelving Deep intoRectifiers: Surpassing…Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet ClassificationThe E2E Dataset: NewChallenges For…The E2E Dataset: New Challenges For End-to-End GenerationExploring VersatileGenerative Language…Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer LearningWhat's Hidden in aRandomly Weighted Neura…What's Hidden in a Randomly Weighted Neural Network?AdapterDrop: On theEfficiency of Adapters…AdapterDrop: On the Efficiency of Adapters in TransformersIntrinsic DimensionalityExplains the…Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-TuningP-Tuning v2: PromptTuning Can Be Comparabl…P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and TasksLoRA-FA:Memory-efficient…LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuningDyLoRA: ParameterEfficient Tuning of…DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank AdaptationQLoRA: EfficientFinetuning of Quantized…QLoRA: Efficient Finetuning of Quantized LLMsDelta-LoRA: Fine-TuningHigh-Rank Parameters…Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank MatricesLoRA-FA:Memory-efficient…LoRA-FA: Memory-efficient Low-rank Adaptation for Large Language Models Fine-tuningTied-LoRA: Enhancingparameter efficiency of…Tied-LoRA: Enhancing parameter efficiency of LoRA with Weight TyingLoRA+: Efficient LowRank Adaptation of Larg…LoRA+: Efficient Low Rank Adaptation of Large ModelsMixLoRA: Enhancing LargeLanguage Models…MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA based Mixture of ExpertsMoRA: High-Rank Updatingfor Parameter-Efficient…MoRA: High-Rank Updating for Parameter-Efficient Fine-TuningParameter-EfficientFine-Tuning for Large…Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive SurveyThe Impact ofInitialization on LoRA…The Impact of Initialization on LoRA Finetuning DynamicsShareLoRA: ParameterEfficient and Robust…ShareLoRA: Parameter Efficient and Robust Large Language Model Fine-tuning via Shared Low-Rank AdaptationA Survey on LoRA ofLarge Language ModelsA Survey on LoRA of Large Language ModelsBone: Block AffineTransformation as…Bone: Block Affine Transformation as Parameter Efficient Fine-tuning Methods for Large Language ModelsRandom Masking FindsWinning Tickets for…Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningLoRA-XS: Low-RankAdaptation with…LoRA-XS: Low-Rank Adaptation with Extremely Small Number of ParametersVeRA: Vector-basedRandom Matrix AdaptationVeRA: Vector-based Random Matrix Adaptation過去の参考文献中心の論文この論文を引用する論文古い新しい

ノードをクリックするとフォーカスを固定、空白をクリックすると本論文に戻ります。ホバーで一時的にプレビューできます。各ノードのページはタイトルから開けます。