litlas
Log inCreate account

TÜLU 3: Pushing Frontiers in Open Language Model Post-Training

Authors: Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, Hannaneh Hajishirzi - arXiv 2024 cited by 858

Language model post-training is applied to refine behaviors and unlock new skills across a wide range of recent language models, but open recipes for applying these techniques lag behind proprietary ones. The underlying training data and recipes for post-training are simultaneously the most important pieces of the puzzle and the portion with the least transparency. To bridge this gap, we introduce Tulu 3, a family of fully-open state-of-the-art post-trained models, alongside its data, code, and training recipes, serving as a comprehensive guide for modern post-training techniques. Tulu 3, which builds on Llama 3.1 base models, achieves results surpassing the instruct versions of Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and Claude 3.5-Haiku. The training algorithms for our models include supervised finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we call Reinforcement Learning with Verifiable Rewards (RLVR). With Tulu 3, we introduce a multi-task evaluation scheme for post-training recipes with development and unseen evaluations, standard benchmark implementations, and substantial decontamination of existing open datasets on said benchmarks. We conclude with analysis and discussion of training methods that did not reliably improve performance. In addition to the Tulu 3 model weights and demo, we release the complete recipe -- including datasets for diverse core skills, a robust toolkit for data curation and evaluation, the training code and infrastructure, and, most importantly, a detailed report for reproducing and further adapting the Tulu 3 approach to more domains.

Llama 2: Open Foundationand Fine-Tuned Chat…Llama 2: Open Foundation and Fine-Tuned Chat ModelsInstruction-FollowingEvaluation for Large…Instruction-Following Evaluation for Large Language ModelsCamels in a ChangingClimate: Enhancing LM…Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2DeepSeek-V3 TechnicalReportDeepSeek-V3 Technical ReportMMLU-Pro: A More Robustand Challenging…MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding BenchmarkDeepSeekMath: Pushingthe Limits of…DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsUltraFeedback: BoostingLanguage Models with…UltraFeedback: Boosting Language Models with High-quality FeedbackThe Llama 3 Herd ofModelsThe Llama 3 Herd of ModelsSimPO: Simple PreferenceOptimization with a…SimPO: Simple Preference Optimization with a Reference-Free RewardUnpacking DPO and PPO:Disentangling Best…Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference FeedbackNemotron-4 340BTechnical ReportNemotron-4 340B Technical ReportHybrid Preferences:Learning to Route…Hybrid Preferences: Learning to Route Instances for Human vs. AI FeedbackInternLM-XComposer2.5-Reward:A Simple Yet Effective…InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward ModelAbstentionBench:Reasoning LLMs Fail on…AbstentionBench: Reasoning LLMs Fail on Unanswerable QuestionsSmall Models Struggle toLearn from Strong…Small Models Struggle to Learn from Strong ReasonersReason-RFT:Reinforcement…Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language ModelsThe Avengers: A SimpleRecipe for Uniting…The Avengers: A Simple Recipe for Uniting Smaller Language Models to Challenge Proprietary GiantsThe SurprisingEffectiveness of…The Surprising Effectiveness of Negative Reinforcement in LLM ReasoningWhy is constrainedneural language…Why is constrained neural language generation particularly challenging?Training on the TestTask Confounds…Training on the Test Task Confounds Evaluation and EmergenceEvaluating the Diversityand Quality of LLM…Evaluating the Diversity and Quality of LLM Generated ContentReward Reasoning ModelReward Reasoning ModelMulti-Step Reasoningwith Large Language…Multi-Step Reasoning with Large Language Models, a SurveyMetaScale: Test-TimeScaling with Evolving…MetaScale: Test-Time Scaling with Evolving Meta-ThoughtsTÜLU 3: PushingFrontiers in Open…TÜLU 3: Pushing Frontiers in Open Language Model Post-TrainingEarlier referencesFocus paperCiting papersOlderNewer

Click a node to pin it, click the empty canvas to go back to this paper, or hover to preview. Open a node’s page from its title.