litlas
ログインアカウント作成

Stronger Generalization Bounds for Deep Nets via a Compression Approach

著者: Sanjeev Arora, Rong Ge, Behnam Neyshabur, Yi Zhang - International Conference on Machine Learning, ICML 2018 被引用: 714

Deep nets generalize well despite having more parameters than the number of training samples. Recent works try to give an explanation using PAC-Bayes and Margin-based analyses, but do not as yet result in sample complexity bounds better than naive parameter counting. The current paper shows generalization bounds that're orders of magnitude better in practice. These rely upon new succinct reparametrizations of the trained net --- a compression that is explicit and efficient. These yield generalization bounds via a simple compression-based framework introduced here. Our results also provide some theoretical justification for widespread empirical success in compressing deep nets. Analysis of correctness of our compression relies upon some newly identified \textquotedblleft noise stability\textquotedblright properties of trained deep nets, which are also experimentally verified. The study of these properties and resulting generalization bounds are also extended to convolutional nets, which had eluded earlier attempts on proving generalization.

Path-SGD:Path-Normalized…Path-SGD: Path-Normalized Optimization in Deep Neural NetworksVery Deep ConvolutionalNetworks for Large-Scal…Very Deep Convolutional Networks for Large-Scale Image RecognitionA PAC-Bayesian Approachto Spectrally-Normalize…A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural NetworksSpectrally-normalizedmargin bounds for neura…Spectrally-normalized margin bounds for neural networksComputing NonvacuousGeneralization Bounds…Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training DataFisher-Rao Metric,Geometry, and Complexit…Fisher-Rao Metric, Geometry, and Complexity of Neural NetworksSharp Minima CanGeneralize For Deep NetsSharp Minima Can Generalize For Deep NetsA Closer Look atMemorization in Deep…A Closer Look at Memorization in Deep NetworksOn Large-Batch Trainingfor Deep Learning…On Large-Batch Training for Deep Learning: Generalization Gap and Sharp MinimaSize-Independent SampleComplexity of Neural…Size-Independent Sample Complexity of Neural NetworksOn the importance ofsingle directions for…On the importance of single directions for generalizationGeneralization with DeepLearning: For…Generalization with Deep Learning: For Improvement on Sensing CapabilityPredicting theGeneralization Gap in…Predicting the Generalization Gap in Deep Networks with Margin DistributionsLearningOverparameterized Neura…Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured DataOn TighterGeneralization Bound fo…On Tighter Generalization Bound for Deep Neural Networks: CNNs, ResNets, and BeyondFine-Grained Analysis ofOptimization and…Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural NetworksRegularization Matters:Generalization and…Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced KernelGeneralizationGuarantees for Neural…Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the JacobianLearning andGeneralization in…Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two LayersFantastic GeneralizationMeasures and Where to…Fantastic Generalization Measures and Where to Find ThemQuantifying thegeneralization error in…Quantifying the generalization error in deep learning in terms of data distribution and neural network smoothnessImproved SampleComplexities for Deep…Improved Sample Complexities for Deep Neural Networks and Robust Classification via an All-Layer MarginGeneralization ErrorBounds of Gradient…Generalization Error Bounds of Gradient Descent for Learning Over-Parameterized Deep ReLU NetworksTheGeneralization-Stabilit…The Generalization-Stability Tradeoff In Neural Network PruningStronger GeneralizationBounds for Deep Nets vi…Stronger Generalization Bounds for Deep Nets via a Compression Approach過去の参考文献中心の論文この論文を引用する論文古い新しい

ノードをクリックするとフォーカスを固定、空白をクリックすると本論文に戻ります。ホバーで一時的にプレビューできます。各ノードのページはタイトルから開けます。