Ensemble methods research
paperAuthors
Credibility Rating
Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: arXiv
Relevant to AI safety as a technique for improving model robustness and uncertainty calibration under distribution shift, which are important properties for reliable and safe deployment of ML systems.
Paper Details
Metadata
Abstract
Modern deep neural networks can achieve high accuracy when the training distribution and test distribution are identically distributed, but this assumption is frequently violated in practice. When the train and test distributions are mismatched, accuracy can plummet. Currently there are few techniques that improve robustness to unforeseen data shifts encountered during deployment. In this work, we propose a technique to improve the robustness and uncertainty estimates of image classifiers. We propose AugMix, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unforeseen corruptions. AugMix significantly improves robustness and uncertainty measures on challenging image classification benchmarks, closing the gap between previous methods and the best possible performance in some cases by more than half.
Summary
AugMix is a data augmentation technique that improves deep neural network robustness to distribution shift by mixing augmented image versions and applying a consistency loss. It significantly improves corruption robustness and uncertainty calibration on ImageNet-C and CIFAR-10-C benchmarks with minimal computational overhead.
Key Points
- •Addresses distribution shift between training and test data, which causes accuracy to drop from 22% to 64% error on ImageNet-C benchmarks
- •AugMix stochastically mixes multiple augmented versions of an image and enforces consistency via Jensen-Shannon divergence loss
- •Improves both robustness to corruptions and uncertainty calibration, closing the gap to optimal performance by over half in some cases
- •Simple to implement with limited computational overhead, making it practical for real-world deployment scenarios
- •Demonstrates that training against specific corruptions leads to memorization; AugMix generalizes to unseen corruptions instead
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Corrigibility Failure Pathways | Analysis | 62.0 |
Cached Content Preview
# AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
Dan Hendrycks
DeepMind
[hendrycks@berkeley.edu](mailto:hendrycks@berkeley.edu "")
&Norman Mu11footnotemark: 1
Google
[normanmu@google.com](mailto:normanmu@google.com "")
&Ekin D. Cubuk
Google
[cubuk@google.com](mailto:cubuk@google.com "")
&Barret Zoph
Google
[barretzoph@google.com](mailto:barretzoph@google.com "")
&Justin Gilmer
Google
[gilmer@google.com](mailto:gilmer@google.com "")
&Balaji Lakshminarayanan
DeepMind
[balajiln@google.com](mailto:balajiln@google.com "")Equal Contribution.Corresponding author.
###### Abstract
Modern deep neural networks can achieve high accuracy when the training distribution and test distribution are identically distributed, but this assumption is frequently violated in practice. When the train and test distributions are mismatched, accuracy can plummet. Currently there are few techniques that improve robustness to unforeseen data shifts encountered during deployment. In this work, we propose a technique to improve the robustness and uncertainty estimates of image classifiers.
We propose AugMix, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unforeseen corruptions. AugMix significantly improves robustness and uncertainty measures on challenging image classification benchmarks, closing the gap between previous methods and the best possible performance in some cases by more than half.
## 1 Introduction
Current machine learning models depend on the ability of training data to faithfully represent the data encountered during deployment.
In practice, data distributions evolve (Lipton et al., [2018](https://ar5iv.labs.arxiv.org/html/1912.02781#bib.bib27 "")), models encounter new scenarios (Hendrycks & Gimpel, [2017](https://ar5iv.labs.arxiv.org/html/1912.02781#bib.bib18 "")), and data curation procedures may capture only a narrow slice of the underlying data distribution (Torralba & Efros, [2011](https://ar5iv.labs.arxiv.org/html/1912.02781#bib.bib38 "")).
Mismatches between the train and test data are commonplace, yet the study of this problem is not. As it stands, models do not robustly generalize across shifts in the data distribution.
If models could identify when they are likely to be mistaken, or estimate uncertainty accurately, then the impact of such fragility might be ameliorated. Unfortunately, modern models already produce overconfident predictions when the training examples are independent and identically distributed to the test distribution. This overconfidence and miscalibration is greatly exacerbated by mismatched training and testing distributions.
Small corruptions to the data distribution are enough to subvert existing classifiers, and techniques to improve corruption robustness remain few in number.
Hendrycks & Dietterich ( [2019](https://ar5iv.labs.arxiv.org/html/1912.02781#bib.bib17 "")) show that classification error of mod
... (truncated, 60 KB total)21092db06414732e | Stable ID: OTg4MjYzMj