Understanding and Mitigating the Tradeoff Between Robustness and\n Accuracy

Abstract

Adversarial training augments the training set with perturbations to improve\nthe robust error (over worst-case perturbations), but it often leads to an\nincrease in the standard error (on unperturbed test inputs). Previous\nexplanations for this tradeoff rely on the assumption that no predictor in the\nhypothesis class has low standard and robust error. In this work, we precisely\ncharacterize the effect of augmentation on the standard error in linear\nregression when the optimal linear predictor has zero standard and robust\nerror. In particular, we show that the standard error could increase even when\nthe augmented perturbations have noiseless observations from the optimal linear\npredictor. We then prove that the recently proposed robust self-training (RST)\nestimator improves robust error without sacrificing standard error for\nnoiseless linear regression. Empirically, for neural networks, we find that RST\nwith different adversarial training methods improves both standard and robust\nerror for random and adversarial rotations and adversarial $\\ell_\\infty$\nperturbations in CIFAR-10.\n