Faster Acceleration for Steepest Descent

Abstract

Recent advances (Sherman2017Area; sidford2018coordinate; cohen2021relative) have overcome the fundamental barrier of dimension dependence in the iteration complexity of solving $\ell_{\infty}$ regression with first-order methods. Yet it remains unclear to what extent such acceleration can be achieved for general $\ell_{p}$ smooth functions. In this paper, we propose a new accelerated first-order method for convex optimization under non-Euclidean smoothness assumptions. In contrast to standard acceleration techniques, our approach uses primal-dual iterate sequences taken with respect to differing norms, which are then coupled using an implicitly determined interpolation parameter. For $\ell_{p}$ norm smooth problems in $d$ dimensions, our method provides an iteration complexity improvement of up to $O(d^{1-\frac{2}{p}})$ in terms of calls to a first-order oracle, thereby allowing us to circumvent long-standing barriers in accelerated non-Euclidean steepest descent.