License: Creative Commons Attribution 3.0 Unported license (CC BY 3.0)
When quoting this document, please refer to the following
DOI: 10.4230/LIPIcs.ITCS.2017.2
URN: urn:nbn:de:0030-drops-81640
URL: http://dagstuhl.sunsite.rwth-aachen.de/volltexte/2017/8164/
Go to the corresponding LIPIcs Volume Portal

Panageas, Ioannis ; Piliouras, Georgios

Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions

pdf-format:
LIPIcs-ITCS-2017-2.pdf (0.7 MB)

Abstract

Given a twice continuously differentiable cost function f, we prove that the set of initial conditions so that gradient descent converges to saddle points where \nabla^2 f has at least one strictly negative eigenvalue, has (Lebesgue) measure zero, even for cost functions f with non-isolated critical points, answering an open question in [Lee, Simchowitz, Jordan, Recht, COLT 2016]. Moreover, this result extends to forward-invariant convex subspaces, allowing for weak (non-globally Lipschitz) smoothness assumptions. Finally, we produce an upper bound on the allowable step-size.

BibTeX - Entry

@InProceedings{panageas_et_al:LIPIcs:2017:8164,
  author =	{Ioannis Panageas and Georgios Piliouras},
  title =	{{Gradient Descent Only Converges to Minimizers: Non-Isolated Critical Points and Invariant Regions}},
  booktitle =	{8th Innovations in Theoretical Computer Science Conference (ITCS 2017)},
  pages =	{2:1--2:12},
  series =	{Leibniz International Proceedings in Informatics (LIPIcs)},
  ISBN =	{978-3-95977-029-3},
  ISSN =	{1868-8969},
  year =	{2017},
  volume =	{67},
  editor =	{Christos H. Papadimitriou},
  publisher =	{Schloss Dagstuhl--Leibniz-Zentrum fuer Informatik},
  address =	{Dagstuhl, Germany},
  URL =		{http://drops.dagstuhl.de/opus/volltexte/2017/8164},
  URN =		{urn:nbn:de:0030-drops-81640},
  doi =		{10.4230/LIPIcs.ITCS.2017.2},
  annote =	{Keywords: Gradient Descent, Center-stable manifold, Saddle points, Hessian}
}

Keywords: Gradient Descent, Center-stable manifold, Saddle points, Hessian
Collection: 8th Innovations in Theoretical Computer Science Conference (ITCS 2017)
Issue Date: 2017
Date of publication: 28.11.2017

DROPS-Home | Fulltext Search | Imprint | Privacy Published by LZI