Nonsmooth Implicit Differentation for Machine Learning
Antonio Silveti-Falls  1@  , Jerome Bolte  1  , Edouard Pauwels  2  , Tam Le  1  
1 : Toulouse School of Economics
Toulouse School of Economics, Toulouse School of Economics
2 : Institut de Recherche en Informatique de Toulouse  (IRIT)
Ministère de l'Enseignement Supérieur et de la Recherche Scientifique

In this talk we discuss a nonsmooth implicit function theorem equipped with an operational calculus that allows for its use to solve practical problems in machine learning. This calculus is special in that it is compatible with backpropagation - allowing, for instance, to replace derivatives by Clarke Jacobians in the usual differentiation formulas for a wide class of nonsmooth problems. We provide several applications such as training neural networks with implicit layers or differentiating solutions to nonsmooth optimization problems. Finally, to show the sharpness of our assumptions, we present numerical experiments showcasing the extremely pathological gradient dynamics one can encounter when applying implicit algorithmic differentiation without the hypothesis of the theorem.

Personnes connectées : 2 Vie privée