Optimizing with constraints:
reparametrization and geometry.

Mon, 14 Sep 2020 ⊕Category: optimization

Some of the most popular strategies for handling constraints in gradient-based optimization, namely: reparametrization, projected gradient, and mirror descent, while seemingly very different, are deeply connected. In this post, we will explore these connections. In particular, we show that mirror descent is equivalent to gradient descent on a reparametrized objective with straight-through gradients.