Do generative adversarial networks always converge?

In terms of theory, this is an open question.

In terms of practice, no they don’t always converge. On small problems, they sometimes converge and sometimes don’t. On large problems, like modeling ImageNet at 128x128 resolution, I’ve never seen them converge yet.

This is probably the most important question about GANs, both in terms of theory and practice. In terms of theory, it would be great to derive a set of conditions under which they converge or don’t converge. In terms of practice, it would be great to modify them in a way that makes them converge consistently.

This paper gives some conditions under which simultaneous gradient descent on two player’s costs will converge: http://robotics.eecs.berkeley.ed...

GANs never satisfy those conditions because the Hessian of the generators costs is all zeros at equilibrium. However, the conditions in this paper are sufficient conditions, not necessary conditions, so it’s possible that GANs can converge anyway.