Expectation-Maximization as lower bound maximization

Abstract

The Expectation-Maximization algorithm given by Dempster et al (1977) has enjoyed considerable popularity for solving MAP estimation problems. This note derives EM from the lower bounding viewpoint (Luttrell, 1994), which better illustrates the convergence properties of the algorithm and its variants. The algorithm is illustrated with two examples: pooling data from multiple noisy sources and fitting a mixture density. 1 Introduction The Expectation-Maximization (EM) algorithm is an iterative optimization technique specifically designed for probabilistic models. It uses a different strategy than gradient descent or Newton's method and sometimes provides faster convergence. However, it is still a local technique, and so is just as susceptible to local minima. The difference between EM and gradient descent is illustrated in figure 1. Starting from the current guess, gradient descent makes a linear approximation to the objective function, then takes some step uphill. Unfortunately,...