Kullback–Leibler (KL) divergence of two Gaussians

In variational inference, we usually assume variational distribution of posterior distribution to be Gaussian. In the end, in many contexts, we may get to compute KL divergence of two Gaussians and this is tractable (in the process of maximizing variational lower bound). Note that KL divergence is a measure of how the distributions are similar. In this article, we prove the computation of KL divergence of two Gaussians is indeed tractable.

Assume two random variables that follow different Gaussian distributions.

$$p(x)=\mathcal{N}(\mu_1, \sigma_1^2) \\ q(x)=\mathcal{N}(\mu_2, \sigma_2^2)$$

Note that the assumption can be relaxed to random vectors with multivariate Gaussian distributions. Then following procedure of computing KL divergence of $p(x)$ and $q(x)$ is done.

$$\begin{align*} KL[p(x)||q(x)] &= \int p(x) \log \frac{p(x)}{q(x)} \, dx \\ &= \int p(x) \big\{-\frac{1}{2}\log(2\pi\sigma_1^2)-\frac{1}{2\sigma_1^2}(x-\mu_1)^2 + \frac{1}{2}\log(2\pi\sigma_2^2)+\frac{1}{2\sigma_2^2}(x-\mu_2)^2\big\} \\ &= \log\frac{\sigma_2}{\sigma_1} - \frac{1}{2\sigma_1^2}\mathbb{E}_p[(x-\mu_1)^2] + \frac{1}{2\sigma_2^2}\mathbb{E}_p[(x-\mu_2)^2] \\ &= \log\frac{\sigma_2}{\sigma_1} + \frac{\sigma_1^2+(\mu_1-\mu_2)^2}{2\sigma_2^2} - \frac{1}{2} \end{align*}$$