Contents

Evidence lower bound

Lower bound on the log-likelihood of some observed data

Part of a series on
Bayesian statistics
Posterior = Likelihood × Prior ÷ Evidence
Background
Bayesian inference Bayesian probability Bayes' theorem Bernstein–von Mises theorem Coherence Cox's theorem Cromwell's rule Principle of indifference Principle of maximum entropy
Model building
Weak prior ... Strong prior Conjugate prior Linear regression Empirical Bayes Hierarchical model
Posterior approximation
Markov chain Monte Carlo Laplace's approximation Integrated nested Laplace approximations Variational inference Approximate Bayesian computation
Estimators
Bayesian estimator Credible interval Maximum a posteriori estimation
Evidence approximation
Evidence lower bound Nested sampling
Model evaluation
Bayes factor Model averaging Posterior predictive
Mathematics portal
v t e

Invariational Bayesian methods, the evidence lower bound (often abbreviated ELBO, also sometimes called the variational lower bound^[1]ornegative variational free energy) is a useful lower bound on the log-likelihood of some observed data.

The ELBO is useful because it provides a guarantee on the worst-case for the log-likelihood of some distribution (e.g. ${\displaystyle p($X)} ) which models a set of data. The actual log-likelihood may be higher (indicating an even better fit to the distribution) because the ELBO includes a Kullback-Leibler divergence (KL divergence) term which decreases the ELBO due to an internal part of the model being inaccurate despite good fit of the model overall. Thus improving the ELBO score indicates either improving the likelihood of the model ${\displaystyle p($X)} or the fit of a component internal to the model, or both, and the ELBO score makes a good loss function, e.g., for training a deep neural network to improve both the model overall and the internal component. (The internal component is ${\displaystyle q_{\phi }(\cdot |x)}$, defined in detail later in this article.)

Definition[edit]

Let ${\displaystyle X}$ and ${\displaystyle Z}$berandom variables, jointly distributed with distribution ${\displaystyle p_{\theta }}$. For example, ${\displaystyle p_{\theta }($X)} is the marginal distributionof${\displaystyle X}$, and ${\displaystyle p_{\theta }(Z\mid X)}$ is the conditional distributionof${\displaystyle Z}$ given ${\displaystyle X}$. Then, for a sample ${\displaystyle x\sim p_{\theta }}$, and any distribution ${\displaystyle q_{\phi }}$, the ELBO is defined as

In the first line, ${\displaystyle H[q_{\phi }(z|x)]}$ is the entropyof${\displaystyle q_{\phi }}$, which relates the ELBO to the Helmholtz free energy.^[3] In the second line, ${\displaystyle \ln p_{\theta }($x)} is called the evidence for ${\displaystyle x}$, and ${\displaystyle D_{KL}(q_{\phi }(z|x)||p_{\theta }(z|x))}$ is the Kullback-Leibler divergence between ${\displaystyle q_{\phi }}$ and ${\displaystyle p_{\theta }}$. Since the Kullback-Leibler divergence is non-negative, ${\displaystyle L(\phi ,\theta ;x)}$ forms a lower bound on the evidence (ELBO inequality)

Motivation[edit]

Variational Bayesian inference[edit]

Suppose we have an observable random variable ${\displaystyle X}$, and we want to find its true distribution ${\displaystyle p^{*}}$. This would allow us to generate data by sampling, and estimate probabilities of future events. In general, it is impossible to find ${\displaystyle p^{*}}$ exactly, forcing us to search for a good approximation.

That is, we define a sufficiently large parametric family ${\displaystyle \{p_{\theta }\}_{\theta \in \Theta }}$ of distributions, then solve for ${\displaystyle \min _{\theta }L(p_{\theta },p^{*})}$ for some loss function ${\displaystyle L}$. One possible way to solve this is by considering small variation from ${\displaystyle p_{\theta }}$to${\displaystyle p_{\theta +\delta \theta }}$, and solve for ${\displaystyle L(p_{\theta },p^{*})-L(p_{\theta +\delta \theta },p^{*})=0}$. This is a problem in the calculus of variations, thus it is called the variational method.

Since there are not many explicitly parametrized distribution families (all the classical distribution families, such as the normal distribution, the Gumbel distribution, etc, are far too simplistic to model the true distribution), we consider implicitly parametrized probability distributions:

• First, define a simple distribution ${\displaystyle p($z)} over a latent random variable ${\displaystyle Z}$. Usually a normal distribution or a uniform distribution suffices.
• Next, define a family of complicated functions ${\displaystyle f_{\theta }}$ (such as a deep neural network) parametrized by ${\displaystyle \theta }$.
• Finally, define a way to convert any ${\displaystyle f_{\theta }($z)} into a simple distribution over the observable random variable ${\displaystyle X}$. For example, let ${\displaystyle f_{\theta }($z)=(f_{1}(z),f_{2}(z))} have two outputs, then we can define the corresponding distribution over ${\displaystyle X}$ to be the normal distribution ${\displaystyle {\mathcal {N}}(f_{1}($z),e^{f_{2}(z)})} .

This defines a family of joint distributions ${\displaystyle p_{\theta }}$ over ${\displaystyle (X,Z)}$. It is very easy to sample ${\displaystyle (x,z)\sim p_{\theta }}$: simply sample ${\displaystyle z\sim p}$, then compute ${\displaystyle f_{\theta }($z)} , and finally sample ${\displaystyle x\sim p_{\theta }(\cdot |z)}$ using ${\displaystyle f_{\theta }($z)} .

In other words, we have a generative model for both the observable and the latent. Now, we consider a distribution ${\displaystyle p_{\theta }}$ good, if it is a close approximation of ${\displaystyle p^{*}}$:

${\displaystyle X}$

${\displaystyle Z}$

${\displaystyle p_{\theta }($x)=\int p_{\theta }(x|z)p(z)dz}

Since ${\displaystyle p_{\theta }($x)={\frac {p_{\theta }(x|z)p(z)}{p_{\theta }(z|x)}}} (Bayes' Rule), it suffices to find a good approximation of ${\displaystyle p_{\theta }(z|x)}$. So define another distribution family ${\displaystyle q_{\phi }(z|x)}$ and use it to approximate ${\displaystyle p_{\theta }(z|x)}$. This is a discriminative model for the latent.

The entire situation is summarized in the following table:

${\displaystyle X}$: observable	${\displaystyle X,Z}$	${\displaystyle Z}$: latent
${\displaystyle p^{*}($x)\approx p_{\theta }(x)\approx {\frac {p_{\theta }(x|z)p(z)}{q_{\phi }(z|x)}}} approximable		${\displaystyle p($z)} , easy
	${\displaystyle p_{\theta }(x|z)p($z)} , easy
${\displaystyle p_{\theta }(z|x)\approx q_{\phi }(z|x)}$ approximable		${\displaystyle p_{\theta }(x|z)}$, easy

InBayesian language, ${\displaystyle X}$ is the observed evidence, and ${\displaystyle Z}$ is the latent/unobserved. The distribution ${\displaystyle p}$ over ${\displaystyle Z}$ is the prior distribution over ${\displaystyle Z}$, ${\displaystyle p_{\theta }(x|z)}$ is the likelihood function, and ${\displaystyle p_{\theta }(z|x)}$ is the posterior distribution over ${\displaystyle Z}$.

Given an observation ${\displaystyle x}$, we can infer what ${\displaystyle z}$ likely gave rise to ${\displaystyle x}$ by computing ${\displaystyle p_{\theta }(z|x)}$. The usual Bayesian method is to estimate the integral ${\displaystyle p_{\theta }($x)=\int p_{\theta }(x|z)p(z)dz} , then compute by Bayes' rule ${\displaystyle p_{\theta }(z|x)={\frac {p_{\theta }(x|z)p($z)}{p_{\theta }(x)}}} . This is expensive to perform in general, but if we can simply find a good approximation ${\displaystyle q_{\phi }(z|x)\approx p_{\theta }(z|x)}$ for most ${\displaystyle x,z}$, then we can infer ${\displaystyle z}$ from ${\displaystyle x}$ cheaply. Thus, the search for a good ${\displaystyle q_{\phi }}$ is also called amortized inference.

All in all, we have found a problem of variational Bayesian inference.

Deriving the ELBO[edit]

A basic result in variational inference is that minimizing the Kullback–Leibler divergence (KL-divergence) is equivalent to maximizing the log-likelihood:

${\displaystyle H(p^{*})=-\mathbb {\mathbb {E} } _{x\sim p^{*}}[\ln p^{*}($x)]}

${\displaystyle \mathbb {E} _{x\sim p^{*}($x)}[\ln p_{\theta }(x)]}

${\displaystyle D_{\mathit {KL}}(p^{*}($x)\|p_{\theta }(x))}

${\displaystyle p_{\theta }\approx p^{*}}$

To maximize ${\displaystyle \mathbb {E} _{x\sim p^{*}($x)}[\ln p_{\theta }(x)]} , we simply sample many ${\displaystyle x_{i}\sim p^{*}($x)} , i.e. use Importance sampling

${\displaystyle N}$

In order to maximize ${\displaystyle \sum _{i}\ln p_{\theta }(x_{i})}$, it's necessary to find ${\displaystyle \ln p_{\theta }($x)} :

${\displaystyle q_{\phi }(z|x)}$

${\displaystyle z}$

So we see that if we sample ${\displaystyle z\sim q_{\phi }(\cdot |x)}$, then ${\displaystyle {\frac {p_{\theta }(x,z)}{q_{\phi }(z|x)}}}$ is an unbiased estimator of ${\displaystyle p_{\theta }($x)} . Unfortunately, this does not give us an unbiased estimator of ${\displaystyle \ln p_{\theta }($x)} , because ${\displaystyle \ln }$ is nonlinear. Indeed, we have by Jensen's inequality,

${\displaystyle \ln p_{\theta }($x)}

${\displaystyle z_{i}\sim q_{\phi }(\cdot |x)}$

${\displaystyle N=1}$

Maximizing the ELBO[edit]

For fixed ${\displaystyle x}$, the optimization ${\displaystyle \max _{\theta ,\phi }L(\phi ,\theta ;x)}$ simultaneously attempts to maximize ${\displaystyle \ln p_{\theta }($x)} and minimize ${\displaystyle D_{\mathit {KL}}(q_{\phi }(\cdot |x)\|p_{\theta }(\cdot |x))}$. If the parametrization for ${\displaystyle p_{\theta }}$ and ${\displaystyle q_{\phi }}$ are flexible enough, we would obtain some ${\displaystyle {\hat {\phi }},{\hat {\theta }}}$, such that we have simultaneously

${\displaystyle p_{\hat {\theta }}\approx p^{*}}$

${\displaystyle q_{\hat {\phi }}(\cdot |x)\approx p_{\hat {\theta }}(\cdot |x)}$

Main forms[edit]

The ELBO has many possible expressions, each with some different emphasis.

$${\displaystyle \mathbb {E} _{z\sim q_{\phi }(\cdot |x)}\left[\ln {\frac {p_{\theta }(x,z)}{q_{\phi }(z|x)}}\right]=\int q_{\phi }(z|x)\ln {\frac {p_{\theta }(x,z)}{q_{\phi }(z|x)}}dz}$$

This form shows that if we sample ${\displaystyle z\sim q_{\phi }(\cdot |x)}$, then ${\displaystyle \ln {\frac {p_{\theta }(x,z)}{q_{\phi }(z|x)}}}$ is an unbiased estimator of the ELBO.

$${\displaystyle \ln p_{\theta }($$x)-D_{\mathit {KL}}(q_{\phi }(\cdot |x)\;\|\;p_{\theta }(\cdot |x))}

This form shows that the ELBO is a lower bound on the evidence ${\displaystyle \ln p_{\theta }($x)} , and that maximizing the ELBO with respect to ${\displaystyle \phi }$ is equivalent to minimizing the KL-divergence from ${\displaystyle p_{\theta }(\cdot |x)}$to${\displaystyle q_{\phi }(\cdot |x)}$.

$${\displaystyle \mathbb {E} _{z\sim q_{\phi }(\cdot |x)}[\ln p_{\theta }(x|z)]-D_{\mathit {KL}}(q_{\phi }(\cdot |x)\;\|\;p)}$$

This form shows that maximizing the ELBO simultaneously attempts to keep ${\displaystyle q_{\phi }(\cdot |x)}$ close to ${\displaystyle p}$ and concentrate ${\displaystyle q_{\phi }(\cdot |x)}$ on those ${\displaystyle z}$ that maximizes ${\displaystyle \ln p_{\theta }(x|z)}$. That is, the approximate posterior ${\displaystyle q_{\phi }(\cdot |x)}$ balances between staying close to the prior ${\displaystyle p}$ and moving towards the maximum likelihood ${\displaystyle \arg \max _{z}\ln p_{\theta }(x|z)}$.

$${\displaystyle H(q_{\phi }(\cdot |x))+\mathbb {E} _{z\sim q(\cdot |x)}[\ln p_{\theta }(z|x)]+\ln p_{\theta }($$x)}

This form shows that maximizing the ELBO simultaneously attempts to keep the entropy of ${\displaystyle q_{\phi }(\cdot |x)}$ high, and concentrate ${\displaystyle q_{\phi }(\cdot |x)}$ on those ${\displaystyle z}$ that maximizes ${\displaystyle \ln p_{\theta }(z|x)}$. That is, the approximate posterior ${\displaystyle q_{\phi }(\cdot |x)}$ balances between being a uniform distribution and moving towards the maximum a posteriori ${\displaystyle \arg \max _{z}\ln p_{\theta }(z|x)}$.

Data-processing inequality[edit]

Suppose we take ${\displaystyle N}$ independent samples from ${\displaystyle p^{*}}$, and collect them in the dataset ${\displaystyle D=\{x_{1},...,x_{N}\}}$, then we have empirical distribution ${\displaystyle q_{D}($x)={\frac {1}{N}}\sum _{i}\delta _{x_{i}}} .

Fitting ${\displaystyle p_{\theta }($x)} to${\displaystyle q_{D}($x)} can be done, as usual, by maximizing the loglikelihood ${\displaystyle \ln p_{\theta }($D)} :

${\displaystyle \ln p_{\theta }($D)}

In this interpretation, maximizing ${\displaystyle L(\phi ,\theta ;D)=\sum _{i}L(\phi ,\theta ;x_{i})}$ is minimizing ${\displaystyle D_{\mathit {KL}}(q_{D,\phi }(x,z);p_{\theta }(x,z))}$, which upper-bounds the real quantity of interest ${\displaystyle D_{\mathit {KL}}(q_{D}($x);p_{\theta }(x))} via the data-processing inequality. That is, we append a latent space to the observable space, paying the price of a weaker inequality for the sake of more computationally efficient minimization of the KL-divergence.^[6]

References[edit]

Notes[edit]

1. ^ In fact, by Jensen's inequality, ${\displaystyle \mathbb {E} _{x\sim p^{*}($x)}\left[\max _{\theta }\sum _{i}\ln p_{\theta }(x_{i})\right]\geq \max _{\theta }\mathbb {E} _{x\sim p^{*}(x)}\left[\sum _{i}\ln p_{\theta }(x_{i})\right]=N\max _{\theta }\mathbb {E} _{x\sim p^{*}(x)}[\ln p_{\theta }(x)]} The estimator is biased upwards. This can be seen as overfitting: for some finite set of sampled data ${\displaystyle x_{i}}$ , there is usually some ${\displaystyle \theta }$ that fits them better than the entire ${\displaystyle p^{*}}$ distribution.

^ By the delta method, we have

$${\displaystyle \mathbb {E} _{z_{i}\sim q_{\phi }(\cdot |x)}\left[\ln \left({\frac {1}{N}}\sum _{i}{\frac {p_{\theta }(z_{i}|x)}{q_{\phi }(z_{i}|x)}}\right)\right]\approx -{\frac {1}{2N}}\mathbb {V} _{z\sim q_{\phi }(\cdot |x)}\left[{\frac {p_{\theta }(z|x)}{q_{\phi }(z|x)}}\right]=O(N^{-1})}$$