MLE and MAP

We will talk about two ways of estimating a parameter: MLE and MAP. Their formulas look really similar to generative learning and discriminative as if one corresponded to generative and the other corresponded to discriminative. However, MLE/MAP has nothing to do with generative/discriminative learning. They just look similar. That’s all.

MLE and Coin Toss¶

Say we have a coin, and we want to find $\theta = P(H)$: the probability that this coin comes up heads when we toss it.

We toss it $n = 10$ times and obtain the following sequence of outcomes: $D=\{H, T, T, H, H, H, T, T, T, T\}$. Therefore, we observed $n_H=4$ heads and $n_T=6$ tails. So, intuitively,

. We will derive this more formally with Maximum Likelihood Estimation (MLE)

Maximum Likelihood Estimation (MLE)¶

The estimator we just mentioned is the Maximum Likelihood Estimate. In particular, we want to find a distribution that makes the data we observed as likely as possible. MLE is done in two steps:

1. Make an explicit modeling assumption about what type of distribution your data was sampled from.
2. Set the parameters of this distribution so that the data you observed is as likely as possible.

Before proceeding to MLE, we already observed our data. Namely, in our coin example, $D, n, n_H, n_T$ are are set values.

We will assume that coin toss is of binomial distribution $Bin(n,\theta)$, where $n$ is number of tosses we made and $\theta$ is the probability coming up head we are trying to estimate. Formally, given this is a binomial distribution with probability $\theta$, we have

MLE Principle: Find $\hat{\theta}$ to maximize $P(D; \theta)$, the likelihood of the data, :

Note we have $P(D;\theta)$ here. ; means $\theta$ is a parameter of this distribution, just like $\mu, \gamma$ are parameters in $\mathcal{N}(\mu, \gamma)$. This will be different from how we view $\theta$ in MAP we will talk later. You can still write this as $P(D\mid \theta)$, but just remember when we say “given $\theta$” in a MLE context, we don’t mean $\theta$ is a Random Variable. It should be treated as a parameter.

A common procedure we take to solve a maximum problem:

1. We don’t want to see all these products, so take the $\log$ of the function to convert them into sums.
2. Compute its derivative, and equate it with zero to find the extreme point.

Applying this procedure:

We can then solve for $\theta$ by taking the derivative and equating it with zero. This results in

Smoothing¶

If $n$ is large and your model/distribution is correct, then MLE finds the true parameters, but the MLE can overfit the data if $n$ is small. It works well when $n$ is large. For example, suppose you observe H,H,H,H,H. $\hat{\theta}_{MLE} = \frac{n_H}{n_H + n_T}= \frac{5}{5} = 1$.

Simple fix: We can add $m$ imaginary throws and get some imaginary results based on our intuition about the distribution. For example, if we have a hunch that that $\theta$ is close to 0.5. Call our intuition probability $\theta'$. The $m$ imaginary throws will result in $m\theta'$ heads and $m(1-\theta')$ tails. Add these imaginary results to our data, so

For large $n$, this is an insignificant change. For small $n$, it incorporates your “prior belief” about what $\theta$ should be.

This idea of “prior belief” actually gives rise to another class of thinking about distribution.

MAP and Coin Toss with Prior Knowledge¶

The Bayesian Way¶

• Frequentists think $\theta$ is the probability distribution from where our data is drawn. Therefore, it should be treated as a constant, something we cannot control.
• Bayesians consider it just another random variable, whose distribution can reflect our prior knowledge / assumption about the data distribution.

Formally, they model $\theta$ as a random variable, drawn from a distribution $P(\theta)$. Note that $\theta$ is not a random variable associated with an event in a sample space. You can specify a prior belief $P(\theta)$ defining what values you believe $\theta$ is likely to take on.

Now, we can look at $P(\theta \mid D) = \frac{P(D\mid \theta) P(\theta)}{P(D)}$ , where

• $P(\theta)$ is the prior distribution over the parameter(s) $\theta$ - what I believe before seeing any data.
• $P(D \mid \theta)$ is the likelihood of the data given the parameter(s) $\theta$ - same as in MLE
• $P(\theta \mid D)$ is the posterior distribution over the parameter(s) $\theta$ - what I believe after observing the data.
• $P(D)$ is the total probability of drawing such a data considering all possible distributions. $P(D) = \int_{\theta'}P(D|\theta')P(\theta')$, but this value is just a constant in our maximization problem and we can ignore it.

A natural choice for the prior $P(\theta$) is the Beta distribution

where $B(\alpha, \beta) = \frac{\Gamma(\alpha) \Gamma(\beta)}{\Gamma(\alpha+\beta)}$ is the normalization constant. Why do we choose this complicated thing as our prior belief?

For a Beta Distribution $P(\theta) \propto \theta^{\alpha - 1}(1 - \theta)^{\beta - 1}$. $\alpha$ 和 $\beta$ 就是 $\theta$ 和 $1-\theta$ 两个方向的权重, relatively

• $\alpha \gt \beta: \theta \to 1$
• $\beta \gt \alpha : \theta \to 0$
• $\beta = \alpha : \theta \to 0.5$

假设 $\alpha =3, \beta=1$, $P(\theta) = \propto \theta^{2}(1 - \theta)^{0} = \theta^{2}$. 观察到 $\theta$ 越大 $P(\theta)$ 越大，即越大的 $\theta$ 越容易被取到

Maximum a Posteriori Probability Estimation (MAP)¶

MAP Principle: Find $\hat{\theta}$ that maximizes the posterior distribution $P(\theta \mid D)$:

For out coin flipping scenario, we get:

A similar calculation will give us

We notice:

• The MAP estimate is identical to MLE with $\alpha-1$ hallucinated heads and $\beta-1$ hallucinated tails
• As $n \rightarrow \infty$, $\hat\theta_{MAP} \rightarrow \hat\theta_{MLE}$ as our prior knowledge $\alpha-1$ and $\beta-1$ become irrelevant compared to very large $n_H,n_T$.
• MAP is a great estimator if an accurate prior belief is available (and mathematically tractable).

“True” Bayesian approach¶

Note that MAP is only one way to get an estimator in Bayesian way. There is much more information in $P(\theta \mid D)$ and we threw away most of them by taking only the max. A true Bayesian approach is to use the posterior predictive distribution $P(\theta\mid D)$ directly to make prediction about the label $Y$ of a test sample $X$ based on dataset $D$. Simply put, it integrates over all possible models $\theta$ to make a prediction. Mathematically, we can represent it as:

Intuition behind each step is:

Unfortunately, the above is generally intractable.

Summary¶

• MLE Prediction: $P[(y|x_t);\theta]$ Learning: $\theta=\operatorname*{argmax}_\theta P(D;\theta)$. Here $\theta$ is purely a model parameter.
• MAP Prediction: $P[y|(x_t,\theta)]$ Learning: $\theta=\operatorname*{argmax}_\theta P(\theta|D)\propto P(D \mid \theta) P(\theta)$. Here $\theta$ is a random variable.
• “True Bayesian” Prediction: $P(y|x_t,D)=\int_{\theta}P(y|\theta)P(\theta|D)d\theta$. Here $\theta$ is integrated out - our prediction takes all possible models into account.

As always the differences are subtle. In MLE we maximize $\log\left[P(D;\theta)\right]$ in MAP we maximize $\log\left[P(D|\theta)\right]+\log\left[P(\theta)\right]$. So essentially in MAP we only add the term $\log\left[P(\theta)\right]$ to our optimization. This term is independent of the data and penalizes if the parameters, $\theta$ deviate too much from what we believe is reasonable.