Probabilistic Linear Regression
We can model $Y$ to be a linear function as before, with an additional noise parameter $\epsilon \sim \mathcal{N}(0,\sigma^2)$. That is, the relation is
Normal distribution has the maximum entropy amongst all distributions with a given variance. The $3-\sigma$ rule also plays a vital role for picking this. That is, $68\%$ of readings deviate less than $\sigma$ from the mean, $95\%$ less than $2\sigma$ and $99.7\%$ less than $3\sigma$ from the mean.
The maximum likelihood estimate of $W$ is calculated in this case. (Remember that calculation is simplified by taking $\log$ to get the Log-Likelihood)
Overfitting and Regularization
Increasing the degrees of freedom causes overfitting. The model essentially brute-forces all data points in training data, which isn’t very helpful. Overfitting is caused by $\vert\vert W_i\vert\vert$ being large, and regularization tries to suppress this.
Bayesian Linear Regression
The problem of overfitting is addressed by a prior. That is, the prior models $W$ by bounding the norm to be smaller than some $\Theta$.
To understand this better, we shall first tackle a simpler coin-tossing example.
I have a newly minted coin which I believe to be fair. I now flip it four times and get 4 heads. The MLE would be 1, but this isn’t taking the prior belief into account. The posterior would be given by the baye’s rule.
The Prior is given by $P(H)$, and the Posterior is given by $P(H\vert D)$. Therefore, the relation is given by:
We ignore the denominator as it is just a normalizing factor, and is constant as the data is known.
If $P(D\vert H)$ follows distribution $d_1$ and the posterior and prior follow the same distribution $d_2$, then $d_2$ is said to be the conjugate prior of $d_1$.
Examples for distribution-conjugate prior pairs include:
- Gaussian - Gaussian
- Bernoulli & Binomial - Beta
- Categorical & Multinomial - Dirchlet
Beta Distribution
Wikipedia page link: Beta distribution
In short, the beta distribution has two parameters $\alpha, \beta$. The value of the distribution always lies between 0 and 1. The pdf is given by:
The advantage of conjugate distributions is that the posterior is always of the same family as the prior, with just the parameters changed. This makes calculations very easy.
In the above coin tossing example, we know that Bernoulli and Beta are conjuagtes. Therefore, we can model the prior of $p$ as a BETA distribution $\text{Beta}(\alpha, \beta)$ for simplicity.
If there are $h$ heads occurring in $n$ tosses, then the posterior distribution turns out to be $\text{Beta}(\alpha+h, \beta+n-h)$.
If we have no prior information regarding $p$, we can model it as a beta with both parameters equal to 1. (The beta is uniform in 0 to 1 when both parameters are 1, just what we intend to have)