Question

Show that the M.L.E. of \({{\bf{\sigma }}^{\bf{2}}}\) is given by Eq. (11.2.3).

Short answer

The desired MLE is obtained by substituting \({\hat \beta _0}\) and \({\hat \beta _1}\)in place of \({\beta _0}\)and \({\beta _1}\) in the given equation \({\hat \sigma ^2} = \frac{1}{n}\sum\limits_{i = 1}^n {{{\left( {{y_i} - {{\hat \beta }_0} - {{\hat \beta }_1}{x_i}} \right)}^2}} \).

Step-by-step solution

Show that the MLE of \({\sigma ^2}\)

Refer to equation 11.2.3 to obtain the equation,\({{\bf{\hat \sigma }}^{\bf{2}}}{\bf{ = }}\frac{{\bf{1}}}{{\bf{n}}}\sum\limits_{{\bf{i = 1}}}^{\bf{n}} {{{\left( {{{\bf{y}}_{\bf{i}}}{\bf{ - }}{{{\bf{\hat \beta }}}_{\bf{0}}}{\bf{ - }}{{{\bf{\hat \beta }}}_{\bf{1}}}{{\bf{x}}_{\bf{i}}}} \right)}^{\bf{2}}}} \).

To show that the MLE of \({\sigma ^2}\) (which is the conditional variance of \({Y_i}\)given values\({x_1}, \ldots ,{x_n}\) for all\(i \in \{ 1,2, \ldots ,n\} \)) is same as in equation 11.2.3, which is,

\({\hat \sigma ^2} = \frac{1}{n}\sum\limits_{i = 1}^n {{{\left( {{y_i} - {{\hat \beta }_0} - {{\hat \beta }_1}{x_i}} \right)}^2}} \)

Where, \({\hat \beta _0}\)and \({\hat \beta _1}\)are the MLEs of \({\beta _0}\) and\({\beta _1}\), the coefficients of linear regression.

Obtain the maximum likelihood function

MLE is a value that maximizes the likelihood function.

Refer to equation 11.2..2 for the conditional joint density of\({Y_1}, \ldots ,{Y_n}\), given\(x = \left( {{x_1}, \ldots ,{x_n}} \right)\)\({f_n}\left( {y\left| {x,{\beta _0},{\beta _1},{\sigma ^2}} \right.} \right) = \frac{1}{{{{\left( {2\pi {\sigma ^2}} \right)}^{n/2}}}}\exp \left( { - \frac{1}{{2{\sigma ^2}}} \cdot \sum\limits_{i = 1}^n {{{\left( {{y_i} - {\beta _0} - {\beta _1}{x_i}} \right)}^2}} } \right)\)

The above equation is also the likelihood function for the parameter,

\({\bf{L}}\left( {{{\bf{\sigma }}^{\bf{2}}}} \right){\bf{ = }}\frac{{\bf{1}}}{{{{\left( {{\bf{2\pi }}{{\bf{\sigma }}^{\bf{2}}}} \right)}^{{\bf{n/2}}}}}}{\bf{exp}}\left( {{\bf{ - }}\frac{{\bf{1}}}{{{\bf{2}}{{\bf{\sigma }}^{\bf{2}}}}}{\bf{ \times }}\sum\limits_{{\bf{i = 1}}}^{\bf{n}} {{{\left( {{{\bf{y}}_{\bf{i}}}{\bf{ - }}{{\bf{\beta }}_{\bf{0}}}{\bf{ - }}{{\bf{\beta }}_{\bf{1}}}{{\bf{x}}_{\bf{i}}}} \right)}^{\bf{2}}}} } \right)\)

Note that the likelihood depends only on \({\sigma ^2}\)since the MLEs of \({\beta _0}\)and \({\beta _1}\)were already found - those are the estimated parameters of linear regression, \({\hat \beta _0}\)and\({\hat \beta _1}\).

Apply natural log as follows,

\(\begin{array}{c}l\left( {{\sigma ^2}} \right) = \ln \left( {L\left( {{\sigma ^2}} \right)} \right)\\ = - \frac{n}{2}\ln \left( {2\pi {\sigma ^2}} \right) - \frac{1}{{2{\sigma ^2}}}\sum\limits_{i = 1}^n {{{\left( {{y_i} - {\beta _0} - {\beta _1}{x_i}} \right)}^2}} \end{array}\)

The derivative of \(l\) (with respect to \({\sigma ^2}\) ) is expressed as,

\(\frac{{\partial \left( {l\left( {{\sigma ^2}} \right)} \right)}}{{\partial \left( {{\sigma ^2}} \right)}} = - \frac{n}{{2{\sigma ^2}}} + \frac{1}{{2{\sigma ^4}}}\sum\limits_{i = 1}^n {{{\left( {{y_i} - {\beta _0} - {\beta _1}{x_i}} \right)}^2}} .\)

Obtain the maximum value

Equate the equation equal to 0,

\(\begin{array}{c}\frac{{\partial \left( {l\left( {{\sigma ^2}} \right)} \right)}}{{\partial \left( {{\sigma ^2}} \right)}} = 0\\ - \frac{n}{{2{\sigma ^2}}} + \frac{1}{{2{\sigma ^4}}}\sum\limits_{i = 1}^n {{{\left( {{y_i} - {\beta _0} - {\beta _1}{x_i}} \right)}^2}} = 0\\\frac{1}{{{\sigma ^4}}}\sum\limits_{i = 1}^n {{{\left( {{y_i} - {\beta _0} - {\beta _1}{x_i}} \right)}^2}} = \frac{n}{{{\sigma ^2}}}\\{\sigma ^2} = \frac{1}{n}\sum\limits_{i = 1}^n {{{\left( {{y_i} - {\beta _0} - {\beta _1}{x_i}} \right)}^2}} \end{array}\)

Therefore, the above value maximizes the likelihood function, but for some values of \({\beta _0}\)and\({\beta _1}\).

As mentioned earlier, values which maximize the likelihood function are \({\hat \beta _0}\)and\({\hat \beta _1}\), so substituting those values into the formula for the stationary point of the function yields the desired MLE of\({\sigma ^2}\),

\({\hat \sigma ^2} = \frac{1}{n}\sum\limits_{i = 1}^n {{{\left( {{y_i} - {{\hat \beta }_0} - {{\hat \beta }_1}{x_i}} \right)}^2}} .\)

Therefore, the desired MLE is obtained by substituting \({\hat \beta _0}\) and \({\hat \beta _1}\)in place of \({\beta _0}\)and \({\beta _1}\).