Question

Let \(\hat{\beta}_{0}\) and \(\hat{\beta}_{1}\) be the OLS intercept and slope estimators, respectively, and let \(\bar{u}\) be the sample average of the errors (not the residuals!). i. Show that \(\hat{\beta}_{1}\) can be written as \(\widehat{\beta}_{1}=\beta_{1}+\sum_{i=1}^{n} w_{i} u_{i},\) where \(w_{i}=d_{i} / \mathrm{SST}_{x}\) and \(d_{i}=x_{i}-\bar{x}\). ii. Use part (i), along with \(\sum_{i=1}^{n} w_{i}=0,\) to show that \(\widehat{\beta}_{1}\) and \(\bar{u}\) are uncorrelated. [Hint: You are being asked to show that \(\left.\mathrm{E}\left[\left(\widehat{\beta}_{1}-\beta_{1}\right) \cdot \bar{u}\right]=0 .\right]\) iii. Show that \(\widehat{\beta}_{0}\) can be written as \(\widehat{\beta}_{0}=\beta_{0}+\bar{u}-\left(\widehat{\beta}_{1}-\beta_{1}\right) \bar{x}\). iv. Use parts (ii) and (iii) to show that \(\operatorname{Var}\left(\widehat{\beta}_{0}\right)=\sigma^{2} / n+\sigma^{2}(\bar{x})^{2} / \mathrm{SST}_{x}\). v. Do the algebra to simplify the expression in part (iv) to equation (2.58) [Hint: \(\left.\operatorname{SST}_{x} / n=n^{-1} \sum_{i=1}^{n} x_{i}^{2}-(\bar{x})^{2} \cdot\right]\)

Short answer

\( \hat{\beta}_1 = \beta_1 + \sum_{i=1}^{n} w_i u_i \); \( \hat{\beta}_0 = \beta_0 + \bar{u} - (\hat{\beta}_1 - \beta_1)\bar{x} \); Variances: \( \frac{\sigma^2}{n} + \frac{\sigma^2(\bar{x})^2}{\text{SST}_x} \).

Step-by-step solution

Express \( \hat{\beta}_{1} \)

The formula for the OLS slope estimator is \( \hat{\beta}_{1} = \frac{\sum (x_i - \bar{x}) y_i}{\sum (x_i - \bar{x})^2} \). Given that \( y_i = \beta_0 + \beta_1 x_i + u_i \), we can substitute and find that \( \hat{\beta}_{1} = \beta_1 + \frac{\sum (x_i - \bar{x}) u_i}{\mathrm{SST}_x} \), where \( \mathrm{SST}_x = \sum (x_i - \bar{x})^2 \) and \( w_i = \frac{x_i - \bar{x}}{\mathrm{SST}_x} \).

Show Uncorrelation between \( \hat{\beta}_{1} \) and \( \bar{u} \)

We use the result from step 1, \( \hat{\beta}_{1} - \beta_{1} = \sum_{i=1}^{n} w_{i} u_{i} \), and the fact \( \sum_{i=1}^{n} w_{i} = 0 \). Consequently, \( \mathrm{E}\left[\left(\hat{\beta}_{1} - \beta_{1}\right) \cdot \bar{u}\right] = 0 \), showing that \( \hat{\beta}_{1} \) and \( \bar{u} \) are uncorrelated.

Express \( \hat{\beta}_{0} \)

The formula for the OLS intercept \( \hat{\beta}_{0} \) can be expressed as \( \hat{\beta}_{0} = \bar{y} - \hat{\beta}_{1} \bar{x} \). Substituting the expression for \( \hat{\beta}_{1} \) gives \( \hat{\beta}_{0} = \beta_{0} + \bar{u} - \left(\hat{\beta}_{1} - \beta_{1}\right) \bar{x} \).

Calculate \( \operatorname{Var}(\hat{\beta}_{0}) \) using \( \hat{\beta}_{1} \) and \( \bar{u} \)

Since \( \hat{\beta}_{1} \) and \( \bar{u} \) are uncorrelated, \( \operatorname{Var}(\hat{\beta}_{0}) = \operatorname{Var}(\beta_{0} + \bar{u} - \left(\hat{\beta}_{1} - \beta_{1}\right) \bar{x}) \). This gives \( \operatorname{Var}(\hat{\beta}_{0}) = \frac{\sigma^{2}}{n} + \frac{\sigma^{2} (\bar{x})^2}{\mathrm{SST}_x} \).

Simplify \( \operatorname{Var}(\hat{\beta}_{0}) \)

Utilize the formula \( \mathrm{SST}_x = n \left(\frac{1}{n} \sum_{i=1}^{n} x_i^2 - (\bar{x})^2\right) \) to further simplify \[ \operatorname{Var}(\hat{\beta}_{0}) = \sigma^{2} \left( \frac{1}{n} + \frac{(\bar{x})^2}{\mathrm{SST}_x} \right) \] in terms presented initially as equation (2.58).

Key concepts

OLS estimation

Ordinary Least Squares (OLS) estimation is a method used to determine the parameters of a linear regression model. It aims to minimize the sum of the squared differences between the observed values and the values predicted by the model. This method is popular in statistical regression analysis because it provides the best linear unbiased estimators under certain assumptions.

In the context of OLS, we estimate two primary parameters: the slope (\( \hat{\beta}_{1} \)) and the intercept (\( \hat{\beta}_{0} \)). These represent the linear relationship between the dependent and independent variables. The slope, \( \hat{\beta}_{1} \), shows how much change in the dependent variable is expected with a one-unit change in the independent variable. The intercept, \( \hat{\beta}_{0} \), indicates where the line crosses the y-axis when the independent variable is zero.

Using OLS estimation involves deriving formulas for \( \hat{\beta}_{1} \) and \( \hat{\beta}_{0} \) by solving the so-called "normal equations," which ensure that the sum of squared residuals is minimized, leading to the least error in predictions.

Variance of estimators

The variance of an estimator helps to measure how much an estimator would vary from sample to sample. In OLS, the variances of the estimators \( \hat{\beta}_{0} \) and \( \hat{\beta}_{1} \) provide an indication of the precision of these estimates.

• A low variance indicates that the estimator is quite precise, meaning repeated samples would yield estimates close to the true parameter.
• A high variance suggests more fluctuation in the estimates with different samples, leading to less precision.

In the exercise, the variance of \( \hat{\beta}_{0} \) is provided as a combination of the sample variance and the variance of \( \bar{x} \), which is the sample mean of the independent variable. This reflects how errors in measuring \( \hat{\beta}_{1} \) can impact the intercept estimator. Simplifying this variance leads to a more clear expression evaluated from equation (2.58), shedding light on the influence of individual sample points and their average on the regression model's intercept.

Uncorrelated random variables

Two random variables are said to be uncorrelated if their covariance is zero. In simpler terms, knowing the value of one variable provides no information about the other. For OLS estimators, demonstrating that certain quantities are uncorrelated can be important for making statistical inferences.

In the exercise, we showed that \( \hat{\beta}_{1} \), the slope estimator, and \( \bar{u} \), the sample average of errors, are uncorrelated. This means that errors in estimating the slope are not systematically related to the average error. Such an uncorrelation arises because the weights \( w_i \) sum to zero, causing the influence on one quantity to cancel out any association with the other.

This concept is vital in regression analysis as it helps in understanding the independence of the slope's estimation error from other components of the model, like the average error.

Statistical regression analysis

Statistical regression analysis involves identifying the relationship between a dependent variable and one or more independent variables. It is a crucial tool for examining patterns, forecasting, and determining causality in a data set. OLS is one of the simplest and most commonly used forms of regression analysis.

OLS regression analysis relies on assumptions such as linearity, constant variance, and normally distributed errors, which if met, ensures the estimators are the Best Linear Unbiased Estimators (BLUE). This implies they have the least variance among all unbiased linear estimators.

The process of regression analysis allows researchers to draw conclusions about how a set of variables influences another variable. This is achieved through estimating the coefficients (like \( \hat{\beta}_{0} \) and \( \hat{\beta}_{1} \)) and assessing their significance. By analyzing these relationships, we gain insights that aid decision-making in various fields such as economics, finance, biology, and engineering.