Question

Some straightforward but slightly tedious algebra shows that $$ \text { SSResid }=\left(1-r^{2}\right) \sum(y-\bar{y})^{2} $$ from which it follows that $$ s_{e}=\sqrt{\frac{n-1}{n-2}}\left(\sqrt{1-r^{2}}\right) s_{y} $$ Unless \(n\) is quite small, \((n-1) /(n-2) \approx 1\), so $$ s_{e} \approx\left(\sqrt{1-r^{2}}\right) s_{y} $$ a. For what value of \(r\) is \(s_{e}\) as large as \(s_{y}\) ? What is the equation of the least-squares line in this case? b. For what values of \(r\) will \(s_{e}\) be much smaller than \(s_{y} ?\)

Short answer

In part a: For \(r\) = 0, \(s_{e}\) equals \(s_{y}\), and the equation of the least-squares line is \(y = \overline{y}\). In part b: \(s_{e}\) is much smaller than \(s_{y}\) when \(r\) value is close to 1 or -1 as they signify strong linear relationship between the variables.

Step-by-step solution

Find the value of \(r\) for which \(s_{e} = s_{y}\)

Use the third equation, \(s_{e} \approx\left(\sqrt{1-r^{2}}\right) s_{y}\), and set \(s_{e} = s_{y}\). This leads to: \(s_{y} = \left(\sqrt{1-r^{2}}\right) s_{y}\); divide by \(s_{y}\) -remembering it cannot be 0- on both sides, and solve for \(r\). From this equation, you can see that \(r\) equals 0.

Finding the least-squares line when \(r = 0\)

A line of best fit (least-squares line) has the formula \(y = mx + c\), where \(m\) is the slope of the line and \(c\) is the y-intercept. When \(r = 0\), the correlation is zero, meaning that there is no linear relationship between the variables. So, the slope of the line \(m = 0\), and the least-squares line becomes \(y = c\). That is, the best predictor of \(y\) when there is no correlation with \(x\) is simply the mean of \(y\), which is \( c = \overline{y}\). Therefore, in this case, the equation of the least-squares line is \(y = \overline{y}\).

For which \(r\) will \(s_{e}\) be much smaller than \(s_{y}\)?

Referring back to the equation from Step 1, notice \(s_{e} < s_{y}\) if and only if \(\left(\sqrt{1-r^{2}}\right) < 1\). That happens when \(|r| > 0\). In essence, for \(s_{e}\) to be much smaller than \(s_{y}\), \(r\) should be close to -1 or 1, with the knowledge that the absolute value of \(r\) is no larger than 1, \(r\) ranges between -1 and 1. Overview of the correlation coefficient \(r\) suggests that the residuals are much smaller than the \(y\)-variable when there is a strong positive or negative linear relationship between the variables, that is, when \(|r|\) approaches 1.

Key concepts

Least Squares Regression

The principle of least squares regression is foundational to understanding how statistical relationships between variables are quantified. When we create a regression model, our goal is to find a line—called the least squares line—that minimizes the difference between the observed data points and the predicted values. This 'difference' is referred to as the residuals, or the distance from the data points to the regression line.

The formula for the least squares line is typically written as \( y=mx+c \), where \( m \) represents the slope and \( c \) the y-intercept. The slope indicates how much \( y \) changes for a one-unit change in \( x \). The y-intercept, on the other hand, tells us the value of \( y \) when \( x \) is zero. Least squares regression is all about adjusting the values of \( m \) and \( c \) so that the sum of the squared residuals is as small as possible—hence the name 'least squares.'

Correlation Coefficient

The correlation coefficient, commonly represented by \( r \), measures the strength and direction of a linear relationship between two variables. It is a value between -1 and 1 where 1 indicates a perfect positive linear correlation, -1 indicates a perfect negative linear correlation, and 0 indicates no linear relationship at all.

When \( r \) is close to 1 or -1, it suggests that the variables have a strong linear relationship, meaning changes in one variable are closely related to changes in the other. If \( r \) is near 0, it would suggest little to no linear relationship; hence the variables don't move together in a specific, predictable pattern. Understanding the value of \( r \) is crucial when analyzing the fit of a least squares regression line, as it affects the spread and variability of residuals.

Sum of Squares Residuals

Within the context of regression analysis, the sum of squares residuals (sometimes just called the sum of squared errors) is a measure of the total variance in the dependent variable that is not explained by the regression model. It's symbolized by \( SS_{Resid} \) and calculated by taking the sum of the squared differences between the observed values and the values predicted by the regression line.

Mathematically, \( SS_{Resid} = \textstyle \text{Sum} ormalsize (y_i - \bar{y}_{i})^2 \), where \( y_i \) are the actual observations and \( \bar{y}_{i} \) are the predictions. A lower sum of squares residuals indicates a model that more accurately predicts the dependent variable. In our exercise, \( SS_{Resid} \) is related to the correlation coefficient by \( SS_{Resid} = (1 - r^2) \textstyle \text{Sum} ormalsize (y - \bar{y})^2 \), showing that as the absolute value of \( r \) increases, the unexplained variance decreases.

Standard Deviation of Residuals

The standard deviation of residuals, often denoted as \( s_e \), tells us about the typical size of the residuals in a regression analysis. Essentially, it measures the typical distance that the data points fall from the regression line. A smaller \( s_e \) suggests that the model has a tighter fit to the data, whereas a larger \( s_e \) indicates more spread in the data points from the line.

The standard deviation of residuals is the square root of the variance of residuals, and in the context of our exercise, it's approximated as \( s_e \thickapprox (\thickapprox1-r^2)s_y \), where \( s_y \) denotes the standard deviation of the dependent variable and \( r \) is the correlation coefficient. From the equation, we can observe that when \( r \) is close to 0, the standard deviation of residuals is approximately equal to the individual standard deviation of the dependent variable. Hence, when the correlation coefficient is low, the predictive power of the regression line is also low.