Introduction to Simple Linear Regression and Least Squares

Think of residuals as the "misses" or errors of our model. If a data point lies above the line, the residual is positive; if it lies below, the residual is negative. Our goal in regression is to find a line where these misses are as small as possible.

Exam Tip: When calculating residuals, remember that they represent the difference between the observed value and the predicted value ($y - \hat{y}$). A residual of zero means the model predicted the outcome perfectly for that specific point.

The Least Squares Estimate

How do we decide which line is the "best"? We use a method called the Least Squares Estimate.

If we simply added up all the residuals, the positive and negative values would cancel each other out, making a very bad line look perfect. To prevent this, we square each residual (making them all positive) and then sum them up. This value is known as the Sum of Squared Residuals (SSR).

The "best" line—the one that provides the best fit for our data—is the unique line that minimizes this sum. By minimizing the sum of the squared errors, we ensure that our estimates for $b_0$ and $b_1$ are as close to the true parameters as possible given our sample.

Evaluating the Fit: $R^2$ and Variance

Once we have our line, we need to know how much better it is than doing nothing at all. If we had no $x$ variable, our best guess for any $y$ would simply be the mean of all $y$ values. This "mean-only" approach has its own amount of error (variance).

To measure the effectiveness of our regression, we use the Coefficient of Determination, denoted as $R^2$.

$R^2$ tells us the proportion of the variance in $y$ that is explained by its relationship with $x$. It is calculated by comparing the "benefit" of adding the slope to the total error of the mean-only model:

$R^2 = \frac{\text{Sum of squared residuals (mean only)} - \text{Sum of squared residuals (line)}}{\text{Sum of squared residuals (mean only)}}$

For example, if $R^2 = 0.243$, we can say that "24.3% of the variance in $y$ is explained by its relationship with $x$". The remaining percentage is due to other factors or random error.

Summary Table of Concepts

By using the least squares method, we move from simply looking at dots on a graph to creating a mathematical model that can quantify relationships and provide meaningful insights into the world.