Correlation and Regression Fundamentals

Calculating $r$ using Z-scores

To calculate the sample correlation coefficient, we first standardize the $x$ and $y$ coordinates by converting them into z-scores. The formula is:

$r = \frac{\sum z_x z_y}{n - 1}$

where the z-scores are calculated as: $z_x = \frac{x - \bar{x}}{s_x}$ $z_y = \frac{y - \bar{y}}{s_y}$

In these formulas, $\bar{x}$ and $\bar{y}$ are the sample means, and $s_x$ and $s_y$ are the sample standard deviations.

Statistical Significance

Just because we calculate an $r$ value doesn't mean the relationship is "real" rather than just a result of random chance in our sample. To test this, we compare $|r|$ to a critical value found in statistical tables. This comparison depends on the degrees of freedom ($df$), which for two variables is calculated as $df = n - 2$. If the absolute value of $r$ is greater than the critical value, we have evidence of a statistically significant linear relationship.

Correlation vs. Causation

One of the most important rules in statistics is that correlation does not imply causation. Just because two variables move together does not mean that one causes the other to change.

There are two main reasons why a correlation might be misleading:

1. Reversed Causality: The relationship might be the opposite of what you think (e.g., $Y$ actually causes $X$).
2. Lurking Variables: A third, unmeasured variable might be influencing both $X$ and $Y$ simultaneously.

[!IMPORTANT] Exam Tip: If a question asks if you can claim "cause-and-effect" based solely on an observational study, the answer is almost always no. To establish causation, researchers must conduct controlled experiments.

Linear Regression and Predictions

When a strong linear relationship exists, we can use it to make predictions using a least-squares regression line. This line is the "best fit" because it minimizes the sum of squared residuals (SSE).

A residual is the error in our prediction—the difference between the actual observed value ($y$) and the value predicted by our model ($\hat{y}$): $\text{Residual} = y - \hat{y}$

The Regression Equation

The equation for the line is typically written in slope-intercept form: $\hat{y} = ax + b$ where $a$ is the slope and $b$ is the y-intercept.

• The Slope ($a$): Represents the predicted change in the response variable for every one-unit increase in the explanatory variable.
• The Y-intercept ($b$): The predicted value of the response variable when the explanatory variable is zero. (Note: Be careful interpreting this if zero is not a realistic value for your data!)

Interpolation vs. Extrapolation

When using our equation to predict values, we must be careful about the range of our data:

• Interpolation: Predicting values within the range of the observed data. This is generally considered reliable.
• Extrapolation: Predicting values outside the range of the observed data. This is much less reliable and can lead to nonsensical results.

Evaluating the Model: $R^2$

How much of the "story" does our regression line actually tell? We use the coefficient of determination ($R^2$) to measure this.

$R^2$ represents the proportion of the total variation in the response variable that is explained by the linear relationship with the explanatory variable. It is calculated as: $R^2 = \frac{SSR}{SST}$ where $SSR$ is the sum of squares due to regression and $SST$ is the total sum of squares.

If $R^2 = 0.75$, it means 75% of the variation in your response variable is explained by your model, while the remaining 25% is due to other factors or random error.