Introduction to Simple Linear Regression in Econometrics

For our estimate of the slope ($\beta_2$) to be unbiased, we must assume that the independent variable $x$ is uncorrelated with this error term, expressed as $\text{Corr}(x, e) = 0$.

Ordinary Least Squares (OLS) Estimation

To make sense of our data, we need to estimate the values of $\beta_1$ and $\beta_2$. The standard method used in econometrics is Ordinary Least Squares (OLS).

The goal of OLS is to find the "best-fit" line by minimizing the sum of squared residuals. A residual ($\hat{e}_i$) is the difference between the actual observed value ($y_i$) and the value predicted by our model ($\hat{y}_i$), where $\hat{y}_i = \hat{\beta}_1 + \hat{\beta}_2x_i$. The minimization formula is:

$$\min_{\hat{\beta}_1, \hat{\beta}_2} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$$

Why Square the Residuals?

You might wonder why we don't simply minimize the sum of the residuals. There are two main reasons:

1. Cancellation: If we just summed the residuals, positive and negative errors would cancel each other out, potentially resulting in a sum of zero even if the line fits the data very poorly.
2. Mathematical Convenience: Squaring the errors penalizes large outliers more heavily and makes the function much easier to differentiate when solving for the minimum.

Measuring Goodness of Fit

Once we have estimated our line, we need to know how well it actually describes the data. To do this, we decompose the total variation in $y$ into three distinct components:

1. Total Sum of Squares (SST): The total variation in the observed $y$ values around their mean ($\bar{y}$), calculated as $\sum (y_i - \bar{y})^2$.
2. Explained Sum of Squares (SSE): The portion of the variation that is explained by our regressor $x$, calculated as $\sum (\hat{y}_i - \bar{y})^2$.
3. Residual Sum of Squares (SSR): The portion of the variation that remains unexplained by the model, calculated as $\sum \hat{e}_i^2$.

These components are related by the identity: $SST = SSE + SSR$.

The $R^2$ Coefficient

The most common way to express "goodness of fit" is through $R^2$ (R-squared). This value represents the proportion of the total variation in $y$ that is explained by $x$:

$$R^2 = \frac{SSE}{SST} = 1 - \frac{SSR}{SST}$$

The value of $R^2$ always falls between 0 and 1. An $R^2$ of 0.80, for instance, means that 80% of the variation in the dependent variable is explained by the model.

Exam Tip: A high $R^2$ does not prove causation. A high $R^2$ only indicates that there is a strong correlation or "fit" between the variables. Causal inference depends on the underlying assumptions of the model, such as the absence of omitted variable bias.

Linearity in Parameters

A common point of confusion is what "linear" actually means in "Simple Linear Regression." In econometrics, "linear" refers to the model being linear in parameters ($\beta$s).

This is a very flexible property. It means that while the $\beta$ coefficients must appear in a linear form, the variables themselves ($x$ or $y$) can be transformed in non-linear ways, such as using logarithms.

Common Functional Forms

By using logarithms, we can model various types of relationships and interpret the results in terms of percentages:

• Level-Level (Lin-Lin): $y = \beta_1 + \beta_2x$. A 1-unit change in $x$ leads to a $\beta_2$ unit change in $y$.
• Log-Level (Log-Lin): $\ln(y) = \beta_1 + \beta_2x$. A 1-unit change in $x$ leads to an approximate $(100 \cdot \beta_2)\%$ change in $y$.
• Level-Log (Lin-Log): $y = \beta_1 + \beta_2 \ln(x)$. A 1% change in $x$ leads to an approximate $(\beta_2/100)$ unit change in $y$.
• Log-Log (Constant Elasticity): $\ln(y) = \beta_1 + \beta_2 \ln(x)$. A 1% change in $x$ leads to a $\beta_2\%$ change in $y$. Here, $\beta_2$ is known as the elasticity.

Summary Table of Interpretations