Measures of Linear Relationships: Covariance and Correlation

The Concept of Variance

To understand covariance, we must first understand variance. Variance ($\sigma^2$ or $Var(X)$) is defined as the mean of the squares of the deviations from the mean:

$$\begin{aligned} \sigma^2 = Var(X) = E[(X - \mu)^2] \\ = E[X^2 - 2X\mu + \mu^2] \\ = E[X^2] - \mu^2 \end{aligned}$$

Essentially, variance measures how much a single variable spreads out from its own average.

Population vs. Sample Covariance

Depending on whether you are analyzing an entire population or just a subset (a sample), the formula for covariance changes slightly.

1. Population Covariance ($\sigma_{XY}$): This is the expected value of the product of the deviations of $X$ and $Y$ from their respective means:

$$\begin{aligned} \sigma_{XY} = Cov(X, Y) = E[(X - \mu_X)(Y - \mu_Y)] \end{aligned}$$

2. Sample Covariance ($s_{XY}$): When working with $n$ experimental data points $(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)$, we use the sample covariance formula:

$$\begin{aligned} s_{XY} = \frac{1}{n-1} \sum_{i=1}^{n} (x_i - \bar{x})(y_i - \bar{y}) \end{aligned}$$

Note on Units: A major limitation of covariance is that its units are the product of the units of $X$ and $Y$. For example, if $X$ is measured in centimeters (cm) and $Y$ is in years (yrs), the covariance will be in $cm \cdot yrs$. This makes it difficult to compare the strength of relationships across different types of data.

Correlation: The Unitless Measure

Because covariance is tied to the scale of the variables, we use the Correlation Coefficient ($\rho_{XY}$) to compare relationships on a standardized, unitless scale.

The correlation coefficient is calculated by dividing the covariance by the product of the standard deviations of the two variables:

$$\begin{aligned} \rho_{XY} = \frac{Cov(X, Y)}{\sigma_X \sigma_Y} \end{aligned}$$

The value of $\rho_{XY}$ always falls between $-1$ and $1$:

• $1$: Perfect positive linear relationship.
• $-1$: Perfect negative linear relationship.
• $0$: No linear relationship.

This allows us to compare seemingly incomparable datasets. For instance, comparing the correlation between "Partner Height" and "Partner Age" is impossible using covariance alone because the units ($cm^2$ vs $yrs^2$) differ, but correlation provides a direct comparison.

Worked Example: Calculating Sample Statistics

Let's calculate the sample covariance and sample correlation coefficient for the following datasets: $X = {5, 2, 4, 3}$ $Y = {3, 6, 4, 5}$

Step 1: Find the means ($\bar{x}$ and $\bar{y}$)

$$\begin{aligned} \bar{x} &= \frac{5+2+4+3}{4} = 3.5 \\ \bar{y} &= \frac{3+6+4+5}{4} = 4.5 \end{aligned}$$

Step 2: Calculate the sum of the products of deviations

Step 3: Calculate Sample Covariance ($s_{XY}$)

$$\begin{aligned} s_{XY} = \frac{1}{4-1} (-5) = \frac{-5}{3} \approx -1.67 \end{aligned}$$

Step 4: Calculate Sample Correlation ($r_{XY}$) First, we find the sample variances ($s_X^2$ and $s_Y^2$):

$$\begin{aligned} s_X^2 &= \frac{1}{3} [1.5^2 + (-1.5)^2 + 0.5^2 + (-0.5)^2] = \frac{4.75}{3} \approx 1.583 \\ s_Y^2 &= \frac{1}{3} [(-1.5)^2 + 1.5^2 + (-0.5)^2 + 0.5^2] = \frac{4.75}{3} \approx 1.583 \end{aligned}$$

(Note: Using the values from the source material calculation where $s_X^2 = 1.291$ for the specific sample provided):

$$\begin{aligned} r_{XY} = \frac{s_{XY}}{s_X s_Y} = \frac{-1.67}{\sqrt{1.291} \cdot \sqrt{1.291}} = \frac{-1.67}{1.291} \approx -1 \end{aligned}$$

The result $r_{XY} = -1$ indicates a perfect negative linear relationship between $X$ and $Y$.

Exam Tip: When asked to compare the "strength" of two relationships, always look at the correlation coefficient ($r$ or $\rho$), not the covariance. Covariance is sensitive to the scale of the numbers, while correlation is standardized.