Statistical Estimation: Methods and Properties

To make the math easier, we often maximize the log-likelihood, $\ell(\theta) = \ln L(\theta)$, because the logarithm is a monotonic function and turns products into sums, which are much easier to differentiate.

Bayesian Estimation

Unlike the previous two methods, which treat the parameter $\theta$ as a fixed but unknown constant, Bayesian estimation treats $\theta$ as a random variable. This approach incorporates prior knowledge about the parameter through a prior distribution $P(\theta)$.

By combining this prior with the likelihood of the observed data, we arrive at the posterior distribution using Bayes' Theorem:

$$P(\theta | \text{data}) = \frac{P(\text{data} | \theta) P(\theta)}{P(\text{data})}$$

The Bayesian estimator is often chosen as the mean or the mode (Maximum A Posteriori, or MAP) of this posterior distribution.

Evaluating Estimator Quality

Once we have derived an estimator, we must ask: "How good is it?" We evaluate estimators based on three fundamental properties: unbiasedness, consistency, and efficiency.

1. Unbiasedness

An estimator $\hat{\theta}$ is said to be unbiased if its expected value is equal to the true parameter value $\theta$. Mathematically:

$$E[\hat{\theta}] = \theta$$

If $E[\hat{\theta}] \neq \theta$, the estimator is biased. The bias is defined as the difference between the expected value and the true value.

Exam Tip: An unbiased estimator is "correct on average." Even if a single estimate is off, if you were to repeat the experiment infinitely many times, the average of those estimates would land exactly on the true parameter.

2. Consistency

Consistency is an asymptotic property, meaning it describes how the estimator behaves as the sample size $n$ approaches infinity. An estimator is consistent if, as you collect more and more data, the estimator converges in probability to the true parameter $\theta$.

Formally, for any small positive number $\epsilon$:

$$\lim_{n \to \infty} P(|\hat{\theta}_n - \theta| < \epsilon) = 1$$

Essentially, a consistent estimator becomes "perfect" as the dataset becomes infinitely large.

3. Efficiency

Efficiency refers to the variance of the estimator. If we have two unbiased estimators, the one with the smaller variance is considered more efficient.

In many cases, we look for the Minimum Variance Unbiased Estimator (MVUE). A common benchmark for efficiency is the Cramér-Rao Lower Bound (CRLB), which provides a theoretical limit on how low the variance of an unbiased estimator can possibly go. An estimator that achieves this lower bound is said to be "efficient."

Summary Comparison

By understanding these methodologies and the properties used to judge them, statisticians can select the most robust tools for analyzing data and making scientific predictions.