Introduction to Probability Theory

The Rules of Probability

Probability is the assignment of a number $P(A)$ to an event $A$. For these assignments to be mathematically sound, they must obey three fundamental axioms:

1. Non-negativity and Boundedness: For any event $A$, $0 \le P(A) \le 1$.
2. Certainty: The probability of the entire sample space is $P(S) = 1$.
3. Additivity for Mutually Exclusive Events: If two events $A$ and $B$ are mutually exclusive (meaning they cannot both occur, so $A \cap B = \emptyset$), then the probability of either occurring is the sum of their individual probabilities: $P(A \cup B) = P(A) + P(B)$.

Exam Tip: Always check if events are mutually exclusive before adding their probabilities. If they can overlap, you must use the General Addition Rule to avoid double-counting the intersection.

The General Addition Rule and Complement Rule

When events are not mutually exclusive, we use the general addition rule to account for the overlap:

$$\begin{aligned} P(A \cup B) = P(A) + P(B) - P(A \cap B) \end{aligned}$$

By subtracting the intersection, we ensure that the outcomes belonging to both $A$ and $B$ are only counted once. Additionally, the complement rule allows us to find the probability of an event not occurring quite easily: $P(A^c) = 1 - P(A)$.

Conditional Probability and Independence

Often, we want to know the probability of an event given that we already have some information. This is known as conditional probability. For any event $B$ where $P(B) > 0$, the probability of $A$ given $B$ is:

$$\begin{aligned} P(A | B) = \frac{P(A \cap B)}{P(B)} \end{aligned}$$

From this definition, we can derive the multiplicative rule, which is useful for calculating the probability of two events happening together:

$$\begin{aligned} P(A \cap B) = P(A | B)P(B) = P(B | A)P(A) \end{aligned}$$

Independence

Two events are considered independent if the occurrence of one does not change the probability of the other. Mathematically, $A$ and $B$ are independent if:

$$\begin{aligned} P(A | B) = P(A) \iff P(A \cap B) = P(A)P(B) \end{aligned}$$

Note that "mutually exclusive" and "independent" are very different concepts: if two events are mutually exclusive, they are actually highly dependent, because if one happens, the probability of the other happening becomes zero.

Advanced Probability: Partitions and Bayes' Rule

A partition of a sample space is a set of events that are both mutually exclusive and exhaustive, meaning they do not overlap and they cover the entire sample space $S$. The simplest partition is ${A, A^c}$.

The Law of Total Probability (LTP)

If we have a partition, we can find the total probability of an event $B$ by summing its conditional probabilities across all parts of the partition:

$$\begin{aligned} P(B) = P(B | A)P(A) + P(B | A^c)P(A^c) \end{aligned}$$

This is often visualized using a probability tree, where you multiply probabilities along the branches and add the results of the different paths.

Bayes' Rule

Bayes' Rule is a powerful tool that allows us to "reverse" conditional probabilities. If we know $P(B|A)$ but want to find $P(A|B)$, we use:

$$\begin{aligned} P(A | B) = \frac{P(B | A)P(A)}{P(B)} \end{aligned}$$

When $P(B)$ is not directly provided, we substitute the Law of Total Probability into the denominator.

Caution: The Base-Rate Fallacy. A common mistake is to judge $P(A|B)$ based solely on the accuracy of the condition $P(B|A)$, while ignoring the "base rate" or the initial probability $P(A)$. For example, even a highly accurate medical test may yield a low probability of actual disease if the disease itself is extremely rare.