Question

Show in the discrete case that if \(X\) and \(Y\) are independent, then $$ E[X \mid Y=y]=E[X] \text { for all } y $$

Short answer

Given two independent discrete random variables \(X\) and \(Y\), we need to show that \(E[X|Y=y] = E[X]\). Begin with the definition of the conditional expectation \(E[X|Y=y] = \sum_{x} x P(X=x|Y=y)\). Since \(X\) and \(Y\) are independent, \(P(X=x|Y=y) = \frac{P(X=x, Y=y)}{P(Y=y)} = \frac{P(X=x)P(Y=y)}{P(Y=y)}\), where \(P(X=x, Y=y)\) was replaced with the product of \(P(X=x)\) and \(P(Y=y)\) due to independence. Cancelling \(P(Y=y)\) in the numerator and denominator simplifies the equation to \(E[X|Y=y] = \sum_{x} x P(X=x)\), which is the definition of expectation \(E[X]\). Thus, we've demonstrated that if \(X\) and \(Y\) are independent, \(E[X|Y=y] = E[X]\).

Step-by-step solution

Definition of Conditional Expectation

To start with, let's recall the definition of conditional expectation for discrete random variables: $$ E[X \mid Y=y] = \sum_{x} x P(X=x \mid Y=y) $$

Using Independence to Simplify the Probability Term

Since we know that \(X\) and \(Y\) are independent, we can use this property to simplify the probability term in the definition of conditional expectation as follows: $$ P(X=x \mid Y=y) = \frac{P(X=x, Y=y)}{P(Y=y)} $$ Since \(X\) and \(Y\) are independent, we can replace the joint probability in the numerator as follows: $$ P(X=x, Y=y) = P(X=x) P(Y=y) $$ Now, let's insert this expression back into the formula for conditional expectation: $$ E[X \mid Y=y] = \sum_{x} x \frac{P(X=x) P(Y=y)}{P(Y=y)} $$

Simplifying the Expression

We can simplify this expression by cancelling the \(P(Y=y)\) term in the numerator and denominator: $$ E[X \mid Y=y] = \sum_{x} x P(X=x) $$

Recognizing the Result as an Expected Value

Let's take a closer look at the last expression: $$ \sum_{x} x P(X=x) $$ Notice that this expression is simply the definition of the expected value of \(X\): $$ E[X] = \sum_{x} x P(X=x) $$ So we can now conclude that: $$ E[X \mid Y=y] = E[X] $$ Thus, our proof is complete. We have shown that if \(X\) and \(Y\) are independent discrete random variables, the conditional expectation of \(X\) given \(Y = y\) simplifies to the unconditional expectation of \(X\).

Key concepts

Independence of Random Variables

Understanding the concept of independent random variables is akin to recognizing paths that don't intersect in a giant maze. Two variables, let's call them 'players' in the game of probability, are said to 'mind their own business' if the outcome of one doesn't sway the fortunes of the other. Quantitatively, this means that the probability of one player scoring a point, labeled as 'event X', remains unchanged regardless of how the other player, event Y, plays the game.

In our maze, if player X takes a left or a right, it does not affect how player Y decides at their own crossroad, thus, the probability that Y takes a left is the same, irregardless of X's choice. In mathematical language, if we denote the probability of events with the symbol 'P', then for independence, it holds that:
\[ P(X=x, Y=y) = P(X=x)P(Y=y) \]
The effect of this independence is crucial when we are trying to understand more complex operations, like calculating the expected influence of one player knowing the move of another, which deftly ties into the notion of conditional expectation.

Expected Value

Now, let's focus on the expected value, often termed the 'team captain' of the probability team. The expected value, denoted by \(E[X]\), is essentially the average score you'd expect a player to achieve after a large number of games, provided the rules of each game remain unchanged. To put it simply, it's like predicting the average number of points a basketball player will score over the course of a season, calculated by taking into account the points scored in each individual game and their frequencies.

In terms of our probability game, if player X has a series of potential scores or outcomes, each associated with a distinct probability, the expected value is the sum of these different scores weighted by the likelihood of occurring:
\[ E[X] = \textstyle\text{Expected Value} = \textstyle\text{average} = \textstyle\text{sum of all possible values of } X \textstyle\text{ times their respective probabilities} \]
This 'team captain' shows how a player is likely to perform overall, a statistical MVP if you will, taking into account all the diverse outcomes and their associated probabilities. It's a foundational concept in probability, integral for understanding more complex statistical measures.

Discrete Random Variables

We've discussed players and captains, now let's meet the team: discrete random variables. In the world of probability, these variables are like a list of precise steps you can take. When you throw a die, for example, the outcome is a discrete random variable because you can only land on one of a finite set of numbers: 1, 2, 3, 4, 5, or 6.

The key characteristic of a discrete random variable, let's call it \(X\), is that it possesses a countable set of outcomes, each with an attached probability. Much like a cafeteria menu with a limited number of dishes, each dish (or outcome) has a probability of being chosen. The complete list of dishes (outcomes) and their probabilities is known as the probability mass function (PMF).

The art of dealing with discrete random variables lies in calculating probabilities and expected values, which often involves summing over their possible values. Getting acquainted with discrete random variables equips you with a fundamental tool for tackling a multitude of problems in statistics and helps to lay out the groundwork for understanding more complex ideas such as distributions and inferential statistics.