Question

Let X be a random variable that takes on 5 possible values with respective probabilities .35, .2, .2, .2, and .05. Also, let Y be a random variable that takes on 5 possible values with respective probabilities .05, .35, .1, .15, and .35. (a) Show that H(X) > H(Y). (b) Using the result of Problem 9.13, give an intuitive explanation for the preceding inequality.

Short answer

The results are

(a)$H\left(X\right)=2.14,H\left(y\right)=2.02$

(b)$X$is closer to unform distribution as compare to$Y$.

Step-by-step solution

Part (a) Step 1: Given Information

We have to prove$H\left(X\right)>H\left(Y\right)$.

Part (a) Step 2: Simplify

Consider

$H\left(X\right)=-\underset{}{\sum _i{p}_i\log p_i=-0.35{\log }_{2}0.35-3·0.2{\log }_{2}0.2-0.5=2.14}$

and on the other side, we have

localid="1651485575012" $$\begin{gathered} H\left(Y\right)=-\underset{}{\sum _i{p}_i\log p_i=-0-05{\log }_{2}0.05-0.35{\log }_{2}0.35-0.1{\log }_{2}0.1} \\ -0.15{\log }_{2}0.15-0.35{\log }_{2}0.35=2.02 \end{gathered}$$

So, we have proved $H\left(X\right)>H\left(Y\right)$.

Part (b) Step 1: Given Information

We have to find an intuitive explanation for the preceding inequality.

Part (b) Step 2: Explanation

Considering random variable $X$has much more uncertainty than $Y$, i.e. its distribution is much closer to the uniform distribution as compare to$Y$and from the problem $9.13.$we know that uniform distribution has maximal entropy.