Question

Show that for any discrete random variable $X$and function$f$

$H\left(f\right(X\left)\right)\le H\left(X\right)$

Short answer

The given statement is proved below.

Step-by-step solution

Given Information

We have to prove that

$H\left(f\right(X\left)\right)\le H\left(X\right)$

simplify

Consider two random variables, $X$and$f\left(X\right)$. Using the formula about the entropy of two random variables, we have that

$H\left(X,f\left(X\right)\right)=H\left(X\right)+H_X\left(f\left(X\right)\right)$

Observe that knowing $X$, random variable $f\left(X\right)$becomes absolutely deterministic, i.e. there is no more uncertainty about its value. Hence $H_X\left(f\left(X\right)\right)=0$which implies that $H\left(X,f\left(X\right)\right)=H\left(X\right)$. On the other hand, we can also write

$H\left(X,f\left(X\right)\right)=H\left(f\left(X\right)\right)+H_{f\left(X\right)}\left(X\right)\ge H\left(f\left(X\right)\right)$

which finally implies that

$H\left(f\left(X\right)\right)\le H\left(X\right)$