Question

The $n$ quantities $x_1,x_2,\dots ,x_n$ are statistically independent; each has a Gaussian distribution with zero mean and a common variance: $$⟨x_i⟩=0⟨x_i^2⟩={\sigma }^2⟨x_i{x}_j⟩=0$$ (a) $n$ new quantities $y_1$ are defined by $y_i=\sum _j{M}_{ij}{x}_j$, where $M$ is an orthogonal matrix. Show that the $y_i$ have all the statistical properties of the $x_l$; that is, they are independent Gaussian variables with $$⟨y_i⟩=0⟨y_i^2⟩={\sigma }^2⟨y_t{y}_j⟩=0$$ (b) Choose $y_1=\sqrt{\frac{1}{n}}(x_1+x_2+\cdots +x_n)$, and the remaining $y_1$ arbitrarily, subject only to the restriction that the transformation be orthogonal. Show thereby that the mean $\overline{x}=\frac{1}{n}(x_1+\cdots +x_n)$ and the quantity $s=\sum _{i-1}^n{(x_i-\overline{x})}^2$ are statistically independent random variables. What are the probability distributions of $\overline{x}$ and $s?$ (c) One often wishes to test the hypothesis that the (unknown) mean of the Gaussian distribution is really zero, without assuming anything about the magnitude of $\sigma .$ Intuition suggests $\tau =\overline{x}/\sqrt{s}$ as a useful quantity. Show that the probability distribution of the random variable $\tau$ is $$p(\tau )=\sqrt{\frac{n}{\pi }}\frac{\mathrm{\Gamma }\left(\frac{n}{2}\right)}{\mathrm{\Gamma }\left(\frac{n-1}{2}\right)}\frac{1}{{(1+n{\tau }^2)}^{n/2}}$$ The crucial feature of this distribution (essentially the so-called Student $t$-distribufion ${}^{\mathrm{\top }}$ ) is that it does not involve the unknown parameter $\sigma$.

Short answer

First, verifying $y_i$ are Gaussian with mean 0 and ${\sigma }^2$. For $y_1$ as $\sqrt{1/n}(\sum x_i)$, $\overline{x}$ is Gaussian and $s$ is chi-squared. The $\tau$ distribution is Student's t-distribution.

Step-by-step solution

Independent Gaussian Variables with Zero Mean

Given that the quantities $x_1,x_2,\dots ,x_n$ are independent Gaussian variables with zero mean, for each $i,j$, we have: $⟨x_i⟩=0$, $⟨x_i^2⟩={\sigma }^2$, and $⟨x_i{x}_j⟩=0$ for $ieqj$.

Transformation by Orthogonal Matrix

The new quantities $y_i$ are defined as: $y_i=\sum _j{M}_{ij}{x}_j$ where $M$ is an orthogonal matrix, meaning $M{M}^T=I$.

Expectation of $y_i$

Calculate $⟨y_i⟩$: $$⟨y_i⟩=⟨\sum _j{M}_{ij}{x}_j⟩=\sum _j{M}_{ij}⟨x_j⟩=\sum _j{M}_{ij}\cdot 0=0$$. So, $⟨y_i⟩=0$.

Variance of $y_i$

Calculate $⟨y_i^2⟩$: $$⟨y_i^2⟩=⟨{\left(\sum _j{M}_{ij}{x}_j\right)}^2⟩=\sum _j{M}_{ij}^2⟨x_j^2⟩={\sigma }^2\sum _j{M}_{ij}^2={\sigma }^2$$. So, $⟨y_i^2⟩={\sigma }^2$.

Independence of $y_i$

Calculate $⟨y_i{y}_j⟩$: $$⟨y_i{y}_j⟩=⟨\left(\sum _k{M}_{ik}{x}_k\right)\left(\sum _l{M}_{jl}{x}_l\right)⟩=\sum _k\sum _l{M}_{ik}{M}_{jl}⟨x_k{x}_l⟩$$. Since $x$'s are independent, non-diagonal terms vanish. $=\sum _k{M}_{ik}{M}_{jk}{\sigma }^2={\sigma }^2{\delta }_{ij}$. Hence, $⟨y_i{y}_j⟩=0$ for $ieqj$.

Showing Mean and Variance of New Quantities

For part (b), define $y_1=\sqrt{\frac{1}{n}}(x_1+x_2+\dots +x_n)$, and remaining $y_i$'s ensuring the transformation remains orthogonal. Calculate $\overline{x}$ and $s$: $\overline{x}=\frac{1}{n}(x_1+\dots +x_n)$ and $s=\sum _{i=1}^n(x_i-\overline{x}{)}^2$. Prove $\overline{x}$ and $s$ are independent.

Probability Distributions

Show distributions of $\overline{x}$ and $s$: $\overline{x}$ is Gaussian with mean 0 and variance ${\sigma }^2/n$, $s$ follows a chi-squared distribution with $n-1$ degrees of freedom.

Distribution of $\tau$

For part (c), define $\tau =\frac{\overline{x}}{\sqrt{s}}$. Find distribution by deriving probability density function: $p(\tau )=\sqrt{\frac{n}{\pi }}\frac{\mathrm{\Gamma }\left(\frac{n}{2}\right)}{\mathrm{\Gamma }\left(\frac{n-1}{2}\right)}\frac{1}{{(1+n{\tau }^2)}^{n/2}}$. This is a Student’s $t$-distribution, independent of $\sigma$.

Key concepts

Orthogonal matrix transformation

An orthogonal matrix is a square matrix whose rows and columns are orthonormal vectors. This means that when we multiply the matrix by its transpose, we get the identity matrix: $$M{M}^T=I$$ When we transform a set of variables, say $\{x_1,x_2,...,x_n\}$, using an orthogonal matrix $M$, each new variable $y_i$ is a linear combination of the original variables: $$y_i=\sum _j{M}_{ij}{x}_j$$ Because $M$ is orthogonal, the new variables $\{y_1,y_2,...,y_n\}$ will retain the properties of the original variables such as mean and variance. Specifically, if $\{x_i\}$ are independent Gaussian variables with zero mean and common variance ${\sigma }^2$, the transform will maintain these properties for $\{y_i\}$. This is powerful because it allows us to manipulate complex transformations while maintaining the statistical properties of the original data.

Statistical independence

Statistical independence between two variables means that the occurrence of one does not affect the probability distribution of the other. For two variables $x_i$ and $x_j$: $$P(x_i,x_j)=P(x_i)\times P(x_j)$$ In the context of the given problem, the variables $\{x_i\}$ are statistically independent, which is demonstrated by the fact that the expectation of their product is zero when $ieqj$: $$⟨x_i{x}_j⟩=0$$ Similarly, if we apply an orthogonal transformation $y_i=\sum _j{M}_{ij}{x}_j$, each transformed variable $\{y_i\}$ retains independence. Hence, $⟨y_i{y}_j⟩=0$ for $ieqj$ means that the transformed variables are also independent. This preserves the structure and allows for simpler calculations in further statistical analysis.

Student's t-distribution

The Student's t-distribution is a probability distribution that there appears when estimating the mean of a normally distributed population when the sample size is small and the population standard deviation is unknown. It is particularly useful in hypothesis testing. For the random variable $\tau$ defined as $$\tau =\frac{\overline{x}}{\sqrt{s}}$$ where $\overline{x}$ is the sample mean, and $s$ is the sum of squared deviations from the mean, the probability distribution is given by $$p(\tau )=\sqrt{\frac{n}{\pi }}\frac{\mathrm{\Gamma }\left(\frac{n}{2}\right)}{\mathrm{\Gamma }\left(\frac{n-1}{2}\right)}\frac{1}{{(1+n{\tau }^2)}^{n/2}}$$ This distribution is significant because it does not involve the unknown variance ${\sigma }^2$, making it ideal for tests such as determining whether a sample mean significantly differs from a hypothesized population mean.

Gaussian random variables

Gaussian random variables, also known as normally distributed variables, follow the Gaussian distribution with a probability density function: $$P(x)=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{(x-\mu {)}^2}{2{\sigma }^2}}$$ where $\mu$ is the mean and ${\sigma }^2$ is the variance. In this exercise, we deal with Gaussian variables $\{x_i\}$ that have a mean $\mu =0$ and variance ${\sigma }^2$. These variables are crucial in physics and many other fields for modeling random processes. When we transform these variables using an orthogonal matrix, the resulting variables $\{y_i\}$ also follow a Gaussian distribution with zero mean and variance ${\sigma }^2$, preserving the overall statistical properties. This property is extremely useful for simplifying the computation and understanding complex systems and transformations.