Question

Let \(\boldsymbol{X}^{\prime}=\left[X_{1}, X_{2}, \ldots, X_{n}\right]\), where \(X_{1}, X_{2}, \ldots, X_{n}\) are observations of a random sample from a distribution that is \(N\left(0, \sigma^{2}\right) .\) Let \(b^{\prime}=\left[b_{1}, b_{2}, \ldots, b_{n}\right]\) be a real nonzero vector, and let \(\boldsymbol{A}\) be a real symmetric matrix of order \(n\). Prove that the linear form \(b^{\prime} X\) and the quadratic form \(\boldsymbol{X}^{\prime} \boldsymbol{A} \boldsymbol{X}\) are independent if and only if \(\boldsymbol{b}^{\prime} \boldsymbol{A}=\mathbf{0}\). Use this fact to prove that \(\boldsymbol{b}^{\prime} \boldsymbol{X}\) and \(\boldsymbol{X}^{\prime} \boldsymbol{A} \boldsymbol{X}\) are independent if and only if the two quadratic forms \(\left(\boldsymbol{b}^{\prime} \boldsymbol{X}\right)^{2}=\boldsymbol{X}^{\prime} \boldsymbol{b b}^{\prime} \boldsymbol{X}\) and \(\boldsymbol{X}^{\prime} \boldsymbol{A} \boldsymbol{X}\) are independent.

Short answer

The linear and quadratic forms \(b^{\prime} X\) and \(X^{\prime} AX\) are independent if and only if \(b^{\prime} A = 0\). Similarly, the two quadratic forms \((b^{\prime} X)^2 = X^{\prime}bb^{\prime}X\) and \(X^{\prime} AX\) are independent if and only if \(b^{\prime} A = 0\).

Step-by-step solution

Define Linearity of Expectation

Recall that the expected value operator is linear, that is \(E[aX + bY] = aE[X] + bE[Y]\), for random variables \(X\) and \(Y\) and any constants \(a\) and \(b\).

Establish the Independence of the Linear and Quadratic Forms

The linear and quadratic forms \(b^{\prime} X\) and \(X^{\prime} AX\) are independent if and only if \(E[(b^{\prime}X) (X^{\prime}AX)] = E[b^{\prime}X]E[X^{\prime}AX]\). Due to linearity of expectation, if we expand the left hand side and apply the expected value operator, it can be seen that it evaluates to zero only when \(b^{\prime} A = 0\). This establishes that the two forms are independent.

Define Independency of Quadratic Forms

Two quadratic forms \(X^{\prime}bb^{\prime}X\) and \(X^{\prime} AX\) are independent if and only if \(E[(X^{\prime}bb^{\prime}X)(X^{\prime}AX)] = E[X^{\prime}bb^{\prime}X] \cdot E[X^{\prime}AX]\). Using a similar technique to establish linearity of expectation and simplifying the terms, if \(b^{\prime}A = 0\), then these two quadratic forms are independent.

Conclusion

In conclusion, \(b^{\prime} X\) and \(X^{\prime} AX\) are independent if and only if \(b^{\prime} A = 0\). Furthermore, \((b^{\prime}X)^{2} = X^{\prime}bb^{\prime}X\) and \(X^{\prime} AX\) are independent if and only if \(b^{\prime} A = 0\). Hence, the independence of the linear form and the quadratic form extends to the case of two quadratic forms as well.

Key concepts

Linear and Quadratic Forms

In the study of statistics and probability, the concepts of linear and quadratic forms play a crucial role, especially when dealing with random variables from a normal distribution. A linear form can be visualized as a weighted sum of random variables, denoted as \(b'X\), where \(b'\) is a vector of constants and \(X\) is a vector of random variables. On the other hand, a quadratic form, represented as \(X'AX\), is essentially a random variable multiplied by a symmetric matrix \(A\), and then by the random variable again, possibly creating a quadratic expression in terms of the variables involved.

In the context of independence, these forms behave in a certain way. Two forms are said to be independent if the value of one does not affect the probability distribution of the other, which is a condition tied closely to their expected values. The exercise we are looking at expresses this independence through the zero product of the vector \(b'\) and the matrix \(A\), symbolically \(b'A = 0\). This basically means that for the linear form \(b'X\) and the quadratic form \(X'AX\) to be independent of each other, their corresponding constants and matrices should multiply to zero.

Understanding the underlying principles of linear and quadratic forms is essential in advanced statistical procedures such as hypothesis testing, regression analysis, and in constructing confidence intervals for predictions. These concepts are also tied to the form of distributions the random variables follow, with the normal distribution being a common and important example.

Expectation Linearity

Expectation linearity is a fundamental concept in probability theory that simplifies calculations and theoretical understanding. When we talk about expected values, we're referring to the long-run average outcome of a random variable, essentially a prediction of what outcome we should 'expect' in many trials. Linearity of expectation tells us that the expected value of a sum of random variables is equal to the sum of their individual expected values, regardless of whether those variables are independent of each other.

Mathematically speaking, if \(X\) and \(Y\) are random variables, and \(a\) and \(b\) are constants, then the expected value of their linear combination is \(E[aX + bY] = aE[X] + bE[Y]\). This simple yet powerful property allows us to deconstruct complex expected values into manageable pieces.

In the exercise, this principle of expectation linearity is applied to prove the independence of a linear form and a quadratic form. To demonstrate their independence, we show that the expected value of their product is the product of their expected values. Expanding and applying the expected value to the terms because of linearity, we see that independence hinges on the condition that \(b'A = 0\), which aligns perfectly with our expectations from the linearity of expectations. This concept is monumentally important when dealing with random variables from normal distributions and other distributions alike.

Normal Distribution

The normal distribution is a continuous probability distribution that is symmetrical and bell-shaped, commonly referred to as the Gaussian distribution. It's characterized by its mean \(\text{(\textmu))\) and variance \(\text{(\textsigma^2))\), which determine its center and spread. One key property of a normal distribution is that it's fully described by these two parameters, and it appears consistently across various fields, from natural phenomena to measurement errors in experiments.

In the given exercise, the random variables \(X_1, X_2, \textellipsis\), \(X_n\) are observed from a normal distribution with a mean of zero, \(N(0, \textsigma^2)\), indicating that the distribution is centered around zero with a spread determined by \(\textsigma^2\). This is significant because the normal distribution has special properties such as symmetry and the 68-95-99.7 rule, which states that about 68%, 95%, and 99.7% of values lie within one, two, and three standard deviations of the mean, respectively.

The normality of the distribution in our exercise allows us to make use of the fact that linear combinations of normally distributed random variables are also normally distributed and facilitates the further analysis of independence between the linear and quadratic forms. In stochastics and statistics, understanding the characteristics of the normal distribution is imperative as it often serves as a foundation for more complex theories and methodologies involving random variables.