Question

Consider the following permutation test for the two-sample problem with hypotheses \((4.9 .7) .\) Let \(\mathbf{x}^{\prime}=\left(x_{1}, x_{2}, \ldots, x_{n_{1}}\right)\) and \(\mathbf{y}^{\prime}=\left(y_{1}, y_{2}, \ldots, y_{n_{2}}\right)\) be the realizations of the two random samples. The test statistic is the difference in sample means \(\bar{y}-\bar{x} .\) The estimated \(p\) -value of the test is calculated as follows: 1\. Combine the data into one sample \(\mathbf{z}^{\prime}=\left(\mathbf{x}^{\prime}, \mathbf{y}^{\prime}\right)\). 2\. Obtain all possible samples of size \(n_{1}\) drawn without replacement from \(\mathrm{z}\). Each such sample automatically gives another sample of size \(n_{2}\), i.e., all elements of \(\mathbf{z}\) not in the sample of size \(n_{1}\). There are \(M=\left(\begin{array}{c}n_{1}+n_{2} \\ n_{1}\end{array}\right)\) such samples. 3\. For each such sample \(j\) : (a) Label the sample of size \(n_{1}\) by \(\mathbf{x}^{*}\) and label the sample of size \(n_{2}\) by \(\mathbf{y}^{*}\). (b) Calculate \(v_{j}^{*}=\bar{y}^{*}-\bar{x}^{*}\). 4\. The estimated \(p\) -value is \(\hat{p}^{*}=\\#\left\\{v_{j}^{*} \geq \bar{y}-\bar{x}\right\\} / M\). (a) Suppose we have two samples each of size 3 which result in the realizations: \(\mathbf{x}^{\prime}=(10,15,21)\) and \(\mathbf{y}^{\prime}=(20,25,30)\). Determine the test statistic and the permutation test described above along with the \(p\) -value. (b) If we ignore distinct samples, then we can approximate the permutation test by using the bootstrap algorithm with resampling performed at random and without replacement. Modify the bootstrap program boottesttwo.s to do this and obtain this approximate permutation test based on 3000 resamples for the data of Example \(4.9 .2 .\) (c) In general, what is the probability of having distinct samples in the approximate permutation test described in the last part? Assume that the original data are distinct values.

Short answer

The solution includes two major parts. The Permutation test for \(\mathbf{x}^{\prime}=(10,15,21)\) and \(\mathbf{y}^{\prime}=(20,25,30)\) that yields a specific test statistic and a p-value. The test statistic and the p-value depends upon the random samples drawn from the combined data. Then, an approximate permutation test is obtained using the Bootstrap method for 3000 resamples. Finally, the general probability in having distinct samples in the approximate permutation test is discussed theoretically.

Step-by-step solution

Compute mean of both samples

Start by calculating the sample means \(\bar{x}\) and \(\bar{y}\) for \(\mathbf{x}^{\prime}=(10,15,21)\) and \(\mathbf{y}^{\prime}=(20,25,30)\) respectively.

Form a combined sample

Combine both sets of data to create one sample \(\mathbf{z}^{\prime}=(\mathbf{x}^{\prime}, \mathbf{y}^{\prime})\).

Generate all possible samples from combined data

Derive all possible samples of size \(n_{1}=3\) without replacement from the combined sample \(\mathrm{z}\). Calculate the total number of such samples \(M=\binom{n_{1}+n_{2}}{n_{1}}\).

Calculate test statistic for each permutation

For each permutation sample, label the sample of size \(n_{1}\) by \(\mathbf{x}^{*}\) and label the sample of size \(n_{2}\) by \(\mathbf{y}^{*}\). Calculate \(v_{j}^{*}=\bar{y}^{*}-\bar{x}^{*}\).

Estimate the p-value

After calculating each permutation's test statistic, compute the estimated p-value \(\hat{p}^{*} = \frac{\#\left\{v_{j}^{*} \geq \bar{y}-\bar{x}\right\}}{M}\).

Bootstrap algorithm for approximate permutation

Modify the bootstrap program to perform the approximation task for 3000 resamples without replacement.

Theoretical probability for having distinct samples

Discuss theoretically the probability of having distinct samples in the approximate permutation test.

Key concepts

Sample Means

In statistics, the term \textbf{sample mean} refers to the average value of a set of data points drawn from a larger population. The sample mean is calculated by summing all the observed values and then dividing by the number of observations. It serves as a reliable estimator of the population mean, assuming the sample is random and representative.

For instance, if you have a sample containing the values \(10, 15, 21\) like in our exercise, the sample mean \(\bar{x}\) is calculated by adding these values together and dividing by the number of observations, which is 3 in this case. Consequently, \(\bar{x}\) would equal \((10 + 15 + 21) / 3\), giving us the sample mean for this particular set. Understanding sample means is crucial for various statistical tests, including hypothesis testing and performing permutation tests.

Bootstrapping

The technique of \textbf{bootstrapping} is a resampling method used to estimate the distribution of a statistic by randomly drawing samples with replacement from the data. This approach enables one to approximate the sampling distribution of almost any statistic and assess the variability of the estimates.

In our exercise, bootstrapping can be adapted for an approximate permutation test by resampling without replacement, generating numerous resampled datasets (here, 3000 resamples are used), each time recalculating the same test statistic used in the original test. By comparing these statistics to the one from your original sample, you can estimate the \(p\)-value, which tells you how likely it is to observe a test statistic as extreme as the one you calculated, purely by chance.

Hypothesis Testing

When it comes to \textbf{hypothesis testing}, you're essentially putting a sample statistic, such as a sample mean, to the test to see if it provides strong evidence against an assumed population parameter under a null hypothesis. It involves comparing your test statistic to a threshold determined by the significance level to either reject or fail to reject the null hypothesis.

In a permutation test, as shown in our textbook exercise, hypothesis testing is performed by generating all possible outcomes under the null hypothesis. You then compare your original test statistic to the permutation distribution of the test statistic. This approach allows us to see how extreme the observed statistic is without making many of the assumptions required by parametric tests, such as those of a normal distribution or homoscedasticity (equal variances). The estimated \(p\)-value from the exercise gives the probability of observing a test statistic as extreme as the observed, assuming that the null hypothesis is true.