Question

A researcher performed a \(t\) test of the null hypothesis that two means are equal. He stated that he chose an alternative hypothesis of \(H_{A}: \mu_{1}>\mu_{2}\) because he observed \(\bar{y}_{1}>\bar{y}_{2}\). (a) Explain what is wrong with this procedure and why it is wrong. (b) Suppose he reported \(t=1.97\) on 25 degrees of freedom and a \(P\) -value of \(0.03 .\) What is the proper \(P\) -value? Why?

Short answer

The researcher's procedure was incorrect because the hypothesis should be determined before observing the data, not after. The proper two-tailed p-value is 0.06, obtained by doubling the one-tailed p-value of 0.03.

Step-by-step solution

Explanation of Incorrect Procedure

The researcher's decision to choose an alternative hypothesis based on the observed sample means is a form of data snooping or p-hacking, which is not a valid statistical practice. The hypothesis should be specified before collecting data or looking at the results to avoid bias. By selecting an alternative hypothesis after seeing that \(\bar{y}_{1} > \bar{y}_{2}\), the researcher increases the likelihood of committing a Type I error, falsely rejecting the null hypothesis.

Understanding the Proper P-value

If the researcher had specified a two-tailed test initially, the p-value would need to be adjusted because a two-tailed p-value represents the probability of extreme values on both ends of the distribution. Since the reported p-value of 0.03 is for the one-tailed test based on the observed direction of the difference, the proper p-value for the two-tailed test would be twice this amount.

Calculating the Correct P-value

To correct the p-value for a two-tailed test, the one-tailed p-value is multiplied by 2. Therefore, the proper p-value when considering both directions of the t-distribution would be \(0.03 \times 2 = 0.06\). This is because both tails of the distribution contain outcomes that are as extreme as or more extreme than the test statistic under the null hypothesis.

Key concepts

Hypothesis Testing

Hypothesis testing is a foundational concept in statistics used to assess the plausibility of a statistical hypothesis. At its core, it involves making an initial assumption, known as the null hypothesis (usually denoting no effect or no difference), and then determining whether the observed data provide enough evidence to reject this null hypothesis in favor of an alternative hypothesis.

In the context of our textbook exercise, the null hypothesis (\(H_0\)) posited that there was no difference between the two means (\( \bar{y}_{1} = \bar{y}_{2} \)). However, a critical mistake was made by choosing the alternative hypothesis (\( H_A: \bar{y}_{1} > \bar{y}_{2} \) ) after observing the sample means. This approach is methodologically incorrect and can lead to biased results. The correct procedure would be to establish the alternative hypothesis before analyzing data to ensure that the test is unbiased and the results are valid.

P-Hacking/Data Snooping

P-hacking, also known as data snooping, happens when researchers manipulate their analysis to obtain significant p-values. It involves practices such as selectively reporting results that support a hypothesis or tweaking statistical models until nonsignificant results become significant.

The danger of p-hacking lies in its ability to produce results that appear valid but are, in fact, spurious. In our exercise scenario, the researcher's choice of alternative hypothesis after observing the data is a form of p-hacking. This increases the risk of finding a false positive, or Type I error, because the test was tailored to the preliminary observation rather than being predetermined.

Type I Error

Type I error refers to the incorrect rejection of a true null hypothesis, also described as a 'false positive' finding. The probability of committing a Type I error is denoted by the significance level, often represented by the symbol \( \alpha \).

When a researcher commits p-hacking, as seen in the exercise, the chance of making a Type I error significantly increases. This is because the criteria for significance were inappropriately manipulated after looking at the data, rather than being based on a predetermined hypothesis and significance level.

P-Value Adjustment

P-value adjustment is a statistical technique used to provide a more accurate significance level when multiple comparisons or tests are conducted. In a two-tailed hypothesis test, we are interested in results that are significantly larger or smaller than those predicted under the null hypothesis, meaning extreme results at either end of the distribution are considered.

In the exercise, the proper p-value should have been calculated on the premise of a two-tailed test from the start. However, the researcher initially computed a one-tailed p-value (\(0.03\)) based on the direction of the mean difference after the fact. To correct for this oversight, the p-value must be adjusted to account for extreme values in both directions, resulting in a two-tailed p-value of \(0.03 \times\ 2 = 0.06\)