Question

Can special study courses actually help raise SAT scores? One organization says that the 30 students they tutored achieved an average gain of 60 points when they retook the test. a) Explain why this does not necessarily prove that the special course caused the scores to go up. b) Propose a design for an experiment that could test the effectiveness of the tutorial course. c) Suppose you suspect that the tutorial course might be more helpful for students whose initial scores were particularly low. How would this affect your proposed design?

Short answer

The observed gain does not prove causation due to potential confounding variables. Use a randomized controlled trial with a control group to test course effectiveness, and consider stratified sampling if low initial scores are thought to affect results.

Step-by-step solution

Understanding the Limitation

The claim that 30 students achieved an average gain of 60 points does not necessarily demonstrate causation. This is because correlation does not imply causation. There could be other factors, such as students' increased motivation, additional study outside the course, or prior experience that may have contributed to the improved scores. Without a control group, it's impossible to isolate the effect of the course.

Designing a Controlled Experiment

To test the effectiveness of the tutorial course, an experimental design should include random assignment of students into two groups: one receiving the tutorial and the other not (control group). Both groups should ideally be equivalent in terms of initial test scores and other relevant variables. After completing the course, both groups should retake the SAT. Comparing the score improvements between the two groups can help infer the course's effectiveness.

Considering Initial Scores in the Design

If you suspect the course is more beneficial for students with initially low scores, stratified random sampling can be used. Students can be grouped by their initial scores, then randomly assigned to either the tutorial or control group within each stratum. This ensures each initial score range is fairly represented in both groups, allowing for an analysis of the course's impact across different initial score levels.

Key concepts

Causation vs Correlation

In any scientific study, it's crucial to distinguish between causation and correlation. Just because two events occur together does not mean one caused the other. For instance, the observed improvement in SAT scores after taking a special study course does not automatically imply that the course caused this improvement. Other factors could also be responsible, such as:

• Increased motivation: Students might be more focused when retaking the test.
• Supplementary study: Students could be studying outside of the course.
• Previous experience: Early exposure to the material might have helped.

The key is that correlation indicates a relationship or pattern between two variables, while causation explicitly shows that one factor causes the other to happen. To establish causation, researchers must control for other variables that could affect the outcome, often through carefully designed experiments.

Controlled Experiment

A controlled experiment is essential for firmly establishing causation. In the case of evaluating the effectiveness of a tutorial course on SAT scores, the experiment must be structured carefully. Here's how you can set it up:

• Random Assignment: Randomly assign students into two groups: one receives the tutorial (intervention group), and the other does not (control group).
• Equivalence: Ensure both groups are similar in terms of initial SAT scores and other characteristics like study habits or motivation levels.
• Testing: After the course, compare the score changes between the two groups. This comparison helps isolate the effect of the tutorial.

By maintaining control over these variables, the experiment can more accurately determine whether the course is the cause of any score improvements.

Stratified Sampling

Stratified sampling enhances the experiment's accuracy by ensuring fair representation of different subgroups. If you suspect that the tutorial course is particularly beneficial for students with lower initial SAT scores, stratified random sampling can help tailor the experimental design:

• Stratification: Divide students into strata based on their initial scores (e.g., low, medium, high).
• Random Assignment Within Strata: Randomly assign students within each stratum to either the tutorial or control group.
• Analysis: This approach ensures that each subgroup is adequately represented, allowing researchers to analyze the course's effects across different initial score levels.

Stratified sampling is particularly useful when there's a known characteristic that might influence the outcome. This method helps in achieving more nuanced insights into how different groups might be affected differently by the tutorial course.