4.3.3 Significance Testing

Introduction

After understanding p-values, the next step in hypothesis testing is significance testing. Significance testing is a statistical method used to determine if a result is due to chance or if it is statistically significant.

The textbook explains that "statistically significant" means the result is likely due to some real effect or factor, not just random chance. Significance testing helps determine if results are meaningful by comparing the test statistic with a critical value and setting a significance level.

Definition

Significance Testing is a statistical method used to determine if a result is due to chance or if it is statistically significant (likely due to some real effect or factor).

In simple words: Significance testing helps you decide whether your results are real or just a coincidence.

Main Idea

The core idea is that not all results are meaningful. Some results occur simply by chance. Significance testing provides a formal way to decide whether a result is likely real or likely due to chance.

• Significance testing helps determine if results are meaningful.
• It compares the test statistic with a critical value.
• Setting a significance level helps decide the threshold for importance.
• Significance testing helps in making data-driven decisions.
• It is crucial for validating research findings.

Core Concept

What is Statistical Significance? The textbook defines: "Statistically significant means it is likely due to some real effect or factor." In other words, a result is statistically significant if it is unlikely to have occurred by chance.

The Significance Testing Process The textbook explains the key elements:

• Test Statistic: A value calculated from the data that measures how much the data deviates from the null hypothesis.
• Critical Value: A threshold value that the test statistic is compared against.
• Significance Level: The threshold for determining if results are significant (often denoted as α, alpha).

• If the test statistic exceeds the critical value (or if the p-value is less than the significance level), the result is statistically significant → reject H₀.
• If the test statistic does not exceed the critical value (or if the p-value is greater than the significance level), the result is not statistically significant → fail to reject H₀.

Example 1: Exercise and Blood Pressure (from the book) The textbook provides a detailed example:

Research question: "Can exercising for six weeks help lower blood pressure?"

Null Hypothesis (H₀): Exercise doesn't affect blood pressure. Alternative Hypothesis (H₁): Exercise lowers blood pressure.

Data collection: Researchers collect data from people who exercised for six weeks. They find a reduction in blood pressure of 8 mmHg.

Significance test: The test checks if this result is real or just a coincidence. By applying statistical formulas, the p-value is calculated.

Result: The p-value indicates that there is only a 0.5% chance (0.005) of getting this result by chance.

Conclusion: Since the p-value is very small (0.005), the null hypothesis is rejected, and the alternative hypothesis is accepted. This means that exercising for six weeks likely lowers blood pressure.

Example 2: General Significance Testing The textbook also provides a general description of significance testing:

"If we formulate a research question 'Can exercising for six weeks help lower blood pressure?' The null hypothesis is 'Exercise doesn't affect blood pressure' while the alternative hypothesis will be 'Exercise lowers blood pressure'."

"For statistical experimentation we collect data from the people who exercised for six weeks and had a reduction in blood pressure of 8mmHg. The significant test will be to check if this result is real or just a coincidence. By applying statistical formula, the P-Value will be calculated which indicates that there is only 0.5% chance of getting this result by chance."

"So, we conclude that since P-Value is very small, that is 0.005 we reject the null hypothesis and accept the alternate hypothesis. It means that exercising for six weeks likely lowers blood pressure."

• A significance level (α) marked on the curve.
• The area under the curve beyond the significance level represents the rejection region.
• If the test statistic falls in the rejection region, the result is significant (reject H₀).
• If the test statistic falls outside the rejection region, the result is not significant (fail to reject H₀).

Key Terms in Significance Testing | Term | Definition | |------|------------| | Test Statistic | A value calculated from the data that measures deviation from H₀. | | Critical Value | A threshold value that the test statistic is compared against. | | Significance Level (α) | The threshold for determining if results are significant. | | Rejection Region | The area beyond the critical value where H₀ is rejected. | | Statistically Significant | A result that is unlikely to have occurred by chance. |

• Helps determine if results are meaningful.
• Helps in making data-driven decisions.
• Is crucial for validating research findings.
• Provides a formal framework for distinguishing real effects from chance.

Intermediate Level

The significance testing process can be summarized as:

Key point: The significance level (α) is chosen before collecting data. It represents the maximum probability of rejecting H₀ when it is actually true (Type I error).

Important Points

• - Significance Testing: A statistical method used to determine if a result is due to chance or if it is statistically significant.
• - Statistically Significant: A result that is likely due to some real effect or factor, not just random chance.
• - Process:
• 1. Formulate H₀ and H₁.
• 2. Choose a significance level (α).
• 3. Collect data.
• 4. Calculate the test statistic.
• 5. Compare test statistic to critical value (or p-value to α).
• 6. Make a decision (reject or fail to reject H₀).
• 7. Draw a conclusion.
• - Example: Exercise and blood pressure — p-value 0.005 → reject H₀ → exercise likely lowers blood pressure.
• - Key Terms: Test statistic, critical value, significance level, rejection region.
• - Purpose: To validate research findings and make data-driven decisions.

Remember

• - Significance Testing = Determining if a result is real or due to chance.
• - Statistically Significant = Likely due to a real effect.
• - Process: H₀H₁ → α → Data → Test statistic → Compare → Decide → Conclude.
• - Example: Exercise and blood pressure — p-value 0.005 → reject H₀.
• - Key Terms: Test statistic, critical value, significance level.
• - Purpose: Validate findings, make data-driven decisions.
• - Statistical significance ≠ practical significance.

Common Mistakes

• 1. Thinking statistical significance means practical significance — A result can be statistically significant but have a very small effect.
• 2. Confusing the significance level with the p-value — The significance level (α) is chosen before the test; the p-value is calculated from the data.
• 3. Thinking a non-significant result proves H₀ — A non-significant result only means there is not enough evidence to reject H₀; it does not prove H₀ is true.
• 4. Ignoring the assumptions of the test — Significance tests have assumptions (e.g., normality, independence) that must be met for the results to be valid.
• 5. Choosing the significance level after seeing the data — The significance level should be chosen before collecting data to avoid bias.
• 6. Thinking significance testing is the same as hypothesis testing — Significance testing is a specific part of hypothesis testing that focuses on determining statistical significance.

Quick Revision

• Significance Testing: A statistical method used to determine if a result is due to chance or if it is statistically significant. Statistically significant means it is likely due to some real effect or factor.
• Key Points:
• Significance testing helps determine if results are meaningful.
• It compares the test statistic with a critical value.
• Setting a significance level helps decide the threshold for importance.
• Significance testing helps in making data-driven decisions.
• It is crucial for validating research findings.
• Example: Research question: "Can exercising for six weeks help lower blood pressure?"
• H₀: Exercise doesn't affect blood pressure.
• H₁: Exercise lowers blood pressure.
• Data: People who exercised for six weeks had a reduction in blood pressure of 8 mmHg.
• Significance Test: Check if this result is real or just a coincidence. The p-value is calculated.
• Result: P-value is 0.005 (0.5% chance of getting this result by chance).
• Conclusion: Since p-value is very small (0.005), reject H₀ and accept H₁. Exercise likely lowers blood pressure.
• Fig. 4.15: Shows a normal distribution curve with a significance level (α) and rejection region.

60-Second Revision

• Significance Testing = Determining if a result is real or due to chance.
• Statistically Significant = Likely due to a real effect.
• Process: H₀H₁ → α → Data → Test statistic → Compare → Decide → Conclude.
• Example: Exercise and blood pressure — p-value 0.005 → reject H₀.
• Key Terms: Test statistic, critical value, significance level.
• Warning: Statistical significance does not necessarily mean practical significance.

Frequently Asked Questions (FAQs)

Q1. What is significance testing? Answer: Significance testing is a statistical method used to determine if a result is due to chance or if it is statistically significant. Statistically significant means it is likely due to some real effect or factor.

Q2. What does "statistically significant" mean? Answer: Statistically significant means a result is likely due to some real effect or factor, not just random chance. It means the observed result is unlikely to have occurred by chance.

• Formulate H₀ and H₁.
• Choose a significance level (α).
• Collect data.
• Calculate the test statistic.
• Compare the test statistic to the critical value (or p-value to α).
• Make a decision (reject or fail to reject H₀).
• Draw a conclusion.

Q4. What is a test statistic? Answer: A test statistic is a value calculated from the data that measures how much the data deviates from the null hypothesis. It is used to determine whether the result is statistically significant.

Q5. What is a critical value? Answer: A critical value is a threshold value that the test statistic is compared against. If the test statistic exceeds the critical value, the result is statistically significant.

Q6. What is a significance level (α)? Answer: The significance level (α) is the threshold for determining if results are significant. It is the maximum probability of rejecting H₀ when it is actually true. A common choice is α = 0.05 (5%).

Q7. What is the exercise and blood pressure example in the textbook? Answer: The example is: "Can exercising for six weeks help lower blood pressure?" The null hypothesis is "Exercise doesn't affect blood pressure," and the alternative hypothesis is "Exercise lowers blood pressure." Data is collected from people who exercised for six weeks, showing a reduction of 8 mmHg. The p-value is calculated as 0.005, which is very small. Therefore, the null hypothesis is rejected, and the alternative hypothesis is accepted. Exercise likely lowers blood pressure.

Q8. What does a p-value of 0.005 mean in the exercise example? Answer: A p-value of 0.005 means there is only a 0.5% chance of getting this result by chance. Since this is very small, the null hypothesis is rejected, and it is concluded that exercise likely lowers blood pressure.

Q9. What is the rejection region in significance testing? Answer: The rejection region is the area beyond the critical value where the null hypothesis is rejected. If the test statistic falls in the rejection region, the result is statistically significant.

Q10. Why is significance testing important? Answer: Significance testing is important because it helps determine if results are meaningful, helps in making data-driven decisions, and is crucial for validating research findings.

Practice Questions

Short Response Questions (SRQs)

• Define significance testing and explain its purpose.
• What does "statistically significant" mean?
• What is the process of significance testing?
• What is a test statistic? What is a critical value?
• Describe the exercise and blood pressure example from the textbook.

Extended Response Questions (ERQs)

• Discuss the concept of significance testing in detail. Why is it important for validating research findings?
• Explain the significance testing process step by step. How is the decision made to reject or fail to reject the null hypothesis?
• Analyze the exercise and blood pressure example. How does the p-value lead to a conclusion?
• You are conducting a study to determine if a new teaching method improves student performance. Describe how you would use significance testing to evaluate the results.
• Compare and contrast significance testing with p-values. How do they work together in hypothesis testing?

Multiple Choice Questions (MCQs)

• Q1. What is significance testing?
• A. A method to determine if a result is due to chance or is statistically significant
• B. A method to collect data
• C. A method to delete data
• D. A method to store data
• Correct Answer: A
• Explanation: Significance testing is a statistical method used to determine if a result is due to chance or if it is statistically significant.

• Q2. What does "statistically significant" mean?
• A. The result is likely due to a real effect
• B. The result is definitely due to chance
• C. The result is invalid
• D. The result is not important
• Correct Answer: A
• Explanation: Statistically significant means the result is likely due to some real effect or factor, not just random chance.

• Q3. What is a test statistic?
• A. A value calculated from the data that measures deviation from H₀
• B. A value that is always 0.05
• C. A value that is always 0.5
• D. A value that is always 1
• Correct Answer: A
• Explanation: A test statistic is a value calculated from the data that measures how much the data deviates from the null hypothesis.

• Q4. What is a critical value?
• A. A threshold value that the test statistic is compared against
• B. A value that is always 0.05
• C. A value that is always 0.5
• D. A value that is always 1
• Correct Answer: A
• Explanation: A critical value is a threshold value that the test statistic is compared against.

• Q5. What is the significance level (α)?
• A. The threshold for determining if results are significant
• B. The average of the data
• C. The middle value of the data
• D. The most frequent value of the data
• Correct Answer: A
• Explanation: The significance level (α) is the threshold for determining if results are significant.

• Q6. In the exercise example, what is the null hypothesis?
• A. Exercise lowers blood pressure
• B. Exercise doesn't affect blood pressure
• C. Exercise increases blood pressure
• D. Exercise has no effect on the body
• Correct Answer: B
• Explanation: The null hypothesis is "Exercise doesn't affect blood pressure."

• Q7. In the exercise example, what is the alternative hypothesis?
• A. Exercise lowers blood pressure
• B. Exercise doesn't affect blood pressure
• C. Exercise increases blood pressure
• D. Exercise has no effect on the body
• Correct Answer: A
• Explanation: The alternative hypothesis is "Exercise lowers blood pressure."

• Q8. In the exercise example, what is the p-value?
• A. 0.005
• B. 0.05
• C. 0.5
• D. 0.0005
• Correct Answer: A
• Explanation: The p-value is 0.005 (0.5% chance of getting this result by chance).

• Q9. What is the conclusion of the exercise example?
• A. Exercise likely lowers blood pressure
• B. Exercise does not affect blood pressure
• C. Exercise increases blood pressure
• D. The data is invalid
• Correct Answer: A
• Explanation: Since the p-value is very small (0.005), the null hypothesis is rejected, and it is concluded that exercise likely lowers blood pressure.

• Q10. What is the rejection region in significance testing?
• A. The area beyond the critical value where H₀ is rejected
• B. The area where H₀ is accepted
• C. The area where H₁ is rejected
• D. The area where the data is invalid
• Correct Answer: A
• Explanation: The rejection region is the area beyond the critical value where the null hypothesis is rejected.

• Q11. Why is significance testing important?
• A. It helps determine if results are meaningful
• B. It makes data more complex
• C. It deletes data
• D. It stores data
• Correct Answer: A
• Explanation: Significance testing helps determine if results are meaningful and is crucial for validating research findings.

• Q12. What does a p-value less than the significance level (α) indicate?
• A. The result is statistically significant, so reject H₀
• B. The result is not statistically significant, so fail to reject H₀
• C. The data is invalid
• D. The test statistic is negative
• Correct Answer: A
• Explanation: If the p-value is less than the significance level (α), the result is statistically significant, and H₀ is rejected.