Statistical significance vs practical significance
In research, data is evaluated through statistical significance and practical significance. Statistical significance is the term we frequently use without true understanding. Statistical significance means an inferential statistic that links sample data to population data (Rosen and DeMaria, 2011). The first step of testing statistical significance is to form a hypothesis. Hypothesis is simply an idea that can be tested. There is no fixed definition for a null hypothesis. Among many definitions, null hypothesis is a statement of “no effect” or “no difference” (Numeracy Project, University of Guelph), and denoted as H0. Alternative hypothesis is denoted as H1 and is the opposite of the null hypothesis. Researchers define the alternative hypothesis as an important phenomenon yet not confirmed. Therefore, researchers work to reject the null hypothesis, in other words, to nullify commonly accepted knowledge. Researchers like you and me get excited when we discover a statistically significant finding. This means that your findings are reliable. However, this is only one side of the coin. Even if you fail to reject the null hypothesis after conducting some appropriate statistical testing, your finding is still not statistically significant. This does not mean that your research is worthless, but commonly accepted facts (i.e. H0) seems to be the winner in this case. That could still be an important finding too!
Before accepting whether the findings are statistically significant or not, researchers must settle for some level of confidence. Commonly used confidence level is 95% meaning that researchers want to be correct about the findings 95% of the time, so that the chance of being incorrect is as low as 5%. Note that there are different tests where we can perform statistical analysis and obtain p-value or the probability value. It is imperative to use the correct statistical tool to calculate p-value. If not, results would be misleading and no longer valid. Suppose that the pre-specified significance level, commonly denoted by the Greek letter alpha (), is 0.05 and the calculated p-value is 0.03. This implies that there is a 3% chance of the null hypothesis being correct. If the calculated p-value is greater than 0.05, we cannot reject the null hypothesis. The p-value should always be less than the pre-specified confidence level to reject the null hypothesis. Researchers typically expect a small p-value so that they can reject the null hypothesis. By rejecting the null hypothesis, researchers accept the alternative hypothesis.
Let’s talk about practical significance. Although statistical significance provides a hint of mathematical importance of the results, practical significance considers the importance of results to the public. In other words, practical significance looks at the effect size (the strength of the relationship between the variables) and relationships between controls and tests which are important for researchers and the audience. Formal statistical tests cannot tell you whether your findings are important to your audience. Therefore, it is required to explain why the findings are important. Researchers can emphasize the practical significance of their findings using several tools. Confidence intervals are one such tool that determines practical significance of findings. A confidence interval is a range of values which gives the researcher the opportunity to calculate population values from sample statistics. There is a margin of error associated with estimated values. From the margin of error (i.e. how many percentage points your results differ from the real population), you can express the importance of your findings to the audience.
The takeaway message from this brief article is that the value of both statistical and practical significance is imperative for a high quality research output. So, it’s very important to continue polishing your statistics!
References:
Rosen, B.L., and DeMaria, A.L. (2011). Statistical significance vs. practical significance: an exploration through health education. American Journal of Health Education, 43(4): 235–241
Statistical versus practical significance, University of Guelph Numeracy Project.
