What do we mean by "significant" results? Is a "significant" result an important one? Is a "significant" result a "real" one? There is recognized ambiguity in the word "significance" used in scientific publications.1 It was suggested that if one wants to use the words "significance" or "significant" in scientific publications, then a reasonable approach is to clarify in each instance the type of "significance" being considered, for example, statistical significance, practical significance or clinical significance.1

Let's consider a recently published JBI systematic review.2 Reviewers specified in the results section of the executive summary that "there were no statistically significant differences observed in[horizontal ellipsis]" and that "there were no statistically significant differences in either[horizontal ellipsis]".2(p.129) There is no ambiguity with regards the type of significance reported. It is clear for readers that statistical significance is discussed, not other type of significance. This is an example of clear explicit reporting of "significant" results.

Let's explore another published JBI systematic review.3 Reviewers specified in the results section of the executive summary that "No significant differences between groups for all reported outcomes were observed".3(p.292) In this example, it is not clear what type of significance is being considered. Is this statement about statistical significance? We don't know. Maybe reviewers are reporting statistical significance; however, we don't know if this is a statement about the magnitude of the difference or the substantive difference between the results from the compared groups. This is an example of ambiguous reporting of "no significant" results.

It is reasonable to consider that most researchers and reviewers use the term statistical significance with the same meaning.4 In simple words, statistical significance refers to results of statistical tests (significance tests as proposed by Fisher or hypotheses tests as proposed by Neyman and Pearson).4 The straightforward meaning of statistical significance is that results of statistical tests indicate statistical evidence against the null (statistical) hypothesis.4 On this basis, a decision has been made to reject the null hypothesis and, in the case of a hypothesis test, to accept the alternative hypothesis.4 Also, statistical significance means that if confidence intervals were used, the null value (zero for effect sizes of differences and one for effect sizes of ratios) was not included between the lower and the upper limits of the confidence interval.4

What is "practical" significance? What is "clinical" significance? We should be aware that different researchers and authors use the terms clinical significance and practical significance with different meanings, and that what some authors call clinical significance may refer to what other authors call practical significance.4 One approach is the following: in simple words, practical significance is about the magnitude and direction of effects as judged by researchers or practitioners, and clinical significance is about the impact on the patient's life as judged by patients.4 Regardless of the terms used. it is important to acknowledge the relevance of the two different perspectives: one perspective is from the researchers and the other from the patients.

Practical significance is indicated by the magnitude and direction of different effect sizes.4 Practical significance refers to the significance of the results based on the importance given by the researcher or practitioner to the magnitude and direction of association measures, correlation measures, risk measures and so on, that is, how big is the difference? How large is the change? How big is the impact from researchers or practitioners point of view?4 For example, a relative risk of 2 indicates a different magnitude compared to a relative risk of 5. Some researchers and practitioners consider a relative risk of 2 as indicating a practically significant result (or large effect), and a relative risk of 4 as indicating a very significant result (or large effect), from a practical point of view.4 There are consensus international conventions and recommended guidelines or practices for the interpretation of the practical significance of effect sizes, suggesting what is considered a "small", "large", and "very large", or "moderate" and "strong" effect (or correlation or association) in different areas or domains of research such as epidemiology or social sciences.4 However, we should be aware that with regards the interpretation of practical significance of the results, there is no such thing as "one size fits all".

In simple words, clinical significance is about the perceived benefits by the patients, that is the real effects in the patient's life, from the patient's perspective.4 For example, a reduction of pain intensity from 100 points to 80 points on a scale from zero to 100 may be considered practically significant by researchers or practitioners but from a clinical point of view is this change is a significant change from patient's perspective? If this change is judged by the patient to represent a real benefit in his or her life, then the result is considered to be clinically significant. Let's consider another example: survival of cancer patients. Practitioners may consider that only an increase of survival of at least three months is a result with practical significance; however, patients may consider that even an increase of survival of just one month is significant from their point of view, that it is worth it and is clinically significant.

As review authors and researchers actively engage in maximizing the quality of conduct and reporting of systematic reviews through editorial comment provided from peer reviews of manuscripts, it rests with us to ensure that in JBI systematic reviews there are no ambiguous statements about "significant" results. Let's not accept the use of "significance" without a qualifier that clarifies the type of "significance" (statistical, clinical or practical) and without clear explanation detailing how clinical or practical significance is defined in the review, and how and why it is interpreted in a specific way.

References