Developments in the Reporting of Score Reliability within Counseling Assessment, Research, and Evaluation

At the turn of the century, there was considerable conversation in Measurement and Evaluation in Counseling and Development (MECD) and other periodicals regarding approaches to reporting internal consistency reliability data within scientific reports (Fan & Thompson, 2001; Henson, 2001; Henson & Thompson, 2002; Onwuegbuzie & Daniel, 2002; Thompson & Snyder, 1998; Thompson & Vacha-Haase, 2000). The collective position was that authors had made strides in the inclusion of these data; however, small distinctions related to characterization and interpretation were obscuring the nature of relationships between variables and intervention outcomes within the broader literature base. Taken together, their recommendations across these commentaries provided encouragement and technical guidance for authors to:

1. Recognize reliability data as a function of test scores, not the tests themselves;

2. Report sample-specific reliability estimates rather than relying on inductions from other studies;

3. Include confidence intervals (CIs) for the overall sample reliability estimate, as well as, key subgroups when indicated by the analytic plan; and

4. Use visual displays to complement textual reporting of reliability estimates and CIs.

A cursory review of the literature in our field indicates that we have made substantial progress in the conceptualization of reliability metrics as data metrics rather than a property of tests. Evidence for this position can be found upstream in the many textbooks used in counseling and related educational programs (Balkin & Kleist, 2022; Barrio Minton & Lenz, 2019; Sheperis et al., 2023; Wester & Wachter Morris, 2018), as well as, guiding documents for the use of assessments and their scores (American Educational Research Association et al., 2014; Lenz et al., 2022). Thus, it is reasonable to conjecture that as a profession our current position along the continuum ranging from this test reliable to these test scores provide evidence for a defensible degree of reliability we are nearer to the latter than the former. While the work may not be done, it is clear that the charge has been accepted and that a paradigm shift is in process.

It can also be argued that a similar degree of progress has been made in the reporting of sample-specific reliability estimates within primary studies. Early reviews of sample-specific score reliability reporting revealed strikingly low rates of non-reporting ranging from 56 to 80% of published reports (Hanson et al., 2002; Miller et al., 2009; Yin & Fan, 2000). Vacha-Haase et al. (2003) review also highlighted the prevalence of reliability induction practices wherein researchers attribute the psychometric properties of test scores from a normative or alternative sample to theirs even when the sample does not reflect a preponderance of overlap among reported intersecting identities. It is a subtle distinction but this induction is misleading due to its disregard of the interaction between sample characteristics, score variability, and related internal consistency estimates that can lead to the attenuation of observed effects. More recent reliability generalization reviews have indicated a greater presence of sample-specific reporting ranging from 57 to 71% of published reports and thus a 29–43% degree of non-reporting or reliability inductions (Lenz et al., in press; McKay et al., 2021; Vincent et al., 2019). Given the American Psychological Association’s (2019) Journal Article Reporting Standards guidance to include sample-specific reliability estimates within primary study reports and the related observation of these practices by peer-reviewed outlets, including MECD, it is possible that rates of omissions and inductions will continue to decrease over time.

In contrast to these developments, a review of the professional counseling literature will indicate modest instances of CIs reported for total sample and subgroup score reliability estimates and minimal cases of visual representations for the related data. The increased capacity of statistical software packages such as IBM SPSS Statistics, JASP, and R to compute multiple forms of reliability with CIs across subgroups provides the greatest opportunity to promote transparency in reporting. Table 1 provides an illustrative example of one approach to including these details along with commonly reported descriptive statistics such as commonly displayed in relation to predictive analyses with a single group of participants. These data depict a scenario wherein the point estimates representing internal consistency reliability are relatively similar across the four predictor variables and within a range that may be regarded as good or acceptable by some common interpretative benchmarks. However, the inclusion of the related CIs provides a clearer depiction of the variation or stability of the reliability coefficients. For example, hypothetical scores on the Shame variable are represented by an alpha coefficient of 0.82 with the expectation that we could be 95% certain that the true value falls somewhere between 0.71 and 0.93. Without the reporting and interpretation of these CIs, we may be content with categorizations of the internal consistency of scores as good when in fact the true estimate may lie at the thresholds of what is considered acceptable or excellent. The inclusion of visual aids may have the virtue of making these relationships more readily observable to readers. These analyses may bear particular relevance to MECD authors who are investigating multi-factor expressions of test scores or measurement invariance between sample subgroups.

Table 1. Illustrative Display of Descriptive Statistics, Bivariate Correlations, and Internal Consistency Reliability Estimates with 95% Confidence Intervals.

Overall, it is clear that counseling researchers have made considerable strides when answering the call for more rigorous reporting practices delivered more than two decades ago. The greatest developments have been actualized in the conceptualization of reliability as a property of data rather than tests and the inclusion of sample-specific reliability estimates within scientific reports. With the availability of current statistics packages, addressing the reporting and interpretation of related confidence intervals may be within our collective developmental potential, as well. The use of visual aids will can only support transparency about the nature of these data.

This special issue of MECD dedicated to score reliability is intended to re-invigorate these early conversations and provide a platform for those that may advance our current status. Kalkbrenner (2024) provided a non-technical review of several internal consistency estimates and guidelines for selecting those that suit particular data characteristics. Balkin et al. (2024) provided an overview of strategies for reporting reliability across diverse samples of participants. Cook and Wind (2024) reviewed the fundamental assumptions and practices for estimating score reliability and precision based on item response theory. Coleman et al. (2024) described the rationale and computational procedures for using intercoder reliability across several investigative paradigms. The remainder of the articles within the issue illustrate strategies to explore the boundaries of score reliability generalization across several measures frequently used in counseling research and practice.

Disclosure statement

No potential conflict of interest was reported by the author(s).