Editorial Statement About JCCAP’s 2023 Special Issue on Informant Discrepancies in Youth Mental Health Assessments: Observations, Guidelines, and Future Directions Grounded in 60 Years of Research

Issue 1 of the 2011 Volume of the Journal of Clinical Child and Adolescent Psychology (JCCAP) included a Special Section about the use of multi-informant approaches to measure child and adolescent (i.e., hereafter referred to collectively as “youth”) mental health (De Los Reyes, 2011). Researchers collect reports from multiple informants or sources (e.g., parent and peer, youth and teacher) to estimate a given youth’s mental health. The 2011 JCCAP Special Section focused on the most common outcome of these approaches, namely the significant discrepancies that arise when comparing estimates from any two informant’s reports (i.e., informant discrepancies). These discrepancies appear in assessments conducted across the lifespan (Achenbach, 2020). That said, JCCAP dedicated space to understanding informant discrepancies, because they have been a focus of scholarship in youth mental health for over 60 years (e.g., Achenbach et al., 1987; De Los Reyes & Kazdin, 2005; Glennon & Weisz, 1978; Kazdin et al., 1983; Kraemer et al., 2003; Lapouse & Monk, 1958; Quay et al., 1966; Richters, 1992; Rutter et al., 1970; van der Ende et al., 2012). Thus, we have a thorough understanding of the areas of research for which they reliably appear when clinically assessing youth. For instance, intervention researchers observe informant discrepancies in estimates of intervention effects within randomized controlled trials (e.g., Casey & Berman, 1985; Weisz et al., 2017). Service providers observe informant discrepancies when working with individual clients, most notably when making decisions about treatment planning (e.g., Hawley & Weisz, 2003; Hoffman & Chu, 2015). Scholars in developmental psychopathology observe these discrepancies when seeking to understand risk and protective factors linked to youth mental health concerns (e.g., Hawker & Boulton, 2000; Hou et al., 2020; Ivanova et al., 2022). Thus, the 2011 JCCAP Special Section posed a question: Might these informant discrepancies contain data relevant to understanding youth mental health? Suppose none of the work in youth mental health is immune from these discrepancies. In that case, the answer to this question strikes at the core of what we produce―from the interventions we develop and implement, to the developmental psychopathology research that informs intervention development.

Precisely 12 years after the 2011 JCCAP Special Section, we dedicated Issue 1 of JCCAP’s 2023 Volume to a Special Issue on informant discrepancies in youth mental health assessment (see De Los Reyes & Epkins, 2023). In this “sequel” to the 2011 JCCAP Special Section, we took stock of what we have learned about these informant discrepancies in the intervening years and sought to build off of this accrued knowledge base. Much of this knowledge was made possible by an increase in studies that went beyond comparing informants’ reports to each other, and examined patterns observed in these reports in relation to scores taken from assessment batteries that leverage multiple modalities of measurement. These measurement modalities have included not only surveys but also interviews, performance-based tasks, behavioral observations, and readings from physiological devices (see also De Los Reyes, Wang, et al., 2023).

Based on research published in the ensuing years, we can definitively answer the question posed by the 2011 JCCAP Special Section: Yes, informant discrepancies often contain domain-relevant information; specifically, data that directly pertain to the youth mental health domains about which informants provide reports. In light of the answer to this question, the 2023 JCCAP Special Issue on informant discrepancies posed new questions with important implications for how youth mental health researchers integrate or model multi-informant data:

• Does the possibility that informant discrepancies contain domain-relevant information require reconceptualizing our paradigms for measurement validation?

• Which youth mental health domains require more research about the informant discrepancies that result from assessing them?

• Does the available evidence about informant discrepancies inform the development of guidelines for integrating multi-informant data?

Multiple contributions to the 2023 JCCAP Special Issue address the first two of these questions. In this editorial statement, we addressed the third question by synthesizing the work produced by this Special Issue and considering the preceding 60 years of research on informant discrepancies in youth mental health. The result is a “top-10” list, comprised of 5 observations, 2 guidelines, and 3 directions for future research. Herein, we delineate these observations, guidelines, and directions, and chart a path for the next generation of research on informant discrepancies in youth mental health. Thorough reviews of the evidence supporting the observations, guidelines, and directions below are available in this Special Issue and elsewhere (see De Los Reyes et al., 2013; De Los Reyes, Talbott, et al., 2022; De Los Reyes, Tyrell, et al., 2022; De Los Reyes, Wang, et al., 2023).

Observations About Informant Discrepancies

Observation #1:

Informant discrepancies commonly manifest when comparing multiple informants’ reports about youth mental health.

In three key respects, the 60 years of research on informant discrepancies provides us with a great deal of data about this clinical phenomenon. First, across these decades, the discipline of youth mental health has undergone many changes to the theories driving research, the development of interventions informed by these theories, the instruments used to assess mental health, and the informants relied upon to provide reports on these instruments. Second, within these 60 years of research, various domains have been the subject of estimating the presence and extent of informant discrepancies, including symptom domains (e.g., anxiety, depression, conduct problems, attention-deficit/hyperactivity disorder [ADHD]), risk and protective factors (e.g., family functioning, peer relations), physical health (e.g., pubertal status, pain), domains relevant to life interference (e.g., psychosocial impairments, quality of life), and psychosocial strengths (e.g., social skills), among others. Hundreds of studies document that informant discrepancies are evident, and often significant, when assessing every one of these domains.

Yet, the large body of work on informant discrepancies also reveals a third intriguing feature. Specifically, not every assessment produces informant discrepancies, and among those that do, they are not all created equal. For instance, some parents and teachers agree on the presence and level of ADHD symptoms displayed by the youth about whom they provided reports. Further, when informant discrepancies occur, there are profound “individual differences” in the composition of this phenomenon and how it “behaves” when comparing pairs of informants’ reports to each other. For example, sometimes discrepancies occur in the direction of one informant rating a greater degree of the domain (e.g., family conflict) relative to the other informant (e.g., adolescent > mother), whereas another pair of informants might rate that domain in a fundamentally distinct way (e.g., mother > adolescent; adolescent and mother provide similar reports). What this means is that these individual differences manifest at both the sample level (e.g., mean comparisons between parent and teacher reports) as well as at the level of individual cases (e.g., parent > teacher for one child; teacher > parent for a second child; parent similar to the teacher for a third child). Taken together, we need to not only understand why informants’ reports disagree, but also probe those instances in which informants’ reports do agree. We also need to understand these patterns of informant discrepancies and informant agreement across multiple levels of analysis (e.g., case, sample, meta-analysis of multiple studies). This reality of how informant discrepancies manifest within and across investigations informs the discussion of the next set of observations.

Observation #2:

Multi-informant, multi-modal approaches to measurement meaningfully contribute to our understanding of youth mental health.

Empirical observations of informant discrepancies trace back at least to the 1950s (Lapouse & Monk, 1958). In the present day, these discrepancies are readily apparent in any research or clinical setting in which one takes a multi-informant approach to assessment. We see two commonalities between the multi-informant assessments conducted decades ago and present-day assessment practices. First, in both of these periods we did not have a single, “gold standard” instrument for assessing any one mental health domain, such as anxiety, depression, conduct problems, or autism. Given the multifaceted nature of mental health domains, it is safe to wager against ever discovering definitive instruments for specific mental health domains. Second, there exists no absolute standard by which to claim that scores taken from a single instrument of a mental health domain (e.g., a self-report survey for youth depression) are “more valid” than scores taken from instruments developed to assess that same domain (e.g., parent report survey for youth depression). Thus, youth mental health studies benefit greatly from comprehensive approaches to assessment.¹ By “comprehensive,” we mean developing and implementing batteries of instruments that (a) leverage multiple modalities of measurement (e.g., surveys, interviews, performance-based tasks, behavioral observations, readings from physiological devices); (b) involve the input of multiple informants or sources of measurement, which could include youth, parents, teachers, peers, and trained and untrained research personnel; and (c) rely on the judgments of professionals and/or evidence-based algorithms for interpreting the results.²

Observation #3:

Informant discrepancies do not appear to be fully explained by issues with measurement reliability.

When discussing what informant discrepancies reflect, we must first consider if they reflect sources of variance in informants’ reports that are irrelevant to understanding the youth mental health domains about which informants provide reports (i.e., measurement confounds). There exist established standards for evaluating the psychometric properties of scores taken from instruments used to assess relevant domains, including standards applied to work published in JCCAP (e.g., De Los Reyes & Langer, 2018). Further, many of our instruments meet or exceed these standards (e.g., Hunsley & Mash, 2018). Suppose these instruments involve collecting reports from informants (e.g., parents, teachers, and youth), and subsequently computing numerical scores based on these reports. In that case, one cannot divorce the psychometric properties of scores taken from the instruments, from the informants whose reports led to these scores. It logically follows that if we apply psychometric standards to scores taken from an instrument, and determine that the instrument produces scores with an acceptable degree of precision (e.g., as indicated by procedures for estimating test-retest reliability or internal consistency of scores), then we cannot also say that the informants whose reports resulted in these scores produce unreliable reports.

This logic flow is essential to convey, because it strikes at the heart of a critical interpretation of informant discrepancies: that they stem from issues surrounding random score variations or measurement errors (Campbell & Fiske, 1959; De Los Reyes, 2011). This interpretation conflicts with several aspects of the literature on informant discrepancies in youth mental health assessments. Consider that these discrepancies manifest to a robust degree, even when scores are taken from extensively studied, well-established instruments used the world over in research and service delivery settings. Further, the number of well-established instruments with evidence of score reliability on their behalf has increased, not decreased, over the last few decades, both in terms of instruments designed to assess mental health dimensionally (e.g., symptom checklists) and discretely (e.g., structured and semi-structured interviews; see Hunsley & Mash, 2018; Youngstrom et al., 2017). As others have noted (Achenbach, 2020), current meta-analytic reviews of these informant discrepancies consist largely of studies that compared informants’ reports taken using well-established instruments. If issues with score reliability accounted for the informant discrepancies we observe, then we should expect correspondence between informants’ reports to have increased over the decades of work on these issues. Yet, the mean estimate of informant discrepancies in youth mental health assessments has not changed in decades, as indicated by identical mean estimates of cross-informant correspondence (i.e., indexed using Pearson r) based on studies conducted between 1960 and 1986 (r = .28; Achenbach et al., 1987) and by estimates based on studies conducted between 1989 and 2014 (r = .28; De Los Reyes et al., 2015). This estimate is remarkably robust (r = .28, with a relatively tight 95% confidence interval of r = .24–.31) across studies conducted in over 30 countries that traverse six continents (De Los Reyes et al., 2019). Put simply, none of the evidence indicates that one can interpret these informant discrepancies as merely a reflection of informants making unreliable reports.

Observation #4:

Informant discrepancies do not appear to be fully explained by issues with measurement validity.

Similar to our discussion about score reliability, we can look to established conventions for drawing valid inferences from studies―and ruling out threats to the validity of these inferences―to judge empirical work testing whether informant discrepancies reflect systematic rater biases. We apply these standards of inference in many other research contexts (see Kazdin, 2022). For instance, when a client experiences improvements in mental health functioning over the course of treatment, there are various parsimonious explanations for these improvements that have little to do with the “active ingredients” of the intervention itself. These explanations include history and maturation processes and regression to the mean effects. To rule out these possible explanations, researchers turn to study designs that facilitate drawing cause-and-effect inferences between receipt of a treatment and changes in client functioning, including controlled, between-group experiments and within-subjects experimental designs (e.g., ABAB). This logic applies to interpreting studies about rater biases in informants’ reports, and the likelihood that they fully account for the informant discrepancies observed in assessments of youth mental health.

In research on rater biases, perhaps the most-studied factors involve those that link informants’ mental states and the degree to which these states cause informants to rate youth inaccurately. For example, the depression→distortion hypothesis posits that informant discrepancies occur when informants’ mood states lead them to rate youth congruent with their mood, rather than rate youth in a way that reflects the reality of their functioning (Richters, 1992). Specifically, mood states like depression might lead an informant to attend to and encode more negative aspects of the youth’s behavior about whom they are making reports. Thus, an informant’s depression may cause this negatively biased processing of a youth’s behavior to “spill over” into the assessment process, such that the depressed informant provides more negative ratings of youth behavior, relative to the reports of non-depressed informants.

When viewed through traditional lenses of interpreting multivariate data, the depression→distortion hypothesis makes intuitive sense, mainly because such lenses presume that discrepancies among data sources signal threats to measurement validity (see Campbell & Fiske, 1959; Garner et al., 1956). Yet, all of the studies of the depression→distortion hypothesis harbor a flaw that clouds the interpretability of the findings. Specifically, informants who make reports in youth mental health assessments do not merely observe the mental health of the youth about whom they provide reports. Decades of research in developmental psychopathology indicate that some of the most commonly used informants in these assessments (i.e., parents and teachers) play significant roles in shaping youth behavior. For instance, caregivers’ mental health can impact parenting practices at home (Goodman et al., 2020), and teacher stress can play a role in instructional performance and classroom management strategies, thereby impacting factors that play key roles in youth mental health (e.g., learning and classroom behavior; Atkins et al., 2017). As such, links between informants’ mental states and discrepancies between informants’ reports of youth mental health could very well reflect the reality that informants’ functioning affects youth functioning. Importantly, no studies of the depression→distortion hypothesis have deconstructed variance in the purported biasing factor (e.g., a parent’s level of depressive mood symptoms) in a way that distinguishes aspects of the factor that likely reflect measurement confounds (e.g., memory distortions) from aspects of the factor that probably influence the mental health of the youth about whom informants make reports (e.g., difficulty sleeping, anhedonia, low mood, excessive guilt, irritability, difficulty concentrating). Without studies addressing this core issue, we can parsimoniously interpret depression→distortion effects as reflecting the notion that informants’ own mental states contribute to developing and maintaining youth mental health concerns (i.e., rather than reflecting a depression-related rater bias).

Observation #5:

Studies leveraging multi-modal validation testing strategies indicate that informant discrepancies often reflect domain-relevant information, or phenomena that inform our understanding about youth mental health domains.

With the last two observations, we ruled out the possibility that the informant discrepancies observed in youth mental health assessments primarily reflect measurement confounds like rater unreliability or rater biases. In part, we have ruled out these possibilities by looking to established conventions for judging the psychometric properties of scores taken from psychological instruments, the same instruments through which informants provide reports. To judge whether informant discrepancies reflect not measurement confounds but, rather, domain-relevant information, researchers have leveraged modified versions of the same validation testing strategies used to judge the validity of scores taken from psychological instruments. One contribution to this Special Issue described this validation paradigm at length (De Los Reyes, Wang, et al., 2023), and so herein we briefly describe some of its core features.

In youth mental health, assessors do not select their information sources at random. Such a random selection process would significantly increase the likelihood that any discrepancies observed would reflect unsystematic, random processes. Instead, a prevailing principle in youth mental health assessment involves selecting informants with unique perspectives about the youth undergoing evaluation, preferably with regard to social environments pertinent to the day-to-day lives of youth and their mental health (e.g., home, school, peer interactions). In these respects, assessors follow an intentional or strategic selection process when constructing multi-informant batteries to measure youth mental health. Suppose a vital feature of this process is to arrive at estimates taken from unique data sources. In that case, we can also arrive at two logical conclusions about what we should typically observe as outcomes of this process, namely that (a) strategic approaches to selecting informants would tend to produce informant discrepancies, and (b) domain-relevant factors ought to account for significant portions of the informant discrepancies observed.

Beyond strategically selecting informants, well-constructed validation tests also call for incorporating study design features that allow one to detect informant discrepancies that contain domain-relevant information. Well-constructed validation tests of informant discrepancies exhibit three key characteristics. First, the analytic models used to characterize informant discrepancies allow researchers to determine the incremental value of informant discrepancies, relative to the informants’ reports used to describe the discrepancies. Second, the validity criteria used to test informant discrepancies are based on assessment modalities that are independent of the informants providing reports (e.g., performance-based tasks, observed behavior, physiological readings, trained clinician ratings), to rule out identifying relations explained by shared method variance. Third, the validity criteria are designed to assess aspects of youth functioning relevant to understanding the mental health domains about which informants provide reports. This third characteristic rules out the possibility that the informant discrepancies are fully explained by measurement confounds because, by definition, a measurement confound (i.e., irrelevant variance) cannot relate to a domain-relevant facet of youth mental health.

The last 15 years of carefully conducted validation tests indicate that, rather than primarily reflecting measurement confounds, informant discrepancies often contain data relevant to understanding youth mental health domains. Studies incorporating the design features described previously have successfully detected domain-relevant informant discrepancies for multi-informant assessments of domains as diverse as autism, anxiety, depression, disruptive behavior, and parenting. These domain-relevant effects have been observed in multi-informant assessments of youth across developmental periods (e.g., preschool, middle childhood, adolescence) and with assessments conducted in a variety of settings (e.g., community settings, controlled laboratory observations, outpatient care, acute/inpatient care, foster care, prospective longitudinal studies, school-based services), as well as across cultures (see also De Los Reyes et al., 2019). The observed effects are compelling. Researchers have demonstrated links between informant discrepancies and such domain-relevant phenomena as youth responses to intervention, trained observers’ ratings of behaviors elicited from laboratory-controlled tasks, physiological functioning, and the specific contexts in which youth display behaviors indicative of mental health concerns. The introductory article to this Special Issue reviews this evidence (see Figure 1 of De Los Reyes & Epkins, 2023).

Guidelines for Integrating Assessment Data That Contain Informant Discrepancies

Guideline #1:

Using procedures to integrate or model multi-informant data requires understanding the assumptions underlying use of these procedures.

Up until this point, we discussed the first half of our “top-10” list, namely the 5 observations that we can derive from the last 60 years of research on informant discrepancies in youth mental health assessments. Informant discrepancies occur often and across assessments of all known domains of youth mental health. Further, these informant discrepancies cannot be fully explained by measurement confounds. Instead, they often contain domain-relevant data that directly pertain to our understanding of youth mental health. This means that informant discrepancies often contain relevant, and potentially important, information (i.e., valid data). These observations have important implications for researchers’ analytic procedures, namely those used to integrate or model data derived from multi-informant assessments. Indeed, not all procedures harbor assumptions consistent with the observations described previously. A fundamental assumption underlying the use of a variety of analytic procedures for integrating multi-informant data (e.g., composite scores, combinational algorithms, multi-rater structural models; see Table 1 of De Los Reyes & Epkins, 2023) is that when informant discrepancies occur within multi-informant data, they reflect measurement confounds (i.e., irrelevant variance). If a researcher applies these procedures to multi-informant data that violate this assumption, the consequence is clear: The procedures leave valid data on the “cutting room floor.” Stated another way, applying these kinds of procedures to data conditions that do not fit the usage assumptions depresses measurement validity. Thus, a key guideline derived from work in this Special Issue is that researchers who seek to integrate multi-informant data to address their study aims must (a) understand the assumptions underlying use of the analytic procedure they seek to apply to their data, and (b) empirically probe these assumptions. By “empirically probe,” we mean use validation testing strategies to discern the “fit” between the procedures used and the data conditions to which they will be applied.

Guideline #2:

When disseminating scientific findings, researchers who leverage a procedure for integrating or modeling multi-informant data must report validity evidence supporting use of that procedure.

Mental health researchers have increasingly become aware of the need for transparency in research practices, and delineating considerations surrounding why and how we address study aims (see Tackett et al., 2017). These issues apply to our analytic procedures for integrating or modeling multi-informant data. They are essential to how we justify implementing analytic procedures to address study aims, namely in the Method sections of peer review journal articles. In fact, we already have an infrastructure in place for increasing transparency in these respects.

Consider the two elements described for Guideline #1, namely the usage assumptions of an analytic procedure and executing empirical tests of the “fit” between that procedure’s usage assumptions and the data conditions to which it will be applied. Publication guidelines for psychology journals typically call for authors to report in the Methods sections of their articles descriptions of the psychometric evidence supporting the use of instruments implemented in the study to collect data and address study aims. As mentioned previously, standards exist for judging this evidence (De Los Reyes & Langer, 2018; Hunsley & Mash, 2018), and thus determining whether the available evidence supports using an instrument in a given study. Researchers weigh this evidence in relation to specific data conditions, most often characteristics of study participants to which instruments were administered. Examples of these characteristics include the developmental level of participants, and the informants used to collect data. Consequently, we already apply psychometric evidence and the assumptions underlying use of our instrumentation to determine whether we were justified in using such instrumentation to collect multi-informant data within specific data conditions (e.g., an individual study of a sample of adolescent clients). Guideline #2 simply extends this notion to the procedures that we ultimately use to integrate or model the data derived from this instrumentation.

As authors―and often reviewers―of peer-reviewed research, we should commit ourselves to scrutinizing the procedures used in research to integrate or model multi-informant data. As scholars, there is broad consensus on the links between study design and the appropriateness (or lack thereof) of drawing specific kinds of inferences from study findings (e.g., that it is inappropriate to draw causal inferences about treatment effects from open trial studies). Similarly, there should be little debate about the following notion. Suppose a researcher applies an integrative procedure to multi-informant data that contains domain-relevant informant discrepancies. If that procedure requires a user to assume that domain-relevant informant discrepancies do not exist, then the procedure impacts measurement, namely by producing under-validated data conditions. By “under-validated,” we mean that the procedure did not capture key sources of valid data, because it erroneously modeled domain-relevant data as measurement confounds.

If an analytic procedure is under-validated, then this has important implications for studies where that procedure is implemented. Historically, psychometricians have noted that levels of score reliability place constraints on observed effect sizes, such that a relatively low reliability limits the upper-bound magnitudes of effect sizes capable of being detected (see Nunnally & Bernstein, 1994). Similarly, when an analytic procedure fails to capture a source of valid data (e.g., domain-relevant informant discrepancies), that failure, in effect, imposes an artificial ceiling on the “universe” of variance available for scholarly inquiry (e.g., predicting or explaining variance in outcomes; see also Makol et al., 2020). Clearly, if prior work indicates that informant discrepancies may reflect domain-relevant information, then there should be heightened scrutiny for research that leverages analytic procedures that assume informant discrepancies cannot contain domain-relevant data. Addressing this issue requires us to monitor uses of these procedures, as peer reviewers and as authors. As with Observation #5, work in this Special Issue discusses examples of analytic procedures that facilitate integrating or modeling multi-informant data in a way that retains domain-relevant informant discrepancies (De Los Reyes, Wang, et al., 2023).

Directions for Research on Informant Discrepancies in Youth Mental Health Assessments

Direction #1:

Principles for constructing validation testing batteries to evaluate the fit of analytic procedures designed to integrate or model multi-informant data.

We close this editorial statement with three recommended directions for future research. Specifically, although informant discrepancies frequently manifest across assessments of all known youth mental health domains, it is important to note that researchers often implement the same or highly similar instruments to assess any given domain, such as diagnosis-specific surveys or structured interviews, as well as broadband behavioral checklists (see De Los Reyes & Langer, 2018). Another consideration is that the outcomes of measurement validation studies, particularly those focused on criterion-related validity, often change, depending on the criterion variable(s) used. This means that whether researchers detect domain-relevant information in informant discrepancies may be dictated by their practices for selecting (a) multi-informant instruments to assess youth mental health domains, and (b) validity criteria for detecting domain-relevant informant discrepancies. It logically follows that evidence supporting the use of a given integrative or modeling procedure may change, depending on the assessment setting or how informant discrepancies may contribute to explaining, characterizing, or predicting domain-relevant phenomena.

Consequently, these considerations call for developing principles to guide the selection of multi-informant instruments and validity criteria. These principles might include guidance on (a) which and how many informants to include in an assessment battery, (b) which and how many instruments to use to collect multi-informant data, (c) the domain-relevant features to capture in validity criteria, and (d) which modalities to use to construct validity criteria. By developing these principles, we might inform the construction of assessment batteries that are optimized for detecting domain-relevant informant discrepancies. Along these lines, this Special Issue includes an article that advanced a principle-based measurement validation paradigm designed to detect domain-relevant informant discrepancies (see De Los Reyes, Wang, et al., 2023). The founding principles of this paradigm may inform more granular principles for guiding specific decisions about informants, multi-informant instruments, and validity criteria. This is a crucial step in the next generation of work on informant discrepancies in youth mental health, and thus we encourage further thinking on these issues.

Direction #2:

Evidence-based approaches for integrating multi-informant data when assessing youth in longitudinal studies.

Several investigations point to the predictive utility of informant discrepancies, namely in their ability to predict domain-relevant outcomes across multi-wave longitudinal studies of various kinds (e.g., controlled trials, prospective longitudinal). Yet, these studies focused on informant discrepancies modeled at a single time point as longitudinal predictors; we do not have strong evidence to indicate whether changes in these discrepancies are also domain-relevant. The next generation of studies addressing the predictive utility of informant discrepancies necessitate multi-modal assessment designs, conducted across multiple assessment waves. These study designs will allow researchers to distinguish the changes in informant discrepancies that are accounted for by measurement confounds from the domain-relevant discrepancies. For example, prior work indicates that, under some circumstances, discrepancies between parent and teacher reports of youth externalizing behaviors index context-specific displays of these behaviors, such that if a parent endorses these behaviors and the teacher does not (or vice versa), the behaviors likely manifest in one context to a greater degree than the other context (e.g., home > school, school > home; for reviews, see De Los Reyes, Talbott, et al., 2022, De Los Reyes, Tyrell, et al., 2022). Consider a study that identified youth who, at the first wave of assessment, were displaying externalizing behaviors via parent reports to a greater degree than reports completed by teachers. What if, in that same study and at the second wave of assessment, the direction of the discrepancies switched (i.e., at wave 2, teachers reported externalizing behaviors to a greater degree than the parents)? An open question is whether these longitudinal shifts in characterizations of informant discrepancies (i.e., parent > teacher at wave 1 vs. teacher > parent at wave 2) are accurately indexing changes in the contexts in which youth display the behaviors about which informants provide reports (e.g., shift from predominantly home-based externalizing behavior at wave 1 over to predominantly school-based externalizing behavior at wave 2). Suppose the utility of informant discrepancies extends to understanding longitudinal changes in domain-relevant processes. In that case, this possibility has important implications for capitalizing on variance in multi-informant data that are typically characterized as error in longitudinal assessments. This issue merits further study.

Direction #3:

Evidence-based approaches for integrating multi-informant data when assessing individual clients.

One final direction for future research focuses on developing strategies for detecting domain-relevant informant discrepancies when they arise within individual cases, such as in youth mental health services. This current Special Issue and the 2011 JCCAP Special Section that preceded it primarily focused on understanding informant discrepancies and detecting domain-relevant discrepancies when working with multi-informant data at the sample level. Yet, we have known for several decades that informant discrepancies manifest with individual cases. No guidelines exist for understanding, interpreting, and integrating multi-informant data within mental health services. Notably, some of the same kinds of decision-making errors about multi-informant data that we see in research settings also appear to manifest in mental health service settings, examples of which include assuming that an “optimal informant” exists for assessing specific domains (e.g., youth self-report for depression), and integrating case-level data using procedures such as combinational algorithms (i.e., “and/or rules”) that, by construction, lose the kinds of domain-relevant information contained in informant discrepancies (e.g., data about context-specific symptoms; for a review, see De Los Reyes, Talbott, et al., 2022).

Consequently, this critical gap in the literature has implications for translating our knowledge about informant discrepancies to concrete strategies for understanding the outcomes of multi-informant assessments and developing procedures for integrating multi-informant data when making clinical decisions. Informant discrepancies can undermine treatment planning and implementation (see De Los Reyes, Talbott, et al., 2022). Thus, developing these new strategies may also improve prospects for accelerating youth psychotherapy effects, which do not appear to have improved across all the decades of randomized controlled trials (Weisz et al., 2019). For instance, consider research that reveals an evidence-based strategy for integrating sample-level estimates of multi-informant data to retain domain-relevant informant discrepancies (e.g., for predicting clinical outcomes). Beyond these sample-level strategies, we require refinements (or new strategies) that are optimized for case-level multi-informant assessment scenarios.

Concluding Comments

In this editorial statement, we briefly delineated a series of observations, guidelines, and directions for future research focused on the most common outcome of multi-informant assessments of youth mental health. Discrepancies commonly occur between estimates of youth mental health and conclusions drawn from these estimates, depending on the informant’s report that led to the estimates. This “top-10 list” is a culmination of not only this Special Issue of JCCAP, but also 60 years of research on these informant discrepancies. We know a great deal about these discrepancies, and thus can come to some definitive conclusions about how often they occur, what they likely reflect, and, importantly, what they are unlikely to reflect. Based on these observations about informant discrepancies, we can also arrive at some preliminary guidelines for integrating multi-informant data in youth mental health research, and improve our practices for transparency in reporting the justification or evidence base for the procedures we use to integrate these data. That said, we do have a great deal more to learn about these informant discrepancies. Thus, we articulated several directions for future research that are by no means exhaustive but nonetheless are of high priority in terms of advancing knowledge about this ubiquitous assessment phenomenon. A dozen years removed from JCCAP’s first Special Section about informant discrepancies (De Los Reyes, 2011), we remain curious about the work that lies ahead. We hope we have sparked curiosity in you as well, and that this statement informs your thinking about this most crucial area of youth mental health research.

Notes

¹ We do not wish to imply that all youth mental health assessments should be comprehensive in scope. The extent of an assessment battery is dictated in large part by the research question, and some research questions may not necessitate a comprehensive assessment (e.g., mental health screening; see Wakschlag et al., 2022). That said, for those assessments that require use of reports from more than one informant, comprehensive assessments as described in this statement facilitate drawing inferences as to what patterns of informant agreement and informant discrepancies reflect. In this respect, one contribution to this Special Issue advanced a measurement validation paradigm to guide construction of these comprehensive assessments (see De Los Reyes, Wang, et al., 2023).

² In arguing for the need to take a comprehensive approach to assessment, it is important to acknowledge that evidence-based techniques for engaging stakeholders and increasing the likelihood that they participate in the assessment process may facilitate optimizing use of this assessment approach (e.g., Becker et al., 2018). For example, when youth mental health studies are conducted in community settings, such as schools, it is important to ground the studies in collaborative practices with community members and families, to improve the likelihood that informant perspectives will represent the community from which they were sampled (see Garbacz et al., 2020).