Conflation between self-report and neurocognitive assessments of cognitive flexibility: a critical review of the Jingle Fallacy

ABSTRACT

Cognitive flexibility is a widely studied construct and is considered an important treatment target for several psychological disorders. The convergence of several independent fields of research has led to assumptions about the assessment of cognitive flexibility – assumptions that are not empirically supported and often conflate different notions of flexibility. This critical review discusses how the conflation of self-report and neurocognitive assessments has seemingly arisen from literature on eating disorders. We describe how seminal early observations of “inflexible” personality characteristics, communication competence research, and investigations of frontal lobe function after injury led to two methods of assessing “cognitive flexibility”. We discuss the impact that conflation of self-report and neurocognitive assessments has had on the field, and we provide recommendations for assessing cognitive flexibility in both research and clinical settings.

Key Points

What is already known about this topic:

(1) Self-report and neurocognitive assessments of “cognitive flexibility” are commonly used in research and clinical practice.

(2) There is uncertainty in the field about whether or not self-report and neurocognitive assessments of “cognitive flexibility” assess similar underlying constructs.

(3) Both clinicians and researchers are susceptible to the jingle fallacy.

What this topic adds:

(1) This narrative critique of the literature reveals that self-report and neurocognitive assessments of “cognitive flexibility” have gradually been conflated over time.

(2) Early research in eating disorders seems to have played an influential role in generating and reinforcing such conflation.

(3) The assumption that self-report and neurocognitive assessments of “cognitive flexibility” are causally linked has no empirical basis and yet it has been used to explain inflexible cognitions and behaviours in people with eating disorders.

KEYWORDS:

• Executive function
• neuropsychology
• personality
• feeding and eating disorders
• surveys and questionnaires

Cognitive flexibility has been broadly conceptualised as the ability to modify cognitive and behavioural strategies in response to novel, changing, or unexpected demands (Cañas et al., 2003; Deák, 2003). The component thought to be central to cognitive flexibility is set-shifting – the ability to switch between different “mental sets” (Tchanturia, Anderluh, et al., 2004). Other components often attributed to cognitive flexibility include “the ability to change thoughts and behaviours” or “the ability to generate alternatives”. Although the ability to be cognitively flexible relies on the intrinsic properties of the human cognitive system, whether cognitive flexibility is a separable component of executive function (Diamond, 2013), or an emergent property that derives from the interplay between different higher-order cognitive processes (Dajani & Uddin, 2015; Ionescu, 2012), is unresolved. Disagreements about the origins of cognitive flexibility likely stem from the lack of conceptual consensus that surrounds cognitive flexibility, as well as the multitude of assessments that have been used to assess this construct (Ionescu, 2012).

Cognitive flexibility is commonly assessed by self-report questionnaires and/or performance on neurocognitive tasks. Although “cognitive flexibility” – a mental ability to switch thinking – has been differentiated from “behavioural flexibility” – a switch in behaviour when environmental contingencies change – available assessments do not seem to make such a differentiation. Perhaps this reflects the view that cognitive and behavioural flexibility are so intertwined that they might be considered inseparable (Uddin, 2021). Common self-report assessments that target “the ability to change thoughts and behaviours” or “the ability to generate alternatives”, include the Cognitive Flexibility Scale (Martin & Rubin, 1995) and the Cognitive Flexibility Inventory (Dennis & Vander Wal, 2010).  Common neurocognitive tasks that assess a switch in behaviour when environmental contingencies change, include the Wisconsin Card Sorting Test (WCST; Grant & Berg, 1948; Heaton et al., 1993), the Trail Making Test (TMT; Reitan, 1955, 1958), and the Intra-Extra Dimensional Set Shift task (Robbins et al., 1998).

Despite empirical evidence that self-report and neurocognitive assessments do not relate well in neither clinical nor non-clinical, samples (e.g., Howlett et al., 2021, 2022; Johnco et al., 2014; Lounes et al., 2011; Miles et al., 2022; Sternheim et al., 2022), these assessments are often used interchangeably, and considered to be equivalent in assessing the construct of cognitive flexibility. Here, we provide a historical overview of how and why self-report and neurocognitive assessments that purport to assess cognitive flexibility may have become conflated over time. We will focus our discussion largely on eating disorders research, which appears to be the key field to have conflated these two assessment approaches, The conflation has since extended to other fields. We outline the unintended consequences of the conflation and we make recommendations for how future research and clinical practice might best respond.

The historical development of cognitive flexibility assessments

Early 20^th century researchers observed that those with frontal lobe damage performed worse than healthy controls on simple sorting tasks (Weigl, 1927). In a series of studies with rhesus monkeys, Settlage et al. (1948) observed that monkeys with frontal cortex lesions lost the ability to complete shifting problems even though they could solve non-shifting problems. On the basis of that work, Berg (1948) developed the WCST to assess “flexibility in thinking” in humans. The WCST is now widely accepted as a test of “cognitive flexibility” (Dennis & Vander Wal, 2010; Johnco et al., 2014; Tchanturia et al., 2012). Similarly, Reitan (1955, 1958) initially developed the TMT to detect brain damage, but since then it has been used to detect poor “cognitive flexibility” (Christidi et al., 2015; Dennis & Vander Wal, 2010; Johnco et al., 2014). Other neurocognitive tasks that have become popularised as cognitive flexibility assessments include the Haptic Illusion Task (Tchanturia et al., 2001; Uznadze, 1966), the Brixton Spatial Anticipation Task (Burgess & Shallice, 1997; Holliday et al., 2005; Lounes et al., 2011; Tchanturia et al., 2002; Tchanturia, Anderluh, et al., 2004), and the Alternate Uses Task (Eslinger & Gratten, 1993; Guilford, 1956).

The development of self-report questionnaires of cognitive flexibility followed a parallel, but separate trajectory. This trajectory seems to have commenced in interpersonal communication competence research. Specifically, Bochner and Kelly (1974) developed a conceptual framework, which proposed that competent communicators possess a set of core skills that are directly observable by others – one of them being the ability to display behavioural flexibility. Wiemann (1977) empirically investigated the relationship between what was called interaction management – a set of rules that guide social interactions (e.g., one person speaks at a time) – and aspects of communication competence, including behavioural flexibility, and found a positive correlation. With these ideas and findings in mind, Martin and Rubin (1994) went on to devise the Communication Flexibility Scale – a questionnaire that assesses a person’s ability to change their communication behaviour in various contextual situations. This study found that the Communication Flexibility Scale positively correlated with other self-report questionnaires designed to capture social desirability and communication adaptability: and that those who scored high on communication flexibility also scored high on social desirability and communication adaptability (Martin & Rubin, 1994). This line of research, along with the fact that cognitive flexibility was thought to be a prerequisite for being behaviourally flexible (Martin & Rubin, 1995), ultimately led to the development, and later the initial validation, of the Cognitive Flexibility Scale (Martin & Anderson, 1998; Martin & Rubin, 1995). Since then, other self-report questionnaires designed to assess cognitive and behavioural flexibility have been developed, including the Cognitive Flexibility Inventory (Dennis & Vander Wal, 2010), the shift subscale of the Behavior Rating Inventory of Executive Function (Roth et al., 2005) and the Detail and Flexibility Questionnaire (DFlex; Roberts et al., 2011). Each of these questionnaires require the participant to reflect on their own thinking and behaviour before making a judgement about how flexible they view themselves to be. One limitation of self-report questionnaires is that they are inherently subjective; thus, level of insight, meta-cognitive awareness and response biases may influence results. For example, poor insight has been reported to be a feature of some patients with eating disorders, and it has been suggested that reduced neurocognitive flexibility may contribute to poor insight (Konstantakopoulos et al., 2011). However, it seems equally possible that poor insight may lead certain patients to overestimate their own flexibility in thinking and behaviour, and thus show inflated scores on a self-report questionnaire. Alternatively, someone who is highly perfectionistic or lacks self-esteem may judge their own cognitive flexibility too harshly, and thus show deflated scores on a self-report questionnaire.

Intuition trumps evidence – conflation of different constructs of cognitive flexibility

The coming together of these separate lines of research appears to have largely occurred in the field of eating disorders, then spread into other fields. In the late 20^th century, Strober (1980) explored personality types using the California Psychological Inventory (Gough, 1956), a 480-item true/false self-report questionnaire that produces scores for 18 personality traits. One such subscale was the Flexibility (Fx) subscale, which comprised of 28-items that were designed to assess the ability to be adaptable in thinking, behaviour, and temperament (Donnay & Elliott, 2003). In Strober’s (1980) study, people with anorexia nervosa scored lower than clinical control cohorts on this “Flexibility” subscale, leading Strober to conclude that there is a distinct type of “personality” that characterises people with anorexia nervosa. That finding was corroborated and extended by Casper et al. (1992) who reported that people with anorexia nervosa scored lower on the California Psychological Inventory Flexibility subscale than both healthy controls and those with bulimia nervosa. This apparent “inflexible personality trait” in people with eating disorders seems to have led researchers to pursue more quantifiable assessments. Here they turned to the neurocognitive tasks outlined above, presumably because they purported to assess “flexibility in thinking”.

From the early 1990’s onwards, there was a steady increase in “cognitive flexibility” research, primarily focused on neurocognitive tasks (e.g., Anderluh et al., 2003; Cavedini et al., 2004; Fassino et al., 2002; Szmukler et al., 1992; Tchanturia et al., 2001, 2002; Tchanturia, Anderluh, et al., 2004). During the last two decades, many studies found poorer performance on neurocognitive tasks of cognitive flexibility in people with anorexia nervosa than in healthy controls (Fassino et al., 2002; Holliday et al., 2005; Lounes et al., 2011; Steinglass et al., 2006; Tchanturia et al., 2001, 2002, 2012; Tchanturia, Anderluh, et al., 2004; Westwood et al. 2016), although many others did not (Cavedini et al., 2004; Dmitrzak-Węglarz et al., 2011; Galimberti et al., 2012; Giel et al., 2012; Miles et al., 2020; Tokley & Kemps, 2007; Vall & Wade, 2015; Van Autreve et al., 2016). Poor performance on neurocognitive tasks was often proposed as a potential contributor to clinical observations of inflexible thoughts and behaviours (i.e., calorie counting and strict exercise regimes) in people with anorexia nervosa (Brockmeyer et al., 2014; Rößner et al., 2017; Tchanturia et al., 2012, 2013). There appears to be no empirical data to support that assertion. Nonetheless, the results of neurocognitive tasks certainly appeared to corroborate findings from self-report questionnaires that consistently showed poorer self-evaluated cognitive flexibility in people with eating disorders than in healthy controls (Dell’Osso et al., 2018; Herbrich et al., 2018; Lao-Kaim et al., 2015; Lounes et al., 2011; McAnarney et al., 2011; Roberts et al., 2011).

The extant literature on cognitive flexibility does not explicitly suggest that self-reported cognitive flexibility is equivalent to performance on neurocognitive cognitive flexibility tasks. However, it is evident, particularly within the eating disorders field, that these assessments of cognitive flexibility have become conflated over time (e.g., Abbate Daga et al., 2014; Friederich & Herzog, 2011; Genders & Tchanturia, 2010; Holliday et al., 2005; Leppanen et al., 2018; Perpiñá et al., 2017; Steinglass et al., 2006; Tchanturia et al., 2012; Tchanturia, Anderluh, et al., 2004). Such conflation becomes particularly clear when we examine the rationales provided in the introductions of various papers. For example, Holliday et al. (2005) writes: “The hypothesized association between set-shifting difficulties and anorexia nervosa have face validity in that individuals with anorexia nervosa are often described as persistent, with rigid, conforming, or obsessional personalities” (p. 1). Leppanen et al. (2018) echoes a similar argument: “As changes in thinking styles are believed to contribute to treatment resistance in AN, interventions targeting neurocognitive processes are needed” (p. 1). That those with eating disorders are often characterised by both types of assessments as exhibiting poorer cognitive flexibility than healthy controls, and that the two assessment types are defined by the same term, intuitively points to a unified underlying construct. At face value, self-report and neurocognitive assessments certainly seem to relate. For example, if a group of people are displaying difficulties on average in both of these types of assessments, which share the one label of “flexibility”, then it would seem reasonable to assume that both types of assessments are revealing a broader problem with cognitive flexibility. However, such assumptions leave researchers and clinicians vulnerable to what has been referred to as the jingle fallacy – the erroneous assumption that two tests assess the same construct because they share similar names or labels – which was first described by a ”Professor Aikins” cited by Thorndike (1904, p. 14) over a century ago. The interchangeable use of these assessments (e.g., Abbate Daga et al., 2011; Arlt et al., 2016; Dell’Osso et al., 2018; Fujino et al., 2019; Malivoire et al., 2019; Oguz et al., 2019; Tchanturia et al., 2012) is predicated on that implicit assumption, the truth of which remains unwarranted. Further, given the abundance of tasks that are currently being used to capture cognitive flexibility, it can also be argued that such conflation is not confined solely to self-report and neurocognitive assessments, but also exists to a large degree within the neurocognitive space itself. Failure to detect the jingle fallacy in psychological research not only has the potential to cause immense confusion and heterogeneity in regards to defining, conceptualising and assessing psychological phenomena – including, but by no means limited to, the construct of cognitive flexibility – but also has putative consequences for the field, such that “we could end up with several constructs under the same name” (Ionescu, 2012, p. 190). Such threats to our field can begin to be mitigated by i) transparently reporting construct measurement (e.g., What is the construct? What measure was used to operationalise the construct? How is the construct quantified?) to avoid falling into a “measurement schmeasurement attitude” (Flake & Fried, 2020) or engaging in questionable measurement practices, and ii) better understanding the ways in which self-report and neurocognitive assessments of “cognitive flexibility” are similar and distinct. Implementing the ten conceptual, empirical, and developmental criteria proposed by Lawson and Robins (2021) would help to elucidate whether these two assessment approaches can be considered as sibling constructs under the parent construct of “cognitive flexibility”.

While self-report and neurocognitive assessments have been implied to target the single construct of “cognitive flexibility”, others have suggested they are causally linked. For example, one proposal is that inflexible behaviours and thoughts (typically assessed via self-report questionnaires, interviews, or informant reports) are underpinned by processing impairments that can be assessed by neurocognitive tasks bearing the same name – assessments of “cognitive flexibility” (Abbate Daga et al., 2014; D’cruz et al., 2013; Leppanen et al., 2018; Steinglass et al., 2006; Tchanturia et al., 2012). To illustrate this point, Abbate Daga et al. (2014) state that “several lines of evidence suggest that some clinical aspects of AN [anorexia nervosa] could mirror alterations of cognitive functions; in particular, those rigid and perfectionistic features that usually characterize affected individuals could be the result of set-shifting inefficiencies” (p. 592). Similarly, Tchanturia et al. (2012) argued that “People with eating disorders (ED) frequently present with inflexible behaviours, including eating related issues which contribute to the maintenance of the illness. Small scale studies point to difficulties with cognitive set-shifting as a basis” (p.1). By contrast, the opposite – that poor performance on neurocognitive tests of cognitive flexibility in people with anorexia nervosa results from their inflexible behaviours and thinking – has also been proposed. For example, “ … clinical observations of impaired cognitive-behavioural flexibility in AN [anorexia nervosa] patients underlie impaired set-shifting and impaired behavioural response shifting” (Friederich & Herzog, 2011, p. 115). However, there is no clear empirical evidence to support either of these opposing theories. Thus, there appears to be three apparent conceptual manifestations in the field: that inflexible thoughts and behaviours cause neurocognitive inflexibility, that neurocognitive inflexibility causes inflexible thoughts and behaviours, and that a single construct – “cognitive flexibility” –exists, which can be assessed through self-reported thoughts and behaviours or neurocognitive tasks. However, it should be noted that such explanations likely oversimplify, and thus downplay the influential role of other factors on cognitive flexibility. Future research might consider conducting mediation analyses to establish whether cognitive (e.g., self-efficacy) and non-cognitive (e.g., sleep quality) factors predispose particular manifestations of cognitive flexibility.

The evidence concerning how well the two types of assessments relate to one another conflicts with all three conceptual manifestations. Primary studies have reported little-to-no relationship between self-report and neurocognitive assessments of cognitive flexibility. For example, Gelonch et al. (2016) found no relationship between the shift subscale of the Behavior Rating Inventory of Executive Function – Adult version and outcomes on the TMT and WCST in healthy individuals and people with fibromyalgia. Lounes et al. (2011) did not detect a relationship between the Cognitive Flexibility Scale and the Brixton Spatial Anticipation Task in people with anorexia nervosa and a healthy cohort. In university students, Gonzalez et al. (2013) reported no association between the Cognitive Flexibility Scale and the TMT and a novel flexibility task that required participants to switch between rules in order to complete a maze-like puzzle. Johnco et al. (2014) reported that the Cognitive Flexibility Scale and Cognitive Flexibility Inventory did not correlate with a range of neurocognitive tasks in older adults with comorbid depressive and anxiety symptoms, but small correlations (r ≈ 0.28) were detected in the healthy comparison group. Two recent extensive meta-analytic reviews (Howlett et al., 2021, 2022), drew data from published and unpublished work and also failed to detect a relationship in non-clinical or clinical cohorts. Two recent studies reported null correlations between a range of different self-report and neurocognitive assessments of cognitive flexibility. Miles et al. (2022) found no significant correlations between the Cognitive Flexibility Inventory and two neurocognitive tasks – the TMT and WCST – in people with and without a lifetime diagnosis of anorexia nervosa; Sternheim et al. (2022) reported no significant association between the TMT and the DFlex in two patient groups: people with anorexia nervosa and those with obsessive compulsive disorder. Together, these findings cast significant doubt about whether these assessments capture the same construct.

A possible explanation as to why neurocognitive assessements and self-report questionnaires of cognitive flexibility do not relate is because neurocognitive tasks lack ecological validity (Chaytor et al., 2006), and by design they are suited to assess momentary performance alone. In contrast, self-report questionnaires by design, assess relatively stable thoughts and behaviours of respondents over longer timescales (such issues have been previously discussed in relation to the construct of self-control (i.e., Wennerhold & Friese, 2020)). It can therefore be argued that if self-report questionnaires were re-designed to capture flexibility “in the moment”, rather than capturing responses that represent “typical thoughts and behaviours”, then perhaps these two assessment approaches would demonstrate desirable correlations. Although self-report questionnaires of cognitive flexibility often have high reliability (Dennis & Vander Wal, 2010; Johnco et al., 2014; Martin & Anderson, 1998), neurocognitive tasks do not. The lack of reliability that is associated with neurocognitive tasks has also previously been described as a possible contributor to the poor correlations that are typically observed between self-report and neurocognitive assessments (Dang et al., 2020). Further, the lack of association between self-report and neurocognitive tasks of “cognitive flexibility” may be a consequence of the vast variety of neurocognitive tasks that are currently available. That is, each test may tap into slightly different underlying processes, which presents unique challenges when attempting to accurately match these tasks to self-report assessments. Another explanation for the poor relationship between self-report and neurocogntive assessments of cognitive flexibility is that these assessments may be tapping into two entirely distinct constructs. One may begin to ponder then, an important question: how is it that a group of people such as those with anorexia nervosa, for example, can perform worse on both assessments, yet performance on one assessment does not relate to performance on another? One might suggest that people can perform differentially on these two assessments (i.e., perform well on one assessment but not the other), but this speculation goes against empirical data that have established that some clinical conditions, such as anorexia nervosa, are associated with deficits on both assessments. However, it is not yet possible to determine whether one (or neither) assessment is accurate in capturing the construct of cognitive flexibility. Notwithstanding, conflating these two assessment tools by generalising the results of one to the other could limit our conceptualisation and understanding of the construct of cognitive flexibility. Further, conflating these assessments could result in mischaracterisations of clinical disorders.

As stated earlier, the problem of conflating self-report and neurocognitive assessments is not unique to the field of cognitive flexibility. The executive functioning literature more broadly has encountered similar problems where self-report questionnaires are presumed to capture identical or similar underlying processes that are assessed by neurocognitive tasks (Toplak et al., 2013). Those presumptions are often unsupported by empirical data obtained from adults (Burmester et al., 2016; Rabin et al., 2006; Spencer et al., 2010), adolescents (Herbrich et al., 2018; Stedal & Dahlgren, 2015) and children (Soto et al., 2020). Further, some clinicians overestimate the association between self-report questionnaires and neurocognitive performance on tasks that assess memory, attention, and thinking/organisation (Spencer et al., 2010), which may also explain why assessments of cognitive flexibility are used interchangeably in clinical practice. Anecdotally, knowing that self-report and neurocognitive assessments do not usually relate well does not necessarily prevent their interchangeable use, nor the generalisation of findings from one assessment tool to another. We propose therefore, two significant implications for clinical practice. First, clinicians could integrate informant-reports as a part of a comprehensive clinical assessment to capture a client’s mental ability to switch thinking and adapt behaviours in accordance with changing contexts. Clinicians may want to implement informant-reports as complementary tools to clinical assessment because these tools are considered to be less subjective and less influenced by other decrements (e.g., a lack of insight). Tools that have been widely implemented and can assist clinicians in avoiding erroneous assumptions include the informant version of the Behavior Rating Inventory of Executive Function (Roth et al., 2005). Second, there is some time urgency about setting the record straight with regard to “cognitive flexibility” and moving forward unencumbered by erroneous assumptions of equivalence between what seem to be distinct constructs.

Limitations of existing research and future recommendations

The conflation of self-report and neurocognitive assessments of cognitive flexibility present problems that can be immediately addressed. First, we should acknowledge that available evidence suggests against a causal relationship between the neurophysiological processes that enable one to perform well on neurocognitive tests, and one’s tendency to report flexibility in one's thoughts and behaviours. Furthermore, given that there is no evidence of a 1:1 relationship between flexible thinking and flexible behaviour, we should also be precise in our language when describing these distinct, but potentially related facets of flexibility. Second, we should cease interchanging self-report and neurocognitive assessments of cognitive flexibility (exemplar studies include Miles et al., 2022; Sternheim et al., 2022). Third, we can recognise as erroneous the conclusion that people with clinically observable or self-reported “inflexible” behaviours can be accurately and precisely evaluated with neurocognitive tasks. This would avoid the current potential of dismissing or overlooking the clinical relevance of self-report questionnaires (which can provide information that is more ecologically valid and client-specific), and should thus promote a more client centric clinical encounter. To provide a more comprehensive assessment of cognitive flexibility, it may be appropriate for clinicians and researchers to use self-report and neurocognitive assessments alongside more ecologically valid tasks – e.g., the Virtual Cooking Task (Chicchi Giglioli et al., 2021) – that allow individuals to be immersed in virtual environments where they are required to perform everyday activities. Poor performance on one of these assessments may indicate underlying issues that may warrant further investigation. For example, neurocognitive tasks may be more sensitive to capturing breakdowns in cognitive flexibility, whereas virtual reality technology may be able to assist in uncovering instances of cognitive inflexibility in real-life scenarios, and self-report questionnaires might provide more nuanced differences in cognitive flexibility. However, users must remain aware that self-report and neurocognitive tasks may be targeting different contructs altogether.

Other unknowns will require more concerted attention. That the two approaches to assessing cognitive flexibility do not appear to relate well may reflect problems with the assessments themselves rather than divergence in the constructs that they aim assess. For example, many neurocognitive tasks may be restricted by the task impurity problem – the phenomenon that a task does not assess one cognitive domain or skill in isolation, but rather relies on a combination of both executive and non-executive processes, making the cause of poor task performance difficult to pinpoint (Miyake et al., 2000; Müller & Kerns, 2015). It should be noted, however, that attempting to “isolate” cognitive flexibility may distort the very construct itself, because cognitive flexibility may emerge from its reliance on various cognitive processes (Dajani & Uddin, 2015). Additionally, breakdowns in cognitive flexibility, as indexed by neurocognitive tasks, may be attributable to deficits in other executive processes such as working memory or inhibition. For instance, problems with inhibition may differentially impact cognitive flexibility outcomes and have no impact on working memory. It is not yet possible to untangle the contributors to poor cognitive flexibility performance, but ongoing identification of the genetic and neural substrates that underpin cognitive flexibility performance may shed some light (Nomi et al., 2017; Verdejo-Garcia et al., 2015; Zhang et al., 2019). Therefore, it is vital that researchers and clinicians are aware of such limitations of neurocognitive tasks when interpreting and drawing conclusions from findings and control for other executive processes where possible.

One criticism of self-report questionnaires is that they can be confounded by negative affect, with studies reporting an association between self-reported cognitive flexibility, and depression and anxiety (Dennis & Vander Wal, 2010; Johnco et al., 2014; Lounes et al., 2011; Miles et al., 2022; Wang et al., 2019; Yu et al., 2020). Although some studies have found that neurocognitive tasks do not appear to be confounded by negative affect (Herbrich et al., 2018; Lang et al., 2015; Lounes et al., 2011; Miles et al., 2022) others studies have found that neurocognitive tasks are modulated by emotional valence (Wang et al., 2017). These assessments likely still have great utility for our field, but rigorous validation and replication studies are still required if we are to understand the possible impact of such factors on results from either assessment approach, let alone if we are to delineate the constructs that self-report questionnaires and neurocognitive tasks actually assess.

Our final recommendation is pragmatic. As clinicians and researchers, we need to clearly identify and articulate what it is that we are interested in assessing. Are we interested in self-reported evaluations of adaptability in thoughts and/or behaviours, or performance on neurocognitive tasks that require shifting strategies, or both? By being more precise about the construct in which we are interested, we can more easily select the best assessment with which to investigate it and avoid erroneous generalisations beyond our data. The broad term of “cognitive flexibility” may well be an umbrella term that is simply too imprecise. Making precise interpretations and not generalising beyond the assessment modality used will facilitate progress in our understanding of these constructs and how they relate to psychopathology.

Conclusion

The conflation of neurocognitive tasks and self-reported outcomes of cognitive flexibility has arisen through the convergence of seemingly similar lines of research and intuitive, but not empirically supported, assumptions. There remains scant evidence in support of the claim that poor performance on neurocognitive tasks will be mirrored by poor performance on self-report questionnaires within individuals, even though there is building evidence that certain clinical cohorts perform poorly as a group on both types of assessments. Future research and clinical practice should give careful consideration when selecting which cognitive flexibility assessments to use by ensuring that the chosen assessments are aligned with the aims of research or the goals of therapy. It is strongly encouraged that both researchers and clinicians be precise when describing or referring to such assessments, and when interpreting results. Lastly, the interchangeable use, and conflation of, neurocognitive tasks and self-reported assessments of “cognitive flexibility” should be avoided.

Disclosure statement

GLM has received support from: Reality Health, ConnectHealth UK, Seqirus, Kaiser Permanente, Workers’ Compensation Boards in Australia, Europe and North America, AIA Australia, the International Olympic Committee, Port Adelaide Football Club, and Arsenal Football Club. Professional and scientific bodies have reimbursed him for travel costs related to presentation of research at scientific conferences/symposia. He has received speaker fees for lectures on pain and rehabilitation. He receives book royalties from NOIgroup publications, Dancing Giraffe Press & OPTP. CB has received support from Workers’ Compensation Boards in Australia. All remaining authors (CAH, SM and AP) declare no potential conflicts of interest (personal or financial) with respect to the contents, authorship, or publication of this manuscript.

Data availability statement

Data sharing is not applicable to this article as no new data were collected or analysed in this manuscript.

Additional information

Funding

AP was supported by a National Health and Medical Research Council (NHMRC) Project Grant [ID 1159953]. CB and GLM were supported by a NHMRC Leadership Investigator Grant [ID 1178444] to GLM. These funding bodies had no role in the writing of the manuscript, or the decision to submit the paper for publication.