The Evaluation and Selection of Creative Ideas in Educational Settings: Current Knowledge and Future Directions

ABSTRACT

Worldwide, student-centered pedagogies have emerged in education to develop creativity. However, these pedagogies do not automatically enhance students’ creativity, because students tend to underestimate and reject creative ideas – even when highly novel ideas are required to solve the problem at hand. Understanding how students evaluate and select ideas is crucial for enhancing creativity. Therefore, this paper reviews research on idea evaluation and idea selection among students. This paper suggests that the evaluation of ideas depends both on specific and general components, and a mild state of affect and openness to experience seems to play a significant role. To improve idea evaluation and idea selection, students should be exposed to a variety of ideas and effective instructional strategies benefit students as well. Teachers should explicitly instruct students to select creative ideas and encourage them to simultaneously generate and refine ideas. However, instructing students to transform their creative ideas into tangible products may unintentionally influence their choices for creative ideas. Balancing novelty and usefulness pose challenges for students during evaluation and selection, and teachers should attune to students’ reactions as much as possible (e.g. accommodating emotional outbursts). Finally, several future trends and important research questions are highlighted.

Introduction

Worldwide, creativity is seen as a crucial competence in all levels of education, ranging from elementary to tertiary education (OECD, 2019; Zahidi, Ratcheva, Hingel, & Brown, 2020). Creativity can be understood as “the competence to engage productively in the generation, evaluation and improvement of ideas that can result in novel and useful solutions, advances in knowledge, or impactful expressions of imagination” (OECD, 2019, p. 8). This definition underscores that creativity does not only consist of the generation of novel and useful ideas, but also their evaluation and selection for implementation.

The evaluation of ideas involves three interrelated mental operations: idea appraisal, forecasting, and refinement (Mumford, Lonergan, & Scott, 2002). Idea appraisal involves identifying relevant standards – such as novelty and feasibility – to judge the viability of ideas, forecasting comprises mentally simulating the consequences of implementing an idea, and refinement involves discarding or changing elements of an idea, elaborating on key details, or combining new elements to improve it (Mumford, Lonergan, & Scott, 2002). The selection of ideas is the process of choosing which ideas or concepts to pursue for further development or implementation (Rietzschel, Nijstad, & Stroebe, 2010).

To develop students’ creativity, numerous student-centered pedagogies have emerged in education (e.g., project-or research-based learning). However, it can be expected that these student-centered pedagogies do not automatically foster students’ creativity, because research has found that students tend to underestimate and reject creative ideas – even when highly novel ideas are required to solve the problem at hand (e.g., Ahn, van Swol, Kim, & Park, 2022; Harvey & Mueller, 2021; Johnson & D’Lauro, 2018). Our understanding of how students evaluate and select ideas and how this can be improved is vital to improve students’ overall creativity. Therefore, the purpose of this paper is to review the state of current research on idea evaluation and idea selection among students, and to identify future trends and important research questions.

State of current research – what do we know?

Understanding idea evaluation and idea selection

To understand students’ ability to recognize and select creative idea, we first need to answer one of the most enduring controversies in the creativity field: whether the evaluation of ideas depends on the specific domain at hand or can be done more generally.

In line with the definition of creativity, a creative idea is one that is both novel (surprising, out of the ordinary) and useful (feasible, effective) as defined within a social context (Plucker, Beghetto, & Dow, 2004; Runco & Jaeger, 2012). Van Broekhoven, Cropley, and Seegers (2021) have examined whether art and science students evaluate creativity in products similarly or differently. For this, 130 art students and 2147 science students were given four pre-defined solutions for the problem of how to improve public trains. For each solution, students were first asked to rate the originality, feasibility, effectiveness, and overall creativity. They found that art students associate originality with creativity more strongly than do science students. In contrast, science students associate feasibility and effectiveness with creativity more strongly than do art students. Therefore, van Broekhoven, Belfi, Borghans, and Seegers (2021) concluded that both science and art students see originality as central to defining the creativity of a product, while the usefulness criterium seems to be more important in determining the creativity of ideas for science students than for art students. In sum, the dominant perspective appears to be that the evaluation of ideas has some general and specific components, where the specific components stem from contextual elements associated with each discipline (e.g., reward systems, norms and culture, demands and constrains present in each discipline).

One reason why it may be difficult to identify or select creative ideas (i.e., novel and useful) is that novelty and usefulness are often negatively correlated (Frederiksen & Knudsen, 2017; Nijstad, De Dreu, Rietzschel, & Baas, 2010; Zacher, Robinson, & Rosing, 2016). Novel ideas are often seen as infeasible and risky, because they are, by definition, untried and it is unknown whether an idea will work out in practice. Due to this novelty-usefulness tension, the evaluation and selection of creative ideas may feel like a risky undertaking, and a first step is to identify psychological characteristics that relate to students’ ability to recognize and select creative ideas.

The role of personality

Several scholars have addressed the relationship between personality and idea evaluation and idea selection (e.g., Lloyd-Cox, Pickering, & Bhattacharya, 2022; Puente-Díaz, Cavazos-Arroyo, Puerta-Sierra, & Vargas-Barrera, 2022; Toh & Miller, 2016). For instance, Lloyd-Cox, Pickering, and Bhattacharya (2022) found that people with higher openness and higher intellect placed a greater emphasis on novelty when evaluating ideas for the Alternative Uses Tasks. In similar lines, Puente-Díaz, Cavazos-Arroyo, Puerta-Sierra, and Vargas-Barrera (2022) found that openness to experience is positively correlated with identifying the strengths and weaknesses of generated ideas, which subsequently facilitated idea selection. Likewise, Toh and Miller (2016) found that people who are more risk prone – closely related to openness to experience – selected more creative ideas than people who are more risk averse.

Furthermore, in line with personality, the regulatory focus theory postulates that promotion focus is associated with more risk taking and flexibility whereas a prevention focus is related to risk aversity and rigidity (Higgins, 1998). Herman and Reiter-Palmon (2011) examined the relationship between regulatory focus and idea evaluation performance. They found that people high in promotion focus rate ideas more accurately on originality but less accurately on quality (i.e., how logical and workable ideas are), whereas those high in prevention focus rate their ideas more accurately on quality but less accurately on originality. Furthermore, de Buisonje, Ritter, de Bruin, Ter Horst, and Meeldijk (2017) examined whether promotion focus, positive affect, and self-affirmation can facilitate creative idea selection. They found that participants selected ideas that were more creative when promotion focus, positive affect, and self-affirmation were induced jointly, compared to a control condition where participants performed corresponding filler tasks.

In conclusion, openness to experience and promotion focus benefit the evaluation and selection of creative ideas.

The role of mood

Next to personality, only a few studies have examined the relationship between mood and idea evaluation (Wang, Li, Li, Dai, & Hu, 2022; Watts et al., 2020). Watts et al. (2020), for instance, elicited affective shifts with short stories among 367 undergraduates, and then asked them to develop a business plan for a failing furniture company. They found that shifts in affective tone and arousal interacted to influence implementation planning, but not idea evaluation. More specifically, implementation planning was strongest among undergraduates who read an unpleasantly toned, high-arousal (i.e., angry) story followed by a pleasantly toned, low-arousal (i.e., relaxed) story, and by undergraduates who read an unpleasantly toned, low-arousal (i.e., sad) story followed by an unpleasantly toned, high arousal (i.e., angry) story. Furthermore, Wang, Li, Li, Dai, and Hu (2022) found that undergraduates with a low-approach-motivated positive affect failed to discriminate the more original uses from the less original uses resulting in a shift to a lenient criterion for the evaluation and a greater acceptance of less creative ideas. These studies suggest that pleasantly toned, high-arousal states like happiness and excitement may be misguided, but a consistent, mild state of affect may help students in their evaluation and selection of creative ideas.

Improving idea evaluation and idea selection

Having a better understanding in idea evaluation and idea selection, the following section discusses potential promising interventions to improve these processes among students.

Exposure to ideas

Based on cognitive literature, exposure to ideas may be key in enhancing students’ ability to recognize both the novelty and usefulness of ideas (George & Wiley, 2020; Wang, Nickerson, & Sakamoto, 2018). In line with the search of ideas in associative memory (SIAM) model, exposure to ideas activates a search for ideas in their short- and long-term associative memory (Nijstad, De Dreu, Rietzschel, & Baas, 2010). This activation of the associative memory is likely to give them more comparison points to others’ ideas than students who have not activated this memory, and subsequently may improve students’ ability to recognize creative ideas. To examine this, van Broekhoven, Belfi, Borghans, and Seegers (2021) randomly assigned 1864 undergraduates to two conditions: task exposure condition and no task exposure condition. In the task exposure condition, students generated and evaluated ideas for the same task. In contrast, students in the control condition generated and evaluated ideas for different tasks. They found that task exposure improves students’ ability to accurately recognize creative and original ideas, and their ability to discriminate between highly feasible and less feasible ideas. In sum, exposure to ideas seems to activate the associative memory which subsequently benefits students in recognizing both the originality and feasibility of ideas.

Instructional strategies

Teachers could employ instructional strategies to help students’ creative idea evaluation and idea selection. In line with the componential theory of creativity (Amabile, 1988), teachers could employ cognitive-based creativity techniques to help students in their evaluation and selection of ideas. This theory postulates that three components within the individual – domain relevant skills, creativity-relevant processes, and intrinsic task motivation – are necessary for any creative response. van Broekhoven, Belfi, Hocking, and van der Velden (2020) examined a 10-hour cognitive-based creativity training on students’ idea evaluation performance (techniques: idea evaluation metric and strength and weakness analysis). As pre- and posttest, the students’ originality and feasibility ratings were compared with experts’ ratings. However, they found no effect of the training on students’ idea evaluation performance. Similarly, Ritter, Gu, Crijns, Biekens, and Wei (2020) examined a one-year cognitive-based creativity training on students’ idea selection performance (techniques: simplify, differentiate, visualize, and tag the problem). They found no effect on students’ idea selection performance. In sum, cognitive-based creativity techniques do not seem to benefit students’ ability in evaluating and selecting creative ideas.

In line with van Broekhoven, Belfi, Hocking, and van der Velden (2020), Reiter-Palmon, Kennel, de Vreede, and de Vreede (2019) found that a rubric – specifying examples of solutions that fit each level of solution usefulness and originality – do not improve idea evaluation, but a structured selection process resulted in the selection of more original solutions. In a structured selection process, students first individually select a few solutions, then share their selection with each other and narrow down their idea selection to two solutions together. Furthermore, Baruah, Paulus, and Kohn (2021) found that a simultaneous focus during idea evaluation yields more original, feasible and elaborated ideas. This means that students can better simultaneously generate and refine their ideas than first generating ideas before refining them. For idea selection, Rietzschel, Nijstad, and Stroebe (2010) found that undergraduates tend to reject original ideas in favor of more feasible ideas when instructed to select the best ideas. Several researchers have – indeed – found that students need to be explicitly instructed to select creative ideas (Rietzschel, Nijstad, & Stroebe, 2010, 2014). Furthermore, van Broekhoven, Belfi, Borghans, and Agnoli (2022) examined whether students select different ideas when they are instructed to construct ideas into tangible products. For this, students were instructed to both select and construct original ideas in the classroom, while other students were only instructed to select the most original ideas. They found that students who were instructed to construct their idea in the classroom turned a blind eye to original ideas and preferred the more feasible ideas. In sum, these studies show that instructional strategies heavily influence the evaluation and selection of ideas.

Teaching activities

Next to instructional strategies, teaching activities may nurture students’ idea evaluation and idea selection as well. Cropley (2018) postulated – for example – that teachers may need to promote self-evaluation in students, delay judging students’ ideas until they have been thoroughly worked out and clearly formulated, learn students to cope with frustration and failure, so that they have the courage to try the new, and encourage flexible thinking in students in evaluating and improving their ideas. In a qualitative study, Van Broekhoven, van Uum, Meijer, Kroesbergen, and Huck (2023) identified several teaching activities to support students in their evaluation and selection of creative ideas. They found that students experience difficulties in finding a way to manage the twin goals of novelty and usefulness in the search for creative ideas. Students often face novel but useless (ineffective or infeasible) ideas, useful ideas that lack novelty, or ideas that are both low in novelty and usefulness. Teachers reported that some students react negatively to the discomforting nature of creative ideas and, subsequently, discard their ideas, while others react positively and are motivated to continue with their idea. Teachers often addressed the negative reactions through affective teaching activities, such as accommodating emotional outbursts, and supporting and encouraging risk-taking. By contrast, student groups’ positive reactions were addressed with both cognitive and metacognitive teaching activities. Teachers asked detailed questions about the problem or the generated idea (cognitive activities), and helped the groups determine their next steps, such as seeking feedback from stakeholders in the field (metacognitive activities). In sum, the inherent tension between novelty and usefulness of creative ideas will trigger positive and/or negative students’ reactions, and teachers should attune to students’ reactions as much as possible.

Future directions

The following section will attempt to identify future trends and important research questions.

A dynamic micro-longitudinal approach to study idea evaluation and idea selection

Arguably, the evaluation of creative ideas is highly interdependent with idea selection. For instance, if an idea is appraised as high risk during the initial forecasting phase, it is unlikely to be selected for further development. A dynamic micro-analytical approach allows researchers to examine the iterative nature of idea evaluation and idea selection. Researchers could then examine how creative ideas are developed from turn-to-turn as the interaction between students unfold itself. For example, Chiu and Lehmann-Willenbrock (2016) have examined how social metacognitive actions (e.g., agree, rudely disagree) affect the likelihood of new ideas to emerge in groups. For this, they have used 3,926 turns of talk by 80 students in 20 groups. They found that a rude disagreement often triggered another rude disagreement, which yielded fewer creative ideas to follow. Similarly, Van Broekhoven, Chiu, van Uum, and Kroesbergen (2023) examined 4,047 utterances by 12 students in 4 trios. They found that asking invitational questions and thinking yield significant more feasible, original and creative ideas, while irrelevant and process talk yields significant less feasible, original and creative ideas. Hence, the dynamic micro-analytical approach to study the creative process has attracted growing interest, and future research could further identify promoting and inhibiting patterns of talk in the idea evaluation and idea selection process of students

Artificial intelligence in the classroom: a potential tool to measure creativity

The issue of measurement of idea evaluation and idea selection is an important one that must be addressed to advance research in this area. A typical method is the Consensual Assessment Technique (CAT, see Amabile, 1982) – a subjective measure – where expert judges independently rate the creativity of creative products (e.g., accuracy is measured in terms of discrepancy between experts’ and participants’ ratings, see Grohman, Wodniecka, & Kłusak, 2006). While this measure allows researchers to have a measure of idea evaluation accuracy, it remains a subjective rating dependent on the knowledge and experiences of experts. A more objective measurement is statistical infrequency or so called “hit rates” (see Runco & Dow, 2004; Runco & Smith, 1992 for more information) or by means of informedness (see Benedek et al., 2016). With statistical infrequency, responses are weighted according to their frequency in the total sample of individuals. While this is more objective, it still bears its limitations by being constrained by that specific pool of ideas. Another set of ideas may contain other ideas, and, therefore, the statistical infrequency would change.

As generative Artificial Intelligence (AI) – such as ChatGPT – is able to respond on insert texts or questions, a few recent studies have shown that automated scoring of verbal and figural divergent thinking can be improved by using large language models (Cropley & Marrone, 2022; Organisciak, Acar, Dumas, & Berthiaume, 2023). In educational practice, this means that teachers could – for example – insert or copy students’ ideas or solutions into ChatGPT and ask ChatGPT to react to them, assess them, and tell them what it finds interesting and why. Future research could examine how large language models could not only improve automated scoring of students’ divergent thinking in typical creativity tasks, but also their divergent thinking in realistic complex problem-solving.

Conclusions

This paper addresses an area in creativity research that has attracted growing interest in the last years. Students tend to underestimate and reject creative ideas – even when creative ideas are required to solve the problem at hand, and, therefore, it is important to get a better understanding of how students evaluate and select ideas, and how this can be improved. As evidenced by the multifaceted approaches and methodologies discussed in this paper, scholars demonstrated a deep commitment to gain a better understanding of the process of idea evaluation and idea selection. Nonetheless, this field stands as an evolving domain, offering a plethora of opportunities for future research.

While several definitions exist, they all share the common element of the production of novel (original, new) and useful (appropriate, feasible) ideas (Plucker, Beghetto, & Dow, 2004; Runco & Jaeger, 2012). However, laypeople see originality as central to defining the creativity of a product while creativity researchers often (implicitly) assume an equal distribution of those two components (Gilson & Madjar, 2011; Rietzschel, Nijstad, & Stroebe, 2010; van Broekhoven, Belfi, Borghans, & Seegers, 2021). This difference in perception on creativity has significant consequences for the field, specifically the measurement of idea evaluation and idea selection. For example, without explicitly instructing participants that a creative idea is both novel and useful, participants are more likely to weigh originality heavier than feasibility in their decision-making process. While it is often acknowledged that the mix of novelty and usefulness embodied in creative ideas can vary, this acknowledgment has yet to permeate the foundational conceptualization of creativity. I propose that our field should get a better understanding of how different groups of lay people determine the creativity of products based on these two key components. For the advancement of our field, it might be imperative to broaden the boundaries of our conceptual framework and incorporate a discernible weight mechanism for the dual constituents of creativity.

Furthermore, as the production of ideas often involves cycles of generating ideas (or divergent thinking) and evaluating ideas (convergent thinking), we need both definitions and theoretical models that take the iterative nature of idea evaluation into account. To my knowledge, OECD is one of the first that developed a definition that explicitly mentions the process of idea evaluation and selection: “creativity is the competence to engage productively in the generation, evaluation and improvement of ideas that can result in novel and useful solutions, advances in knowledge, or impactful expressions of imagination” (OECD, 2019, p. 8). This definition underscores that creativity does not only consist of the generation of novel and useful ideas, but also their evaluation and selection for implementation. Furthermore, the perspective of the theory of complex dynamic systems provides a fruitful avenue for the integration of iterativeness in existing theoretical models. For instance, Mumford’s, Lonergan, and Scott (2002) with three interrelated mental operations – idea appraisal, forecasting, and refinement – may benefit from this perspective that views “iterativeness” as a central characteristic of complex dynamic systems (van Geert & Steenbeek, 2005).

Envisioning the future of educational practice, the acquired knowledge from research on idea evaluation and idea selection has the potential to raise teachers’ awareness on students’ inclinations to discard creative ideas and prefer the more feasible and common ideas. Furthermore, by embracing the dynamic interplay of generating, evaluating, and selecting ideas, educators could nurture students’ ability to refine and revise ideas, and not shy away from those original and risky ideas. Students’ reactions to creative ideas, both positive and negative, require a supporting environment that nurtures emotional resilience among those who recoil from the discomfort of original ideas, while emboldening those who embrace this risky endeavor. Educators could hereby employ a dual approach, integrating both metacognitive and affective teaching activities.

Disclosure statement

No potential conflict of interest was reported by the author(s).