Development of an exploratory creativity assessment scale

ABSTRACT

Exploratory creativity (E-creativity) can be achieved by searching an area of conceptual space governed by specific rules. Existing studies on E-creativity mainly focus on how to use aspects of E-creativity to develop computational creativity tools, but E-creativity assessment scales have not been fully studied. To fill in the gap, this study developed an E-creativity assessment scale based on metrics and experimental determination studies. Eight indexes are promoted through literature investigation, which are related to E-creativity attributes, pre-requirement for the existence of E-creativity, relations between exploratory process and creativity, and results of E-creativity. Then, an empirical case study is applied to investigate the differences between nonprofessionals and professionals when using the developed scale. From the whole research, the results reveal that E-creativity is not simply related to the exploratory process and its concept space; instead, it is also related to the relations between the novelty of the exploratory process and the concept space of E-creativity. The results reflect the role of E-creativity in a creative process. This research provides a further understanding of E-creativity, which can contribute to further develop the definition of E-creativity. The E-creativity assessment scales can be used as a cue to further evaluate machine generated E-creativity.

KEYWORDS:

• Exploratory creativity
• assessment scale
• computational creativity

1. Introduction

Significant efforts have been devoted to the research on creativity. Creativity, as the core term of this research area, has been widely considered and defined by various researchers. For example, creativity is defined as the production of something new and valuable (Childs et al., 2022; Neihart, 1998; Nembhard & Lee, 2017; M. A. Runco & Jaeger, 2012); the process of producing novel and useful ideas (Childs et al., 2006; Sahu & Mukherjee, 2013); or the ability to produce something novel and valuable within a social context (Van Goch, 2018). Rhodes (1961) summarized definition of creativity and proposed the 4P theory to define such a complex and multifaceted phenomenon. Based on the 4P theory, creativity is divided into four strands – person, process, press and products. The 4P theory has received much attention and sparked further research. For example, Park et al. (2016) extended the potential of the 4P theory in neuroscience studies; Walia reviewed the definition of creativity in 2019 (Walia, 2019) based on the development of the 4P theory and Sternberg and Karami (2022) have expanded consideration of creativity to include purpose, press, person, problem, process, product, propulsion and public.

Evaluation of creativity is an active area in creativity research. A variety of creativity assessment methods have been proposed, which generally require human raters to judge the quality of generated creativity (Beaty & Johnson, 2021), such as the Consensual Assessment Technique method (Amabile, 1982), Creative Product Semantic Scale (CPSS; Besemer, 1984; Besemer & O’Quin, 1986), Product Creativity Measurement Instrument (PCMI; Horn & Salvendy, 2006), and Creative Solution Diagnosis Scale (CSDS; Cropley & Cropley, 2005).

With the advent of artificial intelligence, Boden (1998) developed the concept of computational creativity to include combinational creativity, exploratory creativity (E-creativity) and transformational creativity (T-creativity). E-creativity, as one aspect of computational creativity, is related to the generation of novel ideas by the exploration of structured conceptual spaces. E-creativity has demonstrated significant potential for possible findings that can help stimulate design innovation for practical value (Maiden et al., 2010; Yin et al., 2022). At present, E-creativity research has mostly focused on developing tools based on E-creativity principles, such as ‘Ludoscope,’ ‘Black box,’ ‘Narrative Search,’ and ‘DeLeNoX,’ which develops for textual narrative or creative design practice.

Although the theoretical framework of E-creativity has been discussed and developed, there is still a lack of research on E-creativity assessment scale. Contributions mostly focus on developing tools which based on E-creativity to stimulate creativity. The E-creativity evaluation process is usually carried out to test algorithm improvement compared with other models, leaving a gap of proposing standard norms for evaluating E-creativity based on its characteristics.

To summarize, E-creativity assessment scales have not been fully developed. This may hinder future research in the evaluation and automation process for E-creativity. This study, thus, aims to investigate E-creativity assessment indexes. Through metrics identification and an experiment, the study promotes an eight-index E-creativity assessment scale. A case study is then conducted to identify the gap in evaluation process between the nonprofessional and professional. The research contributes to the development of E-creativity assessment. Designers, researchers, and developers can assess the E-creativity of product in a standardized way based on the promoted scale.

2. Literature review

This section reviews creativity and E-creativity, creativity assessment, and existing E-creativity research.

2.1. Creativity and E-creativity

Creativity is one of the fundamental human competencies. As a branch of computer science, artificial intelligence (AI) uses algorithms and machine learning technologies to replicate or simulate human intelligence (Marrone et al., 2022). Considering the importance of creativity in human and the development of AI, computational creativity is promoted with the goal of modeling, simulating or replicating creativity using a computer (Colton & Wiggins, 2012; Iqbal, 2022).

E-creativity is related to generating new ideas within a given conceptual space (Liapis et al., 2013), which can be achieved through constraint satisfaction techniques or evolutionary algorithms. Not only existing in computational creativity, E-creativity also exists in human creativity (Boden, 1998; Guckelsberger et al., 2021). The definition of E-creativity is thus updated to the one which can be used to represent both human and computational E-creativity, to be specific, searching an area of conceptual space governed by certain rules (Riedl & Young, 2006). The meaning of ‘rules’ is further clarified by Wiggins (2006) which is reasoning to an inference within the search. Conceptual spaces are explained as the gather of creation of new ideas (Rebelo et al., 2022). Based on the definition of E-creativity, it can be identified that E-creativity is a special form of creativity.

2.2. Creativity assessment

With awareness of individual differences, creativity assessment measures have received considerable attention. Without an effective scale for measuring creativity, it may be difficult to establish the validity of a particular finding. Self-reporting is an essential method used for assessing personal creativity. By summarizing the personality characteristics of creative people, psychometric inventory tools which can be used to assess personal creativity have been developed and promoted such as Epstein Creativity Competencies Inventory for Individuals (ECCI-i) (Epstein et al., 2008). Press creativity aims to identify how environment affects the creative performance. Factors which were recognized as the influences were identified and then verified such as Swedish Creative Climate Questionnaire (CCQ) (Ekvall, 1996). In product creativity assessment, there are two principal ways to assess product creativity. Some researchers thought creativity can be assessed subjectively by appropriate judgers. Based on this theory, the Consensual Assessment Technique (CAT) (Amabile, 1982) required experts to independently rate generated ideas, artifacts or other forms of creative thinking products. Other researchers have proposed to use a set of metrics such as novelty, value, appropriateness, and infrequency to evaluate product creativity (Yin et al., 2021), such as Creative Product Semantic Scale (CPSS) (Besemer, 1984; Besemer & O’Quin, 1986). Process creativity is mainly assessed with a divergent thinking (DT) test, such as Torrance Tests of Creative Thinking (TTCT) (Torrance, 1966). Based on DT, inventory tools were also developed to assess process creativity such as Runco Ideational Behaviour Scale (RIBS) (M. Runco et al., 2001) and Cognitive Processes Associated with Creativity (CPAC) (Miller, 2014). Some researchers have promoted to use the attributes of creativity to assess creativity. For example, considering novelty is an important attribute of creativity, some researchers proposed to use the novelty level to represent creativity levels.

These creativity-oriented assessment methods indicated the importance of measuring creativity, which inherently triggers researchers’ attention to E-creativity measurement. Also, these methods can be used as the cue to discover the indexes of E-creativity assessment scale.

2.3. Existing E-creativity assessment methods

Existing research has not fully detected the E-creativity assessment methods. Most of the studies aim to develop creativity stimulation tools based on E-creativity knowledge. Then, some studies included a part which tended to verify whether the E-creativity-based tools can stimulate creativity.

Dormans and Leijnen (2013) tended to verify whether their developed E-creativity-based tool (Ludoscope) worked. They conducted two studies. In the first study, the Ludoscope was asked to generate the content of the games step by step based on different grammar. The second study asked Ludoscope to generate lock and key mission structures. Then, the resolution step asked the Ludoscope to generate a structure where each key is associated with at least one lock. Whether the resolution solutions were more novelty was used to represent the performance of E-creativity. Liapis et al. (2013) tried to verified whether their developed E-creativity-based tool ‘DeLeNoX’ worked. The study asked DeLeNoX to detect the iteratively transformed exploration of spaceships. The novelty and diversity of the interative results were ranked to identify whether the E-creativity tools worked. Jennings et al. (2011) tried to assess whether the technology, which can be based on blind variation, selective retention, and principles of E-creativity to generate landscapes, worked. The study asked the technology to generate 25 images and asked 40 participants to assess the quality of the images based on six criteria ‘bright, aesthetic, artistic, captivating, convoluted, and harmonious.’

2.4. Research gap and aims

Although there have been studies assessing E-creativity, most aim to assess effectiveness of the developed tools instead of assessing E-creativity itself. Also, human factors are ignored in the assessment. In addition, although various creativity assessment methods have been promoted, creativity assessment methods cannot be applied to E-creativity assessment directly. According to M. A. Boden (1998), E-creativity is a special form of creativity as well as an indication of capability. Therefore, the creativity assessment criteria cannot report the characteristics of E-creativity accurately. These limitations in E-creativity assessment method may obstruct the development of E-creativity research. To fill in the research gaps, this study attempts to develop an E-creativity assessment scale. The results of this study are expected to be used as a basis for professional and nonprofessional people to assess E-creativity and assist researchers further understand E-creativity.

The reason why the scale-based approach is considered is out of the following consideration. Since now there are no existing E-creativity assessment examples that can be learned from, some inspiration was sought from creativity assessment methods. From the review of Section 2.2 which is about the creativity assessment methods, it can be found that the scale-based approach is the core creativity assessment approach (such as ECCI-i, CCQ, CPSS, TTCT, and CPAC). Therefore, the scale was selected as the assessment frame. In addition, the scale-based approach can contribute to the assessment methods from the following two aspects. On the one hand, the scale-based approach allowed researchers to assess E-creativity quantitatively (Forgeard, 2022). This enables researchers and practitioners to analyze the E-creativity levels, conducted statistical analysis and draw meaningful conclusions from the data. On the other hand, the scale-based approach provided a standardization on how to assess E-creativity (Jarosewich et al., 2002). This makes various raters use the consistent criteria to assess E-creativity. The assessment results thus can be more reliable and the subjective effects of raters can be reduced.

3. Exploratory creativity assessment scale development

Figure 1 shows an overview of the development processes, and two phases are involved: metrics identification and experimental determination, which are described in the following sections.

Figure 1. The development phases of E-creativity assessment scale.

3.1. Phase I: metrics identification

Metrics identification has two steps: dimension development and instruction structure building-up. Dimension development is performed to identify the potential dimensions of E-creativity assessment scales. Followed by the detection of the potential indexes of E-creativity assessment scales. The initial instrument structure thus can be developed.

3.1.1. Dimension development

The original dimensions of E-creativity can be extracted from its definitions (Horn & Salvendy, 2006). Specifically, we searched the E-creativity-related papers in Google Scholar using the keywords ‘E-creativity’ and ‘exploratory creativity’ to identify the definition of E-creativity. Among the retrieved definitions, repeated and paraphrased definitions were removed from our list. Eventually, seven mostly related and representative definitions were selected (Table 1). Six dimensions – rules, feature, structure, concept space, exploratory process, and results – were then summarized from these definitions.

Table 1. Six dimensions are extracted from the definitions of literature.

References	Definition	Dimensions
Riedl and Young (2006, p. 303)	The process of searching an area of conceptual space governed by certain rules.	Conceptual space Exploratory process Rules
M. A. Boden (2009, p. 25)	The person moves through the space, exploring it to find out what’s there (including previously unvisited locations) and, in the most interesting cases, to discover both the potential and the limits of the space in question.	Conceptual space Exploratory process
Ritchie (2009, p. 78)	Exploratory creativity involves the navigation of a structured space of possibilities.	Structure
Hung and Choy (2013, p. 1 & 3)	Explores potentiality of a conceptual space to create a new and unexpected idea. We need E-creativity on structures and features respectively to help secure the balance between novelty (expressed in features) and appropriateness (enforced by structures) of a creative work.	Conceptual space Exploratory process Results Structure Features
Rayasam (2016, p. 18)	The result of exploring a given search space to its full potential and coming up with an idea.	Conceptual space Exploratory process Results
Karimi (2019, p. 260)	Developing new ideas through an established conceptual space.	Results Conceptual space
Karimi et al. (2020, p. 225)	Exploratory (creativity) occurs when users change parameters of some features associated with their current drawing (e.g., changing the size, scale, angle) based on features of the system’s response.	Features

Here, we used one example about how we identified ‘Conceptual space’ and ‘Exploratory process’ dimensions from M. A. Boden’s (2009, p. 25) definition to better explain how the various dimensions were identified and how were these extracted from the various definitions. First, we searched the E-creativity-related papers in Google Scholar using the keywords ‘E-creativity’ and ‘exploratory creativity’ to identify the definition of E-creativity. One of the E-creativity definitions has been promoted by M. A. Boden (2009, p. 25) which is ‘The person moves through the space, exploring it to find out what’s there (including previously unvisited locations) and, in the most interesting cases, to discover both the potential and the limits of the space in question.’ From this definition, the sentence can be divided as ‘the person moves through the space,’ ‘exploring it to find out what’s there (including previously unvisited locations),’ and ‘discover both the potential and the limits of the space in question.’ Based on the content, ‘the person moves through the space,’ are summarized as the ‘Conceptual space.’ ‘Exploring it to find out what’s there (including previously unvisited locations)’ and ‘discover both the potential and the limits of the space in question’ were summarized as ‘Exploratory process.’ Therefore, the definition of Boden included two dimensions which are ‘Conceptual space’ and ‘Exploratory process.’

Rules refer to the requirement for the changing point of E-creativity, such as the change of phone size. It indicates where the concepts are expected to change and give a guideline on which exploratory process should focus (Riedl & Young, 2006). Feature is the attribute of the concepts which can be detected from five senses, such as color and shape. When focused on the same rule, the features may be different. In other words, features can detail the changes among the generated concept. The changes in the features can indirectly report what has happened in an exploratory process (Karimi et al., 2020). Structure is the component of the generated concepts. It indicates how the exploratory processes build up the relations among the features (Karimi et al., 2020). Concept spaces are structured styles of thought. Although many possible thoughts exist within the given conceptual space, only some of them may actually have been thought (M. A. Boden, 2004; Wiggins, 2006). This is the boundary of E-creativity. If the concepts exceed this space, creativity will change from E-creativity to T-creativity. Exploratory process refers to the reasoning to an inference within the search (Wiggins, 2006). This is the core of E-creativity. Results refer to the E-creativity results. E-creativity needs to be displayed (Hung & Choy, 2013). In this way external people can assess it. Therefore, the E-creativity results are essential.

The six dimensions can also be explained from the information processing process in E-creativity assessment (Horn & Salvendy, 2006). When raters are asked to assess E-creativity, final E-creativity results and the design points are provided to raters (Lu & Luh, 2012). Raters need to understand the exploratory process behind E-creativity first and then assess E-creativity (Weiss & Wilhelm, 2020). This understanding includes the rules, features, structure, and concept spaces.

3.1.2. Instrument structure

The instrument is constructed based on the six dimensions, we can further build up the instrument for developing the E-creativity assessment scale. Specifically, we searched the keywords of the six dimensions (‘rule(s),’ ‘feature,’ ‘structure,’ ‘concept space,’ ‘exploratory process,’ and ‘result(s)’) within the papers retrieved in Section 3.1. Then, transformed the statements into antonyms. For example, we have identified that from Hung and Choy (2013), the E-creativity definition has been promoted. Therefore, we assumed that Hung and Choy (2013) may include the indexes which can be used to assess E-creativity. Then, we used ‘rule(s),’ ‘feature,’ ‘structure,’ ‘concept space,’ ‘exploratory process,’ and ‘result(s)’ as the keywords to search in the paper. We have found that when we used the ‘rules’ in the search, there was one statement that ‘Creativity Support Tools (a kind of tools which can be developed based on E-creativity) can be governed by the same set of rules (Hung & Choy, 2013, p. 3)’. This may indicate that one of the indexes of E-creativity is ‘governed by the same set of rules’. Since the study tended to use antonyms to represent the assessment index, the expression was transformed as ‘Follow the same rules lead to the same results – Follow the same rules lead to different results’.

Forty-two related statements were summarized from this method. Forty-two pairs of antonyms are then transformed from the statements. The whole 42 pairs and the statements are listed in Appendix 1. Examples on how the pairs are summarized from the statements are given in Table 2. Among the 42 pairs, the amount of each dimensions is respectively: seven in ‘rules,’ five in ‘feature,’ five in ‘structure,’ seven in ‘conceptual space,’ 13 in ‘exploratory process,’ and five in ‘results.’

Table 2. The examples of the initial E-creativity assessment scale. Each dimension takes one indexed pair as an example.

Dimension	Index	Pairs		Statements
Rule	1	Follow the same rules lead to the same results	Follow the same rules lead to different results	Creativity Support Tools can governed by the same set of rule (Hung & Choy, 2013, p. 3)
Feature	8	The concepts share same features	The concepts share different features	We need E-creativity on structures and features respectively to help secure the balance between novelty (expressed in features) … … (Hung & Choy, 2013, p. 3)
Structure	13	Same features form the same structure among concepts	Same features formdifferent structures among concepts	We need E-creativity on structures and features respectively to help secure the balance between novelty (expressed in features) … … (Hung & Choy, 2013, p. 3)
Conceptual space	18	Concepts belong to the same conceptual space	Concepts belong to different conceptual space	the process of exploring a given conceptual space (Kilicay-Ergin & Jablokow, 2012, p. 1232)
Exploratory process	25	The relationship among concepts is free	The relationship among concepts is constrained	… … is that the person himself makes the connections and the links all by himself unbridled (Rayasam, 2016, p. 18)
Results	38	Novelty and usefulness can be distinguished in the results of exploratory creativity	Novelty and usefulness cannot be distinguished in the results of exploratory creativity	A second useful definition is to discern between novelty and usefulness. (Dormans & Leijnen, 2013, p. 2)

Notably, the reason why we extracted antonyms from statements is that we want to reduce the bias from users on understanding the statements and make users understand the indexes more easily (Horn & Salvendy, 2006). This has been proved by various creativity assessment methods which used antonyms as the assessment index, such as Creative Product Semantic Scale (Besemer & O’Quin, 1986) and product creativity measurement instrument (PCMI; Horn & Salvendy, 2006).

3.2. Phase II: experimental determination

Two studies are conducted to determine the effective indexes in this section. The experts evaluation is first applied to judge all the antonyms from a professional perspective. Then cases were collected to verify the effectiveness of selected pairs of antonyms in a large-scale study, in which exploratory factor analysis (EFA) was applied to the analysis of study results (Groeneveld et al., 2022; Hazeri, 2019).

3.2.1. Experts evaluation

In the evaluation, four experts (aged 26–30; two males; two females) who are professional in design and creativity, are invited. The inclusion of four experts is that (i) the experts should have at least five-year experience in the design field (Kim et al., 2019). (ii) The experts are knowledgeable about computational creativity. (iii) The experts should have experience in using creative assessment scales. Experts need to conform to all the three criteria.

The reason why four experts were selected to perform the experts evaluation task is that this task aims to detect whether the promoted criteria can be used as the E-creativity assessment criteria from experts levels. Considering experts are not an easily accessible group, the participants number of this experts evaluation task is hard to achieve a high amount. The exact participants number of this study is learned from Miller (2014) who develop a creativity assessment scale. In Miller’s paper, two experts were recruited to finish the experts evaluation. Since the subjective risk in the limited number has been realized, four participants were recruited. However, the limited number of experts involved may introduce subjective biases and potentially diminish the study’s reliability

To ensure all experts understand what E-creativity is, the definition of E-creativity was given and explained. To be specific, E-creativity is defined as a kind of computational creativity which can generate novel ideas through the exploration of structured conceptual spaces. Then the experts were asked to assess the forty-two pairs of antonyms using a 5-point Likert scale to report their views on whether they think each pair of antonyms can be used to represent E-creativity (A score of one indicates the pair cannot represent completely a score of five score indicates complete representation). The evaluation lasted about ten minutes per expert. The results of the forty-two pairs are listed in Appendix 2, ranging from 1.75 to 4.75 (SD = 0.635). Cronbach’s alpha is 0.903, which indicates an excellent level of internal consistency.

The average score over three (not inclusive) was determined as the borderline to identify whether the antonyms can be used to assess E-creativity. The reason is that this justification is based on the 5-point Likert, where a score of three represents a moderate, neutral stance, indicating no clear tendency. A score which is less than three means a negative attitude toward whether the antonyms can be used to assess E-creativity. After the filtering, 28 valid pairs were left, which were marked with an asterisk in Appendix 2.

3.2.2. Design case selection

After the twenty-eight pairs were selected, they are further tested in design cases. Ten suitable E-creativity design cases were first identified in the scale development study with pre-defined criteria. The criteria are as follows: (i) the design point is clear and detailed; (ii) the idea development process can be summarized in a clear, brief, and understandable way; (iii) the idea development is not determined by mechanical or technical changes. Each design case was explained based on design areas, design points, exploratory process, and visual design case. The ten examples are collected from different product categories. One of the design cases is shown in Figure 2. All of the ten design cases are shown in Appendix 3.

Figure 2. A design case for E-creativity.

Nine participants (aged 20–25, three males, six females) who have design background were recruited. They were asked to evaluate whether the design case can be used as the case for E-creativity scale development. To be specific, the ten design cases were displayed in a random order. Participants first scrutinized each design case individually. They were then asked to use a 7-point Likert scale to report whether they thought this design case is good enough as the case for E-creativity scale development (one means it is absolutely unsuitable; seven means it is a perfect design case). The whole study lasted about ten minutes for each participant. The results of ten design cases are shown in Table 3. The average score of the ten design cases ranged from 3.56 to 5.56. The Cronbach’s alpha is 0.784, which indicated that the internal consistency is in an acceptable level. As a result, the top five ranked design cases were selected to be used in the following study.

Table 3. The mean value and SD results of ten design cases.

Design case	Mean	SD
iPad pencil*	5.56	1.42
Google logo*	5.44	1.01
Nike shoes*	5.11	1.45
Transformers*	4.67	1.50
Aeron Chairs*	4.44	1.51
iPhone	4.34	1.33
Tmall co-branded poster	4.33	1.22
Mickey Mouse	4.22	1.30
Olympic mascot	4.11	1.27
Lay’s potato chips	3.56	0.73

*means the top five cases which were used in design case study.

It is notable that existing research has several design cases. However, the design case of this study is based on product and graphic design. The case of existing research thus may not be a good selection. For example, the example from Dormans and Leijnen (2013) is based on game content; the example of Liapis et al. (2013) is based on shapes; the example of Jennings et al. (2011) is based on landscapes. Although games, shapes, and landscapes are categorized as design cases, our study specifically concentrated on product and graphic design. Additionally, the study tried to select practical case designs as examples, while the literature cases were all based on the authors’ own developed tools. Due to these two considerations, design cases from the literature review were not used as references in our study.

3.2.3. Scale development based on design cases

A case-based scale development is conducted to further test whether the filtered 28 pairs are suitable to be the E-creativity assessment indexes in the five design cases. A total of 116 participants (57 male, 59 female; aged 18–26) who are nonprofessional in design and creativity were recruited. They were asked to assess whether the given antonyms can be used to express E-creativity in the given design cases. If they thought an antonym can be used to express E-creativity in a given design case, they were required to choose the more appropriate word of the antonym pair and then rate its expressiveness on a scale from one to three (Score one means weak expression, Score two means mediate expression, Score three means strong expression). If they thought the antonym cannot be used to express, they need to select zero. There are five design cases and each design case has 28 pairs to be justified. The whole study lasted about 25 min for each participant.

Notably, we recruited nonprofessional participants to further identify E-creativity assessment indexes because E-creativity is not exclusive to professionals. Instead, it is also present in the ideation processes of nonprofessionals. To make the E-creativity assessment scale more universally applicable, we included nonprofessionals in determining the ratings.

3.2.4. Results

In the case study, there is no respondent with zero variation (SD = 0 or respondents who answered the exact same value for every question in the questionnaire). Therefore, the evaluations performed by 116 raters were checked to be valid. The skewness for the entire dataset ranged between −0.913 and 0.185 and the kurtosis ranged between −1.462 and −0.179. This indicates that the data is normally distributed (Field, 2013).

Based on the Independent Samples Test, the differences between females and males are not statistically significant (95% CI [−0.17658, 0.08175], t(114) = −0.727, p > 0.788). The inter-judge reliability has been calculated using Cronbach’s coefficient alpha. The consistency among the raters was also assessed and the results are shown in Table 4. The overall Cronbach’s coefficient alpha is 0.839 which indicated a ‘good’ level of internal consistency.

Table 4. Inter-judge reliability among five design cases.

	Design case 1	Design case 2	Design case 3	Design case 4	Design case 5	Overall
Cronbach’s coefficient alpha	0.725	0.813	0.693	0.696	0.76	0.839

The results were analyzed with the exploratory factor analysis (EFA). Prior to it, the degree of factorability of the measurement variables was initially evaluated. The Kaiser–Meyer–Olkin (KMO) measure of sample adequacy was 0.885, which indicated the sample adequacy was meritorious and greater than the commonly recommended threshold of 0.6 (Hair et al., 2010). Bartlett’s tTest of Sphericity was also significant (x² (378) = 5723.463, p = 0.000 < 0.001).

After applying the EFA, based on the screen plot, four factors were suggested and 47.51% of the variance can be explained. The oblique rotation algorithm was employed to interpret the underlying structure of the four-factor. All items with absolute loadings smaller than 0.3 were considered insignificant and hence removed (Horn & Salvendy, 2009). The items with more than one component, have absolute loading of 0.3 more, this may mean the items are related to different components and thus are not the best choices as the assessment indexes and thus were removed. Fifteen pairs were removed due to either low loadings or cross-loadings higher than the cutoff value (0.3) (Cropley & Kaufman, 2012). In consequence, eight pairs of antonyms were identified as the scale items for E-creativity. The exact results of the four-factor model extracted from the EFA analysis are shown in Table 5, in which the pairs are represented with index numbers. The whole correspondence relations between the index number and the pair can be found in Appendix 1.

Table 5. The four-factor model extracted from the EFA analysis.

Index	Dimension	Component
		1	2	3	4
34*	Exploratory process	0.716	0.227	−0.182	−0.003
33*	Exploratory process	0.691	0.233	−0.135	−0.025
10*	Feature	0.687	−0.116	0.246	0.079
15*	Structure	0.664	−0.127	0.155	0.009
1*	Rules	0.654	−0.021	0.155	0.066
4*	Rules	0.647	−0.045	0.102	−0.009
22*	Conceptual space	0.617	−0.088	0.17	−0.268
8	Feature	0.563	−0.072	0.183	0.262
40	Results	0.506	−0.275	−0.206	−0.226
20	Conceptual space	0.482	0.055	0.139	−0.224
3	Rules	0.176	0.568	0.181	−0.079
23*	Conceptual space	0.276	0.507	0.083	−0.058
21	Conceptual space	−0.068	0.481	0.227	0.032
32	Exploratory process	0.557	0.246	−0.531	0.108
25	Exploratory process	0.53	0.188	−0.451	0.134
2	Rules	0.525	0.308	0.105	0.094
19	Conceptual space	−0.518	0.27	−0.118	0.489
41	Results	0.452	−0.437	−0.118	−0.104
18	Conceptual space	−0.449	0.351	−0.054	0.439
9	Feature	0.138	0.636	0.338	−0.045
14	Structure	0.213	0.534	0.321	−0.014
7	Rules	−0.39	0.5	−0.047	−0.04
42	Results	−0.479	−0.262	0.546	−0.1
6	Rules	0.36	−0.349	0.24	0.519
17	Structure	0.423	−0.379	0.132	0.457
35	Exploratory process	0.368	−0.312	−0.042	0.044
13	Structure	0.631	0.04	0.304	0.098
26	Exploratory process	0.648	0.347	−0.26	0.006

*means the identified antonyms as the scale items from EFA analysis. *Bold font means the index in that component is significant.

A confirmatory factor analysis was then conducted to check the validity of the four components suggested by the EFA analysis. The results are shown in Figure 3. The covariance between Component 1 and 2 is 0.465 (<0.6), which means the Component 1 and Component 2 are independent.

Figure 3. Confirmatory factor analysis results of the four components.

For Component 1, the value of the chi-square test is significant (x² (65) = 318.254, p = 0.000 < 0.001). The ratio of chi-square to the degree of freedom (CMIN/DF) is 4.896, which indicates that the component is in reasonable fit. The goodness-of-fit index (GFI) is 0.915, which indicates an acceptable fit between the implied component and the observed data. The value of the root means square error of approximation (RMSEA) is attained at 0.000 which indicates a good fit between the implied covariance metrics and the observed data.

For Component 2, the value of the chi-square test is significant (x² (2) = 10.385, p = 0.000 < 0.001). The CMIN/DF is 5.192. The GFI is 0.991; The CFI is 0.878; The NFI is 0.861. These results indicates that the results of Component 2 is in a reasonable level. The value of RMSEA is attained at 0.085, which indicates a fit between the implied covariance metrics and the observed data.

From the above analysis, the pairs whose covariance between components and index are over 0.6 are selected as the E-creativity assessment scale (Hoque et al., 2018). The EFA results of the eight indexes are all positive values. This indicated that the positive and negative side for the eight indexes are suitable. The selected eight indexes are as shown in Table 6.

Table 6. The eight indexes which can be used to assess E-creativity.

Index	Pairs
	Negative	Positive
1	Follow the same rules lead to the same results	Follow the same rules lead to different results
4	Change of concepts are homogeneous	Change of concepts are heterogeneous
10	Changes of features are homogeneous among concepts	Changes of features are heterogeneous among concepts
15	Changes of structures are homogeneous among concepts	Changes of structures are heterogeneous among concepts
22	Conceptual space among concepts is consistent	Conceptual space among concepts is inconsistent
23	The novel levels of exploratory process and structure of concepts are irrelated	The novel levels of exploratory process and structure of concepts are related
33	The changes among concepts are simple	The changes among concepts are complex
34	The changes among concepts are small	The changes among concepts are big

4. Discussion

The experts evaluation and case study identified the indexes that can be used to report the existence of E-creativity. The rationale for the identified eight indexes is discussed as follows. Since E-creativity is related to following the rules to generate different creative ideas within a relevant conceptual space, Index 1 (which is related to rule dimension) and Index 22 (which is related to conceptual space dimension) are the pre-requirement of the existence of E-creativity. Index 23 is related to conceptual space dimension, but it is also the combination of exploratory process and the creativity. It suggests the existence of relations between creative levels and generated concepts in an exploratory process. As E-creativity is a special form of creativity, it needs to satisfy the basic requirements of creativity, which can be reflected by Index 4, Index 10, and Index 15 (Hung & Choy, 2013). Index 33 and Index 34 identify the existence of E-creativity from the perspective of creative results. The changes in ideas are originated from E-creativity and they can reflect the existence of E-creativity at some levels.

The E-creativity assessment indexes can be used to re-define E-creativity. From the results in the experimental determination phase, E-creativity is a novel concept-changing process. During its process, people can explore various concepts which have different features based on the same design point. The results of the experimental determination phase indicated the implicit relations and interaction among ideas behind the E-creativity process.

Although our studies and analysis provide preliminary evidence for validity of the scale, there are some statistic limitations throughout the research. For example, there are two components which only included one criterion. This limits the analysis on its significant levels and interaction with other components. In addition, the order of positive and negative phrases in each pair is fixed which may have an impact on the participants’ decisions, so more investigation is needed to understand this impact. Furthermore, the assessment scales were summarized from experts’ judgment. It is notable that limited research that we can learn from on what the E-creativity assessment criteria are. This is also the aim of our study – developing the E-creativity assessment scales to fill the existing gap in E-creativity assessment. Therefore, we followed the process suggested by Horn and Salvendy (2006) who get the assessment criteria by asking experts to justify whether it is the suitable assessment criteria based on cases. In other words, no literature somehow can confirm what the experts said. Also, although E-creativity could exist in various forms (such as games, shapes, and landscapes), our study focused on product and graphic design as examples. While we aim for the scale to apply broadly, it was only tested in these two areas. This limitation raises questions about its universality in other E-creativity design contexts. In the future, more areas will be considered.

Moreover, it is notable that the six dimensions are all from the definition of E-creativity. This is a method used by existing creativity assessment scale development processes (Horn & Salvendy, 2006). The dimensions beyond the definition are not included because the quality of the dimensions cannot be guaranteed. We applied this method and reviewed existing E-creativity definition. We searched the E-creativity-related papers in Google Scholar using the keywords ‘E-creativity’ and ‘exploratory creativity’ to identify the definition of E-creativity. We stopped the search until no more E-creativity definition appeared, ensuring a comprehensive collection. Thus, we considered the six dimensions are encompassing. However, we acknowledged that in the future, more methods to validate the scope of the dimensions should be considered. Finally, due to the limited research on the factors of E-creativity, the initial forty-two indexes and six dimensions may be not inclusive enough to represent the potential indexes of E-creativity completely.

5. Empirical study

This empirical study was conducted to compare the performances of professional and nonprofessional raters when using the scale. Specifically, we try to identify the differences in assessment scores and index results between nonprofessional and professional raters, as well as to compare their internal consistencies when using the scale to assess E-creativity.

5.1. Method

Twenty-three professional raters (fourteen female, nine male) and 37 nonprofessional raters (eighteen female, nineteen male) are recruited in this study. Professional raters are those who have more than five years experience in creativity and design while nonprofessional raters are those who have less than three years experience in creativity and design. The empirical study was designed and then conducted via an online questionnaire platform.

At the beginning of the questionnaire, participants were given an introduction about what E-creativity is and the concepts that would be used in the study. Then, participants were asked to assess to what degree the given antonyms can be used to express E-creativity in the given design cases. If they thought an antonym pair can be used to express E-creativity in the given design case, they needed to select which term of the pair is better and then indicated the degree from one to three (one means weak, while three means strong). If they thought the antonym could not be used to express the E-creativity of the given design case, they needed to suggest zero. An example case was given to ensure all participants can understand the assessment process. In the questionnaire, there were three design cases (Figure 4) and each one has eight pairs (listed in Table 6) to be judged. It took around 10 min to finish the questionnaire for each rater.

Figure 4. Design cases used for empirical study.

6. Results

Each index is a pair of antonym and only one in the antonym has the trend to represent the existence of E-creativity. For the convenience of analysis, the positive side degree is marked with a positive sign (1, 2, or 3); while the negative side degree is marked with a negative sign (−1, −2, or −3). The results of the differences between nonprofessional and professional raters in the three design cases are shown in Table 7.

Table 7. Results of the differences between nonprofessional and professional raters in the three design cases.

		The index of items
		Index 1	Index4	Index10	Index15	Index22	Index23	Index33	Index34	Overall average
Appeara-nce of Logitech mouse	non-professional	0.43	0.49	0.49	0.89	0.59	1.16	1.19	1.19	0.80
	professional	-0.83	0.39	0.22	1.00	0.09	1.30	1.00	0.74	0.49
	Sig	0.772	0.366	0.020	0.385	0.044	0.334	0.054	0.090	0.848
Icon of iOS	non-professional	1.22	0.81	0.68	0.76	0.54	1.43	1.05	1.03	0.94
	professional	0.57	0.83	0.65	0.96	0.48	0.17	0.39	0.96	0.63
	Sig	0.595	0.319	0.196	0.888	0.384	0.002	0.473	0.911	0.498
MIUI Sports Band	non-professional	0.19	0.16	0.27	0.78	0.27	0.95	0.68	0.51	0.48
	professional	-0.09	0.48	0.39	0.35	-1.04	0.61	0.09	0.61	0.18
	Sig	0.449	0.358	0.840	0.532	0.277	0.208	0.909	0.606	0.262
Overall	non-professional	0.61	0.49	0.48	0.81	0.47	1.18	0.97	0.91	0.74
	professional	-0.12	0.57	0.42	0.77	−0.16	0.69	0.49	0.77	0.43
	Sig	0.221	0.623	0.821	0.655	0.935	0.262	0.322	0.919	0.385

*The bold font means the higher score for the index in the design case.

The italic font means the negative side results for the index in the design case.

The underline font means the highest score among all the indexes in the design case.

The Cronbach’s alpha is 0.887 for nonprofessional raters while the Cronbach’s alpha is 0.882 for professional raters. This indicated that the both types of raters have very high consistency of the results.

7. Discussion

The ratings given by nonprofessionals and professionals nearly range from −1.5 to 1.5. This small score range may cause by the zero-centered 7 point-Likert scale. Participants are less likely to give extreme scores (3 or −3). From the results, it can be seen that the identification of the positive side is at a reliable level, especially for the nonprofessional group. In the professional group, Index 1 and Index 19 were reported as a negative side, which is opposite to nonprofessionals. The two indexes are related to the definition of E-creativity, so this difference can be explained that the professional and nonprofessional raters have a different understanding of E-creativity.

By comparing the results between nonprofessional and professional raters, it can be found that the scores given by nonprofessional raters are often higher than that of professional raters. This may be because the nonprofessionals have a loose requirement on E-creativity and thus affected the rating results. The exceptional case is that professional raters give a higher score of Index 15 compared with nonprofessional raters.

Comparing the scores among eight indexes, it can be seen that Index 23 is the most important index for E-creativity from nonprofessionals’ perspectives. On the other hand, Index 15 and Index 34 are the most important index for E-creativity from professional people’s perspectives.

Although there are some score differences between professional and nonprofessional raters, the results report no statistical significance (p=0.385). This indicates that although the differences existed, the difference between professional and nonprofessional raters is not significant. When we used the developed E-creativity assessment scale, the professional levels may not affect the assessment results in a significant level. Also, it is noted that the empirical study in Section 5 is conducted based on our metrics identification and experimental determination study in Section 3. The reliablity of the studies in Section 3 may affect the results in Section 5.

8. Conclusion

This research developed an E-creativity assessment scale, in which eight indexes were promoted in the metrics identification and experimental determination studies. These indexes are related to the E-creativity attributes, pre-requirement for the existence of E-creativity, relations between exploratory process and the creativity, and E-creativity results. An empirical study was then applied to verify the difference between nonprofessionals and professionals. The results suggest that the scores given by nonprofessional raters are often higher than that of professional raters, but the difference is no significant.

Apart from promoting an E-creativity assessment scale, the results of this study also enhances understanding of E-creativity and contributes to refining the definition of E-creativity. The identified indexes can guide systematic training and assessment of E-creativity by humans or machines. Our research results reflected the scale of E-creativity in a creative process, especially in a product creation process. It is revealed that E-creativity is not simply related to the exploratory process and its concept space; it also encompasses the relationship the novelty of these processes and the concepts generated from exploratory process.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/21650349.2024.2319772

Additional information

Funding

The work was supported by the National Key Research and Development Program of China [2022YFB3303304]; National Natural Science Foundation of China [62207023].