“C”ing the light – assessing code comprehension in novice programmers using C code patterns

ABSTRACT

Background and Context

Feedback on one’s progress is essential to new programming language learners, particularly in out-of-classroom settings. Though many study materials offer assessment mechanisms, most do not examine the accuracy of the feedback they deliver, nor give evidence on its validity.

Objective

We investigate the potential use of a preexisting set of C code snippets as the basis for a high-quality C programming ability assessment tool.

Method

We utilize the Rasch Model and the Linear Logistic Test Model to evaluate the validity and accuracy of the code snippets and to determine which C operations contribute most to their overall difficulty.

Findings

Our results show that these code snippets yield accurate assessments of programming ability and reveal the degree of difficulty associated with specific programming operations.

Implications

Our results suggest that the code snippets could serve as the basis for sophisticated, valid, and fair code comprehension skill assessment tools.

KEYWORDS:

• Programming skills
• C language
• assessment tool
• measurement quality

1. Introduction

With the increasing complexity of software codebases and the emergence of interconnected devices to handle everyday tasks, the demand for proficient programmers has steadily grown. The U.S. Bureau of Labor Statistics reported that, between 2020 and 2030, the demand for programmers will grow by 22%, substantially higher than the average growth rate of 8% for other job categories (Bureau of Labor Statistics, U.S. Department of Labor, 2022). Keeping up with this demand is challenging, largely because of the difficulty of mastering programming languages.

Though there is currently no shortage of books, education platforms, and videos that can teach almost any programming language, many may not provide sufficient repetition to help learners internalize new concepts. More importantly, these resources often lack a way to give accurate and reliable feedback so students can judge if they are making progress. In a classroom setting, test taking can provide such feedback, and can enhance learning performance and achievement (McDaniel & Fisher, 1991; Vollmeyer & Rheinberg, 2005). A novice developer can also find online tools and resources to test acquired skills (see, e.g. Brusilovsky & Sosnovsky, 2005; Dancik & Kumar, 2003; Lobb & Harlow, 2016). The question is how accurate are the assessments these resources provide, and how representative are the results? To date, little work has been done to verify the validity of these educational tools (i.e. whether they actually measure the targeted ability) and the reliability of the generated measures in terms of accuracy.

In this paper, we evaluate an existing set of student test performance data, originally collected as responses to a series of C code snippets (Gopstein et al., 2017), to judge the set’s ability to yield fair, valid, and accurate measurements of student performance. Employing Item Response Theory (IRT) – specifically the Rasch Model and the Linear Logistic Test Model – we assessed two different groups of student responses to these snippets. We also were able to pinpoint the components within the code that contribute to the overall difficulty of the snippets.

Our initial results not only suggest that the snippet set could be used to measure programmers’ C skills, but can also provide insights into which C operations are most difficult for novice programmers to comprehend. With the insights from this work, we envision an ability to design adaptive tools for programming languages that can automatically adjust to advanced programming topics as students demonstrate their mastery of previous concepts. These snippets could help more accurately assess a student’s comprehension of programming topics in a way that existing performance-based tools cannot.

2. Background and related work

In designing our study, we reviewed previous work in code comprehension theories and approaches, and strategies for measuring latent abilities. In this section we identify and summarize these approaches and offer some background on the dataset of C code snippets used in the analysis.

2.1. Code comprehension studies

Program comprehension, or code comprehension, is a broad topic that has been studied from an extensive number of perspectives. Some of these studies look at factors within the code itself, while others look towards identifying programmer perceptions when engaging with a piece of code.

In the former category are studies like Hofmeister et al. (2019), who looked at the effect of different identifier naming styles (single letters, abbreviations, and words) on program comprehension. Hofmeister et al. (2019) asked 72 professional C# developers to locate defects in source code snippets. They found that code is more difficult to comprehend when identifier names contain only letters and abbreviations, as opposed to words. Bauer et al. (2019) updated an earlier study on whether different levels of indentation in code played a role in developer comprehension. Buse and Weimer’s work Buse and Weimer’s work (Buse & Weimer, 2008) studied local code features of small program snippets.

A few of the specific code patterns investigated by Gopstein et al. (2017) were also the subject of earlier investigations. Elshoff and Marcotty’s (1982) work introduced the idea of clarifying transformations to improve the readability of source code. Dolado et al. (2003) tested whether code that contained side-effects was more likely to cause subjects to misinterpret its function.

In the latter category of work that focused more on developer perceptions, there are studies like Duran et al. (2018), that proposed assessing the complexity of a program from a learning point of view. Using a number of short programs as case studies, the authors applied this approach as a way to illustrate why one program or construct is perceived as more complex than another. Tashtoush et al. (2013) also designed a model of software readability by asking programmers questions about what features they found confusing. Jones (2006) looked for a correlation between developers’ knowledge of relative binary operator precedence and the amount of actual experience they have had handling the respective binary operator pair.

Lastly, several of these studies provide a rationale for the importance of developing code comprehension skills. Izu et al. (2019) observed that “Novice programmers should develop program comprehension skills as they learn to code so that they are able both to read and reason about code created by others, and to reflect on their code when writing, debugging or extending it”. Towards this end, Izu’s study collects and defines learning activities that explicitly address key components of program comprehension. On a more practical note, improving code comprehension skills can reduce the amount of time software developers must spend on this task. Feitelson (2023) cited two different studies (Minelli et al., 2015; Xia et al., 2018) that found, on average, developers spend between 58% and 70% of their time on code comprehension activities. Thus, a better understanding of code could free programmers to spend more time and resources on other tasks.

2.2. Assessment tools

Novice learners can choose from a number of web-based auto-gradable assessment tools. Dancik and Kumar’s web-based C++ tutor (Dancik & Kumar, 2003) even allows users to specify the preferred level of explanations about the correct answer (no feedback, correct/incorrect feedback, or detailed explanations about the correct answer). Other tools, such as those in Lister et al. (2004) use a multiple choice format or ask participants to fill a blank in the code assignment with the correct statement. A limitation of these tools is that, unlike similar instruments used for personality tests (e.g. DeYoung et al., 2007) and intelligence tests (e.g. Cooper & Petrides, 2010), many programming skill tests do not examine the quality of the measurements they generate. However, several exemplary approaches do exist (see Margulieux et al. (2019) for a comprehensive review of validated instruments in Computing Education Research). For example, FCS1 (Tew & Guzdial, 2011) and SCS1 (Bockmon et al., 2019; Parker et al., 2016) are multi-lingual validated instruments for content knowledge assessment. In addition, results generated by Bergersen et al. (2014) instrument for measuring Java programming skills, and Herman et al. (2014) evaluation of the digital logic concept inventory were both extensively evaluated. Yet, many other tools implicitly assume that their measurements are reliable and fair, but use no formal mechanisms to test these assumptions. Measurement models, such as the ones defined in Section 2.3, enable researchers to examine whether crucial assumptions about the measurement process are met. If the assumptions are valid, researchers can trust that the measurements generated are accurate (Molenaar, 1995b, p. 11f).

2.3. Measuring abilities: item response theory

Assessing a student’s ability at solving any particular task is difficult because human personality traits and abilities cannot be measured or observed directly. Instead, these latent or hidden constructs can only be inferred through human behavior. Generally, a student’s progress is measured by performance on tests. The higher the score a student receives, the higher we assume their capabilities in that subject. During performance testing, we also infer that solving harder items¹ requires a higher level of ability than solving easier items. Yet, even the most capable students might make mistakes, for example, if they read the problem incorrectly. Conversely, students with limited abilities might get a few difficult items correct by making lucky guesses. Therefore, a student’s test performance does not reflect their actual latent ability perfectly. There likely will be some statistical noise in their results.

One method to deal with “noise” when measuring constructs like ability is Item Response Theory (IRT; Ayala, 2022; De Boeck & Wilson, 2004; van der Linden, 2016). IRT assumes the correctness of a test taker’s response to an item is probabilistic rather than deterministic. Given a test item with a specific difficulty, a student with a higher latent ability should be more likely to solve it correctly than a student with a lower latent ability in the respective subject. By modeling the response process as probabilistic, we naturally take into account the existence of “noise” in the answers. Conversely, we expect that answers from students with a specific latent programming ability are more likely to be correct for easy items than for difficult items. Therefore, the probability to correctly answer an item depends on both item difficulty and a student’s abilities. In the following, we introduce two IRT models used in our study, the Rasch Model and Linear Logistic Test Model.

2.3.1. Rasch Model

In our study, we used two IRT models to evaluate responses to a dataset of C code snippets, described in more detail below. The first is the Rasch Model (RM), which allows for estimation of both latent abilities and item difficulties based on a set of binary data $\mathit{X}$, i.e. correctness/incorrectness of subjects’ answers on items (Molenaar, 1995b; Rasch, 1960). In the RM, the probability that a person $\mathit{v}$ ($\mathit{v}=1,\dots ,\mathit{n}$) with a latent ability ${\theta }_v$ (e.g. programming skill) gives a correct response to item $\mathit{i}$ ($\mathit{i}=1,\dots ,\mathit{k}$, e.g. coding task) with a difficulty of ${\beta }_i$ is defined as:

(1) $\mathit{P}(X_{\mathit{vi}}=1|{\theta }_v,{\beta }_i)=\frac{{\mathrm{e}}^{{\theta }_v-{\beta }_i}}{1+{\mathrm{e}}^{{\theta }_v-{\beta }_i}}.$(1)

Researchers can obtain estimates for both subjects’ ability parameters (${\theta }_v$) and the item difficulty parameters (${\beta }_i$) (Mair & Hatzinger, 2007a; Molenaar, 1995a) by using Maximum Likelihood Estimation, a common statistical estimation method. However, the RM is based on a series of assumptions, such as:

• Local stochastic independence, i.e. the probability that a person can solve an item correctly does not depend on whether this person solved another item correctly;

• Measurement invariance between subgroups. This means for each subgroup of persons (e.g. grouped by sex or nationality), the items measure the construct equally, i.e. “fairness” in the measurement;

• Equal discriminatory power of all items, i.e. all items measure the respective ability with equal precision;

• Unidimensionality, or all items measure the same latent ability, i.e. comprehending C code.

Validating these assumptions requires several likelihood-ratio tests (LRT; Andersen, 1973) with different split criteria, such as by sex, lower/higher ability group, or nationality. For any given item set, all of the model’s assumptions would need to be met for the model to hold (Fischer, 1995a). Should the RM be deemed valid for an instrument, one can derive several valuable properties. First, the unweighted sum test scores of a group of participants are sufficient estimates to judge their abilities (Fischer, 1995a, p. 17); that is, it is statistically admissible to interpret a person’s total sum score as representative of their ability level. Second, ability parameter estimates are metrically scaled instead of being ordered categorical or nominal values. Therefore they can be treated as metric values in further analyses. Third, item scores given by the participants are sufficient estimates to judge an item’s difficulty level, and the instrument measures fairly so no subgroups of participants are being discriminated against. Finally, the measure is specifically objective, which means that no matter the subset of items a person was given during a test, the ability estimate is still valid (Fischer, 1995a, p. 27ff).

The specific objectivity of RM also allows for adaptive testing (van der Linden & Glas, 2010). In adaptive testing procedures, test takers start off with an item of moderate difficulty and, depending on the correctness of their answer, the following item will be either easier or harder. By always adapting the item difficulty of the subsequent item based on one’s ability demonstrated in previous items, test takers will only be confronted with items suiting their ability levels (van der Linden & Glas, 2010). This is in contrast to conventional procedures in which test takers have to answer a multitude of items, including many that will either be too easy or too hard. This under- or overload in conventional procedures can lead to fatigue or annoyance in test takers. In contrast, adaptive testing can counteract these issues and result in more accurate assessments.

2.3.2. Linear logistic test model

To take a closer look at which operations make an item difficult, we used a second IRT model called the Linear Logistic Test Model (LLTM; Fischer, 1995b; Scheiblechner, 1971, 1972). In essence, the LLTM is an RM that confines itself to estimating only the difficulty parameters of reoccurring item properties. For any given item, such as the code snippets examined in this study, LLTM models its estimated level of difficulty as a result of the individually challenging properties (i.e. “subtasks”) that must be completed. Our work refers to these individual properties as basic components. For example, if a math equation involves an addition and a multiplication, the entire equation is an item, and addition and multiplication are its basic components. However, these two distinct operations may differ in their levels of difficulty (for an extensive example, see Anderson et al., 2022). Using LLTM, we can identify which components make an item difficult to solve. As this decomposition of item properties entails additional explanatory power, the model is discussed as an item explanatory IRT model (De Boeck & Wilson, 2004).

Using LLTM, item $\mathit{i}$’s difficulty parameters ${\beta }_i$ in an RM are decomposed into a number of basic components ${\eta }_j$ ($\mathit{j}=1,\dots ,\mathit{p}$) that each adds a degree $w_{\mathit{ij}}$ to item $\mathit{i}$’s difficulty ${\beta }_i$. In LLTM, the RM-formula given in (1) is employed with the following substitution:

(2) ${\beta }_i=\sum _{\mathit{j}=1}^p{w}_{\mathit{ij}}{\eta }_j$(2)

Identifying basic components in LLTM occurs a priori through a theory-based or rational investigation of the items. As illustrated in our previous example of a math equation, researchers need to go through all items and identify what operations are present. The basic components constituting each item are then summarized in a $\mathit{k}\times \mathit{p}$ matrix containing weights $w_{\mathit{ij}}$, where $\mathit{k}$ is the number of items and $\mathit{p}$ is the number of basic components across all items. Typically, the value of $w_{\mathit{ij}}$ is 1 or 0, indicating whether or not a specific component $\mathit{j}$ is present in item $\mathit{i}$.

Since an LLTM is an RM with additional restrictions, the validity of the underlying RM needs to be examined first. As noted by De Boeck and Wilson (2004, p. 62), the item difficulty parameters ${\beta }_i$ retrieved by an LLTM will not perfectly equal the difficulty parameters estimated by the RM. Therefore, we need to test statistically if the LLTM parameters (depicting our “explanation”, $w_{\mathit{ij}}$, for the item difficulties) describe the respective dataset equally well as the RM parameters. The restrictions imposed by LLTM can be tested using likelihood-ratio tests (LRTs) and graphically inspected by plotting the models’ parameters against each other (Fischer, 1995b, p. 147).

2.4. C code snippets: atoms of confusion

The C code snippets used in this work had previously served as the dataset in a series of studies designed to identify elements in code that can cause programmers to misjudge outputs (Gopstein et al., 2017, 2018, 2020). Gopstein et al. (2017) developed an item set of C code snippets, initially selected from the International Obfuscated C Code Contest, and therefore acknowledged as difficult to understand. The research team labeled them as “atoms of confusion” (or atoms for short), because they consisted of the smallest possible snippets of code ($5$–$7$ lines of code on average) capable of confusing a programmer about its output. For each of these atoms they created a transformed, non-confusing version. An example pair of confusing and non-confusing code snippets can be found in Figure 1.

Figure 1. Example for a snippet pair that is supposed to be non-confusing (a) or confusing (b), depending on whether there is an assignment of a constant inside the while condition.

Gopstein et al. (2017) conducted a study in which participants from the US hand evaluated the code snippets, and found most of the items considered confusing were indeed harder to answer correctly. Subsequently, Zhuang et al. (2023) conducted the same experiment with a group of participants from China in order to investigate perception and performance differences between the American and the Chinese population.

In the original studies, the authors were primarily concerned with identifying types of confusing coding patterns. While these types mostly referred to one specific operation within the item that was expected to cause the confusion, the categories did not account for all the operations present in the respective code snippet. For example, to correctly answer the item posed in Figure 1(a), subjects would need to have a mastery of the method of a while loop itself, the not (!) operator, and the while condition.² In this study, we investigate the item set created by Gopstein et al. (2017) from a more detailed perspective. As a result, we contribute novel insights by disentangling the operations in these C snippets that must be correctly completed for an accurate response.

The real-world presence of C patterns from the dataset ranges from very common to infrequent (Gopstein et al., 2018). Gopstein et al. (2018) also stated that the ubiquitousness of these patterns within open source projects seems to be determined by their acceptance (or rejection) among developers within particular software engineering specializations. As to what the students were or were not taught prior to our testing, it would have been impossible to collect syllabi at the time of data collection since the tests were conducted at different sites and times. Yet, based on the experience of the authors from their own campuses, it is believed most of the patterns invoke programming language constructs that would be taught (operators, order of operations and executions, etc) in most CS1 and CS2 classes.

3. Research goals

In our study, we employ the Rasch Model to evaluate whether the code snippets from Gopstein et al. (2017) could serve as a basis for a C skills assessment tool. If the RM holds, we can reasonably assume the beneficial properties outlined in Section 2.3.1 to be valid. Thus, we can assume that responses to the snippets can provide an adequate assessment of C programming skills.

Using RM to assess the quality of measurement and LLTM to examine different properties of the code snippets (e.g. logical operations or loops used in the snippet), we seek answers to the following questions:

• Can we affirm the quality of the items in terms of how accurately they measure comprehension?

• How difficult for novices are the various C operations in the snippets?

If our RM and LLTM analyses yield satisfying results, we can use these items for adaptive testing and automated test construction. Moreover, the results may provide insights for further improvement of the item pool, which we discuss in Section 7.

4. Instrument, dataset and study methods

4.1. Instrument and study sample

The instrument for this study utilizes the original “atoms of confusion” developed by Gopstein et al. (2017). Half of the snippets are considered non-confusing to subjects while the second half are considered confusing (see Figure 1 for an example). Although the confusing snippets were derived from the International Obfuscated C Code Contest, a later study revealed that these code patterns were prevalent in actual open source projects. “There are over 3.6 million atoms in 14 popular open source C/C++ projects and 4.38% of lines have an atom” (Gopstein et al., 2018, p. 1). Considering this high prevalence of confusing atoms in real world projects, and that 50% of the snippets in the item set are also transformed/non-confusing, we hypothesize the items are worth examining as a possible assessment tool. It should be noted though that 18 of the 71 code snippets we finally analyzed were designed to be answered with a true/false response. This type of item is not ideal for the IRT analysis we would conduct because it increases the chance of responses based on pure guessing. However, we decided to keep them in our dataset as the aim of our research is to analyze the current set of code snippets and its measurement properties as is.

Our study used two data samples, one from US participants ($\mathit{n}=73$ participants; Gopstein et al., 2017) and one from Chinese participants ($\mathit{n}=60$ participants; Zhuang et al., 2023). In the latter sample, our analysis excludes data from five subjects who did not report their amount of programming experience. We only considered items that met the following criteria:

1. They were exactly the same code snippets in both samples: In the original studies, subjects had only received a random subset (66%) of all the items in order to prevent fatigue. To ensure comparability, we only included snippets that completely matched in both samples.

2. They were shown to more than $45\mathrm{\%}$ of the subjects: Same as the reason above, the two datasets contain missing values. To avoid imprecision in estimation, we excluded items with too many missing values.

3. They did not result in all correct or all wrong answers across participants: When all participants got their answer right (or wrong) for an item, the estimation algorithm would have to calculate the logarithm of 0 (or divide by zero in case all answers are incorrect), which is undefined. Thus, one can not determine a valid difficulty estimate for these items.

The dataset analyzed in the current study therefore consisted of answers of $\mathit{n}=128$ subjects on 71 items. Demographically, the sample consisted of participants from the US ($n_{\mathrm{US}}=70$) and from China ($n_{\mathrm{CH}}=58$). Of our 128 subjects, $81.25\mathrm{\%}$ were self-reported as male, and the rest self-reported as female. Subjects had an average experience with C programming of $\mathit{M}=18.45$ ($\mathit{SD}=26.19$) months and a median experience of $\mathit{Md}=10$ ($Q_1=6$, $Q_3=18.5$) months. At the time of the study, $86.61\mathrm{\%}$ of the participants were in Computer Science and Engineering bachelor’s degree programs. On average, subjects correctly answered $\mathit{M}=75.77\mathrm{\%}$ ($\mathit{SD}=13.92\mathrm{\%}$) of the atoms they were shown. There was a low correlation between months of experience in C programming and the accuracy rate with $\mathit{r}=.274$, $\mathit{p}=.002$.

4.2. Research methods

4.2.1. Decomposition into basic components

The first step in our study was to conduct an a priori decomposition of the C-code items³ (“atoms”) created by Gopstein et al. (2017) to assess the basic components of each item. We acknowledge that there are several approaches to decomposing programming code, e.g. Soloway’s (1986), Schulte’s Block Model (Schulte, 2008), or the model of hierarchical complexity (Commons et al., 1998; Duran et al., 2018). While these models offer a profound understanding of code complexity, they emphasize hierarchical structures of code operations and their dependencies and interactions. However, our study aimed to analyze the difficulty of underlying basic components. So, for this purpose, we followed the approach developed and used by Luxton-Reilly et al. (2017), in which a group of experts analyzed already small pieces of code to see how they could be decomposed into even smaller tokens. They do so because they found that “seemingly straightforward questions can involve a relatively large number of tokens, and that misunderstanding any of these tokens can prevent students from answering the original question” (Luxton-Reilly et al., 2017, p. 62). As this conception is fully in line with the theoretical assumptions of LLTM, we decided to follow this principle.

Utilizing the approach of Luxton-Reilly et al. (2017), we manually and sequentially went through all the code snippets with the following procedure. We started with a table listing each snippet in a row. Beginning with the first snippet, we identified which tokens (i.e. basic operational components) are present, added a column for each unique component, and marked the component in the column with a 1. We repeated this process for all the snippets (rows) in the table. Whenever a new component is identified that doesn’t fall within an existing column, we added a new one and placed a 1 in the cell accordingly. Iterating this process for all snippets several times to ensure the correctness of the matrix, we obtained our final table. It comprises the established item characteristics in a binary weight matrix, where $w_{\mathit{ij}}=1$ indicates the presence of component $\mathit{j}$ in item $\mathit{i}$, and $w_{\mathit{ij}}=0$ indicates its absence. Our a priori defined matrix consisted of $\mathit{p}=26$ basic components occurring in the $\mathit{k}=71$ items we used for our analysis, resulting in a $71\times 26$ weight matrix. The matrix is available as supplementary material.⁴ The 26 basic operational components we identified are listed and described in Table 1.

Table 1. The 26 basic operational components we identified in the items and short descriptions of the challenges that arise from each of them. These components represent the columns of our weight matrix.

Component	Description of the Challenging Aspects of a Component
if-else statement	To solve this operation, one has to be familiar with the logic of if-else.
condition equals zero	There is a test expression in an if statement or while loop, which evaluates a single value to false if the value is 0 and to true in any other case.
logical expression	The atom contains a logical expression of any kind – one has to know that these evaluate to true or false and how to further treat these statements.
comma separator	The snippet contains a comma that serves as a separator in the item. It has to be recognised as a delimiter and must not be confused with a comma operator.
comma operator	The code snippet contains a comma serving as a binary operator. To solve this item, one has to know that the first element will be evaluated and discarded, and then the second element will be evaluated and returned. This operator must not be confused with a comma separator.
AND operator	Subjects need to know that the overall statement evaluates to true only if both operands before and after a AND operator (&&) evaluate to true.
OR operator	In order to solve this snippet, one has to know that a logical statement with an OR operator ($\parallel$) evaluates to true, if at least operand evaluates to true.
comparison operator	Within a logical expression in the atom, there is a numerical comparison ($\ge ,>,\le ,<$). Subjects have to be familiar with these symbols and need to know their differences.
NOT operator	To solve this item, one needs to know the NOT operator ($!=$) and apply it correctly.
equal operator	One has to apply an equal operator ($==$) within a logical expression correctly and must not confuse it with an assignment operator ($=$).
modulo operator	Solving the atom requires subjects to know the modulo operator and to use it correctly.
ternary operator	Solving the item requires the subject to be familiar with the ternary operator ($?:$).
maths	To solve the item one is required to apply some mathematical operations going beyond simple addition or multiplication (e.g. operator precedence).
while-loop	One has to know how a while loop works and be able to apply the logic.
for-loop	One has to know how a for loop works and be able to apply the logic.
post in-/decrement	The post increment and decrement operator has to be known, applied correctly, and it must not be confused with a pre increment/decrement operator.
pre in-/decrement	Subjects have to know pre in- and decrement operators and must not confuse them with post increment/decrement operators.
macro	The item contains a macro, which subjects need to understand and correctly apply to the code.
preprocessor	There is a preprocessor statement within the code, which has to be identified as such and subjects have to know that these are evaluated at first.
hexadecimal numerals	The code contains numbers in hexadecimal numeral system, which have to recognised as such and have to be translated to decimal.
octal numerals	The code contains octal numerals, which have to recognised as such and have to be translated to decimal.
type conversion	Some variables in the code will be (implicitly) converted to a different variable type by the compiler, which subjects need to anticipate to master the item.
variable type	The code snippet contains a variable type other than integer. One has to be familiar with other variable types, recognise them as such and treat them correspondingly.
semicolon terminator	Subjects need to be aware that the semicolon terminates the previous statement and that after the semicolon a separate statement begins.
zero based numbering	The atom contains indexing and subjects have to be aware of that the sequence starts with zero.
reversed subscripts	Subjects need to know that the subscript operator is commutative in order to solve these items.

4.2.2. Analysis

We began by estimating an RM and looking for any potential violations of its assumptions. We tested the hypothesis that “the RM holds” through likelihood-ratio test (LRTs). If the RM does not hold, it indicates that one or several of the model’s assumptions, such as dependency between items, varying measurement precision of items, or unfairness of measurement across subgroups, have been violated (Fischer, 1995a). We then graphically inspected the distribution of item difficulties and ability parameters. Following that, we estimated an LLTM based on our weight matrix and compared the model against the RM using an LRT (Fischer, 1995b, p. 147). The correlations between RM parameters and LLTM parameters served as additional indicators of how well our LLTM (specifically our weight matrix) recovers the RM parameters and therefore serves as goodness-of-fit measures. We chose a level of $\mathit{\alpha }=.01$ for conducting significance tests. In order not to inflate alpha when testing multiple times, we corrected $\mathit{\alpha }$ to ${\alpha }^{\ast }=.01/\mathit{m}$, where $\mathit{m}$ is the number of tests conducted at one time.

Both the RM and the LLTM are estimated using conditional maximum likelihood estimation (CML; Mair & Hatzinger, 2007a; Molenaar, 1995a; Scheiblechner, 1971). We used the statistical software R (R Core Team, 2021) to analyze the data, and the R package eRm (Extended Rasch Modelling; Mair & Hatzinger, 2007b; Mair et al., 2021) for further analysis of the estimated RM and LLTM. Any structural values that were missing because subjects did not see every single item were handled by having the eRm package compute separate parameters for each item subgroup with the same missing patterns. Estimates were then “aligned” using items that the subgroups have in common (Fischer & Ponocny, 1994; Mair & Hatzinger, 2007b).

5. Analyses results

Below we present the results of our two rounds of analysis. Section 5.1 describes how we tested the validity of our Rasch model, and what it revealed about participant abilities and the relative difficulty of the snippets. In Section 5.2, we enumerate what the LLTM model revealed about the snippet components, and compare the relative validity of the two models.

5.1. Rasch model analysis

5.1.1. Rasch model validity

To assess the RM’s validity, we conducted LRTs for the three different split criteria: (1) median person score (i.e. higher/lower ability group), (2) population (US vs. China), and (3) median programming experience (i.e. high/low experience group). The results are shown in Table 2. None of the LRTs showed significantly different parameter estimates for the investigated data subsets for the significance level where ${\alpha }^{\ast }=0.01/3=.0033$ is corrected for $\mathit{m}=3$ conducted tests. Therefore, the RM holds for this dataset.

Table 2. Results of the conducted LRTs for the split criteria of median of person score, population, and median of experience. The columns refer to the ${\chi }^2$ test statistic (LR), the degrees of freedom $\mathit{df}$, the critical ${\chi }^2$ value for a corrected significance level ${\alpha }^{\ast }=0.01/3=.0033$ of the respective test distribution, as well as the associated $\mathit{p}$-value.

Split criterion	${\chi }^2$	$\mathit{df}$	${\chi }_{\mathrm{crit}({\alpha }^{\ast })}^2$	$\mathit{p}$-value
Person score (median, high/low)	82.41	51	82.63	0.0035
Sample (US, Chinese)	73.28	62	96.45	0.1547
Experience (median, high/low)	70.05	61	95.20	0.1999

The item difficulty estimates for both subgroups in all three split criteria are depicted in Figure 2, which was created using the GMX package (Alexandrowicz, 2022a, 2022b). This graphical model allows us to inspect item difficulties across subgroups. By relating each item’s difficulty in both subgroups, we gain insights as to whether the items are equally difficult in both groups. Item difficulty estimates lying on the 45° line are equally difficult for both groups in the comparison. In Figure 2, all confidence ellipses, or two-dimensional confidence intervals overlap with the 45° diagonal line. This indicates that, statistically, the items are equally difficult for all subgroups and the code snippets do not discriminate against subgroups. Thus, a first analysis of our RM does not reveal violations of the model assumptions and we can retain the null hypothesis “The RM holds”.

Figure 2. Graphical model checks for the three LRTs. Each plot depicts item difficulty parameter estimates (“betas”) for the subgroups of each LRT (person score, population, and experience). Blue dots represent item difficulty estimates, grey ellipses represent the two-dimensional confidence intervals (${\alpha }^{\ast }=.0033$) associated with the item estimates.

5.1.2. Person and item parameters

The RM yielded well-balanced distributions of person and item parameters, as depicted in Figure 3. The upper section shows the distribution of the subjects’ ability parameters, where larger values on the horizontal axis indicate higher C programming ability. Similarly, larger values on the horizontal axis of the lower section indicate the greater difficulty of an item. Note that the upper and lower sections of Figure 3’s horizontal axis are aligned at the same values. If a person has an ability parameter equal to the difficulty parameter of a specific item, said person has a 50% chance of mastering that item. Those with less ability have a lower chance of giving a correct answer, whereas those with higher ability have a significantly better chance of doing so.

Figure 3. Person-item map of our RM. The upper histogram reflects the distribution of our subjects’ ability parameters. The lower part of the figure depicts the item difficulty for every item in our dataset. The $\mathit{k}=71$ items are marked by horizontal ticks on the left and are ordered by difficulty (from easy items on top to difficult items on the bottom).

From Figure 3, we can see that the distribution of item difficulty parameters and subject ability parameters cover slightly different ranges. Many items fall into the lower spectrum of item difficulty and, thus, are quite “easy items” for our sample. In Figure 3, we find some items with a difficulty lower than $\mathit{\theta }=-1.5$. All persons in our sample have programming abilities clearly exceeding the ability required to solve these items. On the other hand, some subjects in our sample have very high programming abilities (e.g. subject ability parameters greater than $\mathit{\theta }=4$), but there are little to no items that fall within this spectrum. Although the snippets are indeed capable of assessing subject abilities outside their item difficulty range, measurement precision is higher for subjects within the overall range of item difficulties (Ayala, 2022, p. 31ff.). Thus, we find the item set to be rather easy for the present sample (who, on average, had more than one year of programming experience) and the distributions suggest that our item set would be even more suitable for subjects who are less experienced in C programming.

5.2. LLTM analysis

5.2.1. Component analysis

Based on our theoretically established, pre-specified weight matrix of item components, the LLTM estimation yielded the data shown in Table 3. These parameters indicate the amount of (latent) difficulty the respective basic component introduces to an item. Low/negative values for these parameters indicate that the respective component introduces little difficulty (i.e. easy operations), while high/positive values indicate more difficult operations. For example, consider the atom shown in Figure 1 (a). In this item, the components include condition equals zero, NOT operator and while-loop.⁵ Therefore, our predefined weight matrix contains $w_{\mathit{ij}}=1$ only for these three operations, but $w_{\mathit{ij}}=0$ for all other operations. Substituting into (2), we multiply each basic component’s difficulty with its respective weight $w_{\mathit{ij}}$ (0 or 1) and summarize. As all the components not present in the atom become zero due to their weight, the atom’s difficulty is only the sum of the present components’ estimates ($0.768,-0.294,+1.379$). Thus, this code snippet would have an item difficulty of $1.853$. We can see that the components macro ($\mathit{\eta }=5.224$) and octal numerals ($\mathit{\eta }=5.051$) introduce the highest level of difficulty to the items. Conversely, the operations zero based numbering ($\mathit{\eta }=-1.187$) and comma separator ($\mathit{\eta }=-0.979$), and the concept of logical expression ($\mathit{\eta }=-0.834$) added the least amount of difficulty.

Table 3. Estimates of $\mathit{\eta }$ parameters (basic components) of our LLTM, their standard errors ($\mathit{S}.\mathit{E}.$), the respective 99% confidence interval, as well as the number of atoms containing the respective component ($n_{\mathit{atoms}}$).

	Estimate	$\mathit{S}.\mathit{E}.$	$99\mathrm{\%}$ CI	$n_{\mathit{atoms}}$
macro	$5.224$	$0.432$	$[4.11,6.34]$	$1$
octal numerals	$5.051$	$0.335$	$[4.19,5.91]$	$1$
comma operator	$4.278$	$0.337$	$[3.41,5.15]$	$1$
variable type	$2.250$	$0.206$	$[1.72,2.78]$	$11$
semicolon terminator	$2.096$	$0.142$	$[1.73,2.46]$	$11$
equal operator	$1.910$	$0.148$	$[1.53,2.29]$	$15$
reversed subscripts	$1.678$	$0.419$	$[0.60,2.76]$	$1$
pre in-/decrement	$1.648$	$0.146$	$[1.27,2.03]$	$11$
while-loop	$1.379$	$0.279$	$[0.66,2.01]$	$8$
modulo operator	$1.359$	$0.522$	$[0.02,2.70]$	$1$
OR operator	$1.278$	$0.199$	$[0.77,1.79]$	$6$
condition equals zero	$0.768$	$0.169$	$[0.33,1.20]$	$19$
AND operator	$0.695$	$0.198$	$[0.18,1.21]$	$3$
ternary operator	$0.618$	$0.333$	$[-0.24,1.48]$	$2$
type conversion	$0.608$	$0.239$	$[-0.01,1.22]$	$3$
post in-/decrement	$0.420$	$0.173$	$[-0.03,0.87]$	$15$
comparison operator	$0.406$	$0.247$	$[-0.23,1.04]$	$14$
for-loop	$0.023$	$0.345$	$[-0.87,0.91]$	$2$
maths	$0.008$	$0.347$	$[-0.88,0.90]$	$3$
if-else statement	$-0.049$	$0.176$	$[-0.50,0.40]$	$30$
hexadecimal numerals	$-0.147$	$0.323$	$[-0.98,0.69]$	$1$
NOT operator	$-0.294$	$0.295$	$[-1.05,0.47]$	$5$
preprocessor	$-0.505$	$0.293$	$[-1.26,0.25]$	$3$
logical expression	$-0.834$	$0.250$	$[-1.48,-0.19]$	$29$
comma separator	$-0.979$	$0.199$	$[-1.49,-0.47]$	$8$
zero based numbering	$-1.187$	$0.387$	$[-2.18,-0.19]$	$2$

5.2.2. Comparing RM with LLTM

Our last task was to compare the validity of our DM model with the LLTM model. Using an LRT, we found a significant difference between the two models with ${\chi }^2(44)=281.71$, $\mathit{p}<.001$, indicating that we cannot fully recover the RM item difficulty parameters with LLTM, as shown in Table 4. As already noted in Section 2.3 and through the work of De Boeck and Wilson (2004) and Fischer (1995b), this result is not surprising. Therefore, we pursued other means to further assess the usefulness and appropriateness of our LLTM.

Table 4. Model fit of the estimated RM and LLTM as well as results of a ${\chi }^2$ difference test for model comparison. $\mathit{LL}=$ logLikelihood; $n_{\mathit{par}}=$ number of model parameters; $\mathit{LR}({\chi }^2)=$ likelihood ratio, also denoting a ${\chi }^2$ value; $\mathit{df}=$ the difference in degrees of freedom between the models.

Model	$\mathit{LL}$	$n_{\mathit{par}}$	$\mathit{LR}({\chi }^2)$	$\mathit{df}$	$\mathit{p}$-value
RM	−1861.3	70
LLTM	−2143.0	26	281.71	44	$<.001$

An investigation of the correlation between RM item difficulty parameters and the retrieved item parameters in the LLTM revealed a coefficient of $\mathit{r}=.857$, $\mathit{p}<.001$ between the two models, as shown in Figure 4. This means that $r^2=.7338=73.38\mathrm{\%}$ of variance in our RM item parameters can be explained by the LLTM’s basic parameters $\mathit{\eta }$ and our weight matrix. Thus, by only considering the different operations present in each item, we can explain a great amount of item difficulty. The relationship of subject ability parameters estimated by the two models yields a correlation of $\mathit{r}=.998$, $\mathit{p}<.001$. Hence, the subject ability parameters of the RM and the LLTM almost perfectly correspond with each other. This means we can obtain virtually the same ability estimates with both models.

Figure 4. Estimated RM item difficulty parameters plotted against the (mean-centered) item difficulty parameters reconstructed by LLTM’s estimated basic parameters. The 45° line indicates the hypothetical line the dots would be at if parameters were perfectly equal.

To validate our weight matrix, we used an approach proposed by Baghaei and Hohensinn (2017), who suggested a validation method that follows the logic of parallel analysis (Horn, 1965). To obtain an estimate of the expected lower bound of correlation between LLTM item parameters and RM item parameters, we simulated $\mathit{S}=500$ weight matrices with the same dimensions as our matrix ($71\times 26$). For these matrices, we randomly chose each element of the matrix to be $0$ or $1$, with the probability to be $1$ equal to the overall proportion of $1$’s in our original matrix. Each simulated matrix was used to estimate an LLTM with our dataset, and for each we attained the (simulated) correlation $r_s$ between original RM parameters and simulated LLTM parameters. Comparing our originally obtained correlation of $\mathit{r}=.857$ to the distribution of the correlations of the simulated matrices yielded a significant $\mathit{z}$-score of $\mathit{z}=4.03$, $\mathit{p}<.001$. This provides evidence for the validity of our basic operational components: Our weight matrix that defines the “structure” of the operational components in the snippets fits the data better than the random weight matrices do (i.e. better than would be expected by chance). Thus, the structure we assume to underlie item difficulties is capable of capturing an underlying pattern.

Overall, we found our RM to be valid. Thus, the results do show that we can perceive the (unweighted) sum of correct items as a valid measure of programming ability. This makes the item set suitable for adaptive testing. Analysis of the LLTM revealed highly satisfactory recovery of the RM item and subject ability parameters, indicating that the mere consideration of operational code components yields valid results in terms of assessment. Although the comparison of the model fit of RM and LLTM was significantly in favor of RM, our additional LLTM analyses indicate its great practical relevance. The highly satisfactory correspondence of the models’ subject ability parameters indicate that automated test construction based on the LLTM’s operational components is viable, as LLTM and RM yield basically equal estimates of a person’s abilities.

6. Discussion

In the present study we (1) investigated the C code snippets’ measurement quality by means of analyzing their conformity with RM, and (2) conducted a first attempt to reveal the differences in the difficulty of operations in the C programming language. In the following section, we discuss the implications of the results presented in Section 5, answer the research questions in Section 3, and discuss our methodology and results in the context of other studies that have used IRT to evaluate assessment tools.

6.1. Evaluation of the code snippets’ measurement accuracy

To assess the measurement quality of the items, we deployed the RM and tested its validity. Analysis of the RM did not reveal violations of the model’s assumption and therefore the model is considered valid. This first investigation showed that the C code snippets tested seem to fulfill the requirements for accurate measurement. We also found evidence that the snippets used in the study would measure even more precisely in a sample set with lower programming abilities, as compared to our rather experienced sample set. While this evidence does not impair our results, it does indicate the value of the present item set in evaluating the comprehension levels of a less experienced target group, such as those just starting to program.

Comparing the LLTM to the RM, we found the RM fit the data significantly better than the LLTM. Although this indicates that our restricted model (LLTM) does not describe the data as well as the unrestricted model (RM), it was highly satisfactory in retrieving the original RM item parameters ($\mathit{r}=.857$, explaining $73.38\mathrm{\%}$ of variance) and our predefined weight matrix significantly outperformed random weight matrices. Additionally, we obtained virtually the same subject ability parameters comparing the RM with the LLTM ($\mathit{r}=.998$), indicating that both models yield equivalent results regarding ability parameters. This is in line with previous research showing that despite differing LLTM and RM item difficulty parameters, subject ability parameter estimates are largely unaffected by this difference (Alexandrowicz, 2011). As the constraints imposed by LLTM are rather strict (Fischer, 1995b) and the LRT is known to be highly sensitive to the differences in model fit, researchers suggest focusing on the theoretical and practical relevance of the results instead of overrating its statistical significance (Alexandrowicz, 2011; Fischer & Formann, 1982).

Based on our initial results from both the RM and LLTM models, it appears the C code snippets we tested can be relied on to generate accurate measurements of student performance. This answers our first research question in Section 3. With some improvement, these items can be used for adaptive testing and automated test construction (Section 7).

6.2. Difficulty levels of C-Code operations

To estimate the difficulty of various C operations and answer the second research question in Section 3, we used the LLTM model, which exhibited differences in the amount of difficulty each operational component contributed to the snippets. We found that macros, octal numerals, and the comma operator are the most difficult components in our item set. Framed practically, this means that items containing these components will be much harder to solve than items containing components like logical expressions or post in-/decrement operations. While they did not investigate octal numerals separately, Gopstein et al. (2018) found macros and comma operators to occur less frequently in real-life software projects compared to other operations (Gopstein et al., 2018). So, it is possible that subjects may simply be less familiar with these operations in comparison to the others identified as atoms. Or perhaps the opposite is true, and macros and comma operators are used less frequently in practice because these operations are hard to grasp. Lack of familiarity might also explain why we found pre in-/decrement operations to be more difficult to subjects than post in-/decrement operations, as the latter are more prevalent in practice (Gopstein et al., 2018).

The relationship between experience and program comprehension has been rather extensively addressed over the years, and suggests some insights for future exploration. McKeithen et al. (1981) observed that experienced programmers are able to organize the keywords of a language by the similarity of their functions, e.g. for and while, which makes these properties easier to recall and apply. Shaft and Vessey (1995) emphasized that experienced programmers have an edge over novices because they better understand the problem to which any given program is to be applied. However, Soloway and Ehrlich (1984) caution that even these veteran programmers can lose their advantage if the coding conventions on which they rely are violated in a piece of code.

Experienced programmers are also more likely to be aware of code constructs that over time have been avoided by the programming community (Roehm et al., 2012), most notably goto statements (Marshall & Webber, 2000), and magic numbers (Kernighan & Pike, 1999). Through experience, programmers have also learned to avoid certain patterns that have been dubbed “code smells”, or, as Fowler defined them, “structures in the code that suggest […] the possibility of refactoring” (Fowler, 2018). Since there was only a minimal difference in the experience levels of the two groups in our study, an extensive survey of participants’ knowledge of C was not gathered. In future work, we will take these factors into account. Controlling for students’ prior knowledge and their familiarity with these concepts in future studies can help to shed light on this open question.

When it comes to logical operations, we found that the component logical expression, which refers to understanding the general concept of logical operations, was quite easy for our sample. However, the actual logical operators themselves (and, or, greater/less than, equal, not equal) were more difficult for the subjects to solve. One explanation could be that the evaluation of logical operations is error prone. For example, one could confuse the greater than ($>$) and greater than or equal to ($\ge$) operators. In the latter case the statement also evaluates to true if both numbers to be compared are equal. Confusing these operators could result in an “off-by-one-error” (Rigby et al., 2020). In that sense, the concept of logical operations might be easy to grasp, but the accurate execution of a statement takes a lot more effort. Another interesting finding is that indexing itself, including the fact that C uses zero based numbering, seems to be quite easy for subjects, but reversed subscripts are much more of a challenge. One reason for this difference might be that plain indexing is taught early in C education, whereas reversed subscripts are highly uncommon (Gopstein et al., 2018).

Lastly, though the code snippets investigated in this study are limited to C, most of the operations we analyzed, such as loops, if-else statements, or logical expressions, are also common in other programming languages. Analyzing multiple other programming languages in a manner similar to what we presented here might help to identify and address operations that are difficult across the board. On the other hand, if operations in other languages varied in perceived difficulty, this might yield additional insights into which syntax is the most intuitive and least error prone for a specific operation (e.g. the syntax of a for loop differs between Python and C). Thus, other research efforts focused on programming syntax comprehension (for example, Brun et al., 2023; Stefik & Siebert, 2013) could benefit from applying the approach we used in this study.

6.3. Integrating in context: IRT in Computer Science education research

Several researchers in the computer education research community have applied IRT to evaluate the measurement properties and the quality of assessment tools. For example, Bergersen et al. (2014), Berges and Hubwieser (2015) and Davidson et al. (2021) evaluated psychometric properties of code writing exercises (programming performance, source code complexity, and a mix of code-tracing and writing tasks, respectively). Our study, however, focused only on code comprehension, so we examined other strategies specifically used to evaluate assessments for this property. These included FCS1 (Tew & Guzdial, 2011) and SCS1 (Parker et al., 2016), which were closely examined using IRT both at the time of publication and afterward (Duran et al., 2019; Xie et al., 2019). However, all of these analyses used either the two- or the three-parameter logistic model (2PL or 3PL; Birnbaum, 1968). These IRT models contain additional parameters (one “guessing parameter” per item, capturing the chance the subject guessed, and one “discrimination parameter” per item, capturing the capability of items to differentiate among participants). These options make this model less restrictive than the Rasch Model.

While they can provide great benefits and additional insight, these models also come with limitations. An individual’s total sum score is not a sufficient statistic for estimating a person’s ability due to the added guessing and discrimination parameters incorporated in these models. Furthermore, as the weighting parameters remove specific objectivity from the measure, Andersen’s Likelihood Ratio Test cannot be performed (Andersen, 1973; Glas & Verhelst, 1995). A re-validation of SCS1, however, used the Rasch Model to test the tool and found similar results to the original study (Bockmon et al., 2019). One notable difference between the analyses of FCS1/SCS1 and our study is the item difficulties: While all of the studies mentioned here (Duran et al., 2019; Parker et al., 2016; Tew & Guzdial, 2011; Xie et al., 2019) identified the instrument items as very difficult for students, the items in our study were rather easy for our subjects. This difference could possibly be attributed to the pseudocode/multi-lingual nature of FCS1 and SCS1 as compared to C language used for the item set we investigated. Finally, our study employed the additional LLTM to reveal differences in C operations’ difficulties. This model allows researchers to investigate snippets beyond their overall item difficulty and to learn about the varying difficulties of its distinct components.

7. Limitations and future directions

Although we are confident of the overall validity of our results, there are some limiting factors that need to be mentioned. First, the item set used was developed for a different purpose, i.e. to investigate the degree of confusion introduced by C code snippets (Gopstein et al., 2017). Therefore, some of the items contained unusual configurations of operations. Indeed, the original authors had chosen many of these snippets for their complex and ornate constructions. As such, the results may be less concrete than they would have been if we were working with more commonplace items. Also, there were some components in our analysis that only occurred in one single item, leading to larger standard errors and less reliable estimation. Another item feature that ran somewhat counter to our purposes is that some atoms, mostly those containing if-else statements, were designed to elicit a true or false answer. These types of items increase the chance of a subject successfully guessing the answer, so a correct answer by a subject does not necessarily reflect true ability. As this is a potential threat to the validity of our model, existing items that elicit only a true or false response should be reformulated. Logical expressions in items (e.g. printf(x < 5) that can only be true/false) should be replaced by an open answer format (e.g. printf(x) could take any value).

Second, our data contains missing values as noted above, because the original study did not present every item to all participants due to concerns about fatigue. Thus, we were not able to perform all possible RM model tests. Although the RM can be estimated regardless of missing values, and LRTs can be conducted, other tests such as the Martin-Löf test for different groups of items (Christensen et al., 2002) or the variety of non-parametric tests (Ponocny, 2001) currently cannot be conducted with established tools if values are missing (Mair et al., 2021). Even in light of the premises on which our analysis was based, we can still obtain valuable insights from our study.

Looking toward future iterations of our study, we recommend making several improvements to the test items as follows: 1) Generate new items that contain components currently underrepresented in the item set, such as macro operators, octal and hexadecimal numerals, or comma operators. This will enable us to more accurately assess difficulties caused by these components; 2) Adapt items to more systematically include a balanced number of components, i.e. items with one single component, as well as ones with two, three, and more in different combinations; 3) Ensure the items contain no missing values to enable the use of different model tests of the RM; and 4) Exclude or reformulate items that require only true or false answers.

If these aspects are taken into account, the development of a C assessment tool could be a highly promising goal for future research. Such a tool could accelerate mastery of C by offering reliable feedback to learners about their progress. Further, if the RM and LLTM hold for a larger and more complete dataset, these results can be used in adaptive testing applications where the difficulty of test items is tailored to the individual test taker’s ability. Testing tools designed in this manner can increase measurement accuracy while also increasing motivation for test takers (van der Linden & Glas, 2010), reducing exhaustion, boredom and frustration.

8. Conclusion

The analyzed item set of C code snippets originally created for the Atoms of Confusion project seems to be suitable as an assessment for C programming ability, especially for novice learners. Our tests showed the items measure programming skills precisely, and that the sum of solved items is a valid measure of C programming skills. We found evidence that these items can be applied in an adaptive testing setting, thus preventing test takers from becoming demotivated or exhausted. We identified macros, octal numerals, and comma operators as the C components that introduced the greatest amount of difficulty to items, whereas zero-based numbering, comma separators, and the concept of logical expressions added little difficulty. Additionally, these results could serve as a basis for automated test construction.

Acknowledgement

This work is supported in part by the University of Colorado Faculty Revitalization Fellowship Program and by the National Science Foundation under Award No. 1444827.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. In this paper, we use the terms “item” and “test questions” interchangeably, as the former term is commonly used in psychological measurement.

2. Although the code itself contains post in-/decrement operators, subjects do not inevitably need to master these operations to correctly answer the item, as the while loop is never entered in this example.

3. We want to clarify that we decomposed the items prior to statistical analysis, but after data sampling was complete. Thus, participants had no knowledge of the analysis criteria.

4. https://anonymous.4open.science/r/item-component-weight-matrix-ED75https://anonymous.4open.science/r/item-component-weight-matrix-ED75

5. As already noted in Section 2.4, although the snippet actually contains post in-/decrement operators, these do not necessarily need to be mastered in order to get the item correct because the while loop is not entered in this example.