“C”ing the light – assessing code comprehension in novice programmers using C code patterns

Figures & data

Figure 1. Example for a snippet pair that is supposed to be non-confusing (a) or confusing (b), depending on whether there is an assignment of a constant inside the while condition.

Table 1. The 26 basic operational components we identified in the items and short descriptions of the challenges that arise from each of them. These components represent the columns of our weight matrix.

Component	Description of the Challenging Aspects of a Component
if-else statement	To solve this operation, one has to be familiar with the logic of if-else.
condition equals zero	There is a test expression in an if statement or while loop, which evaluates a single value to false if the value is 0 and to true in any other case.
logical expression	The atom contains a logical expression of any kind – one has to know that these evaluate to true or false and how to further treat these statements.
comma separator	The snippet contains a comma that serves as a separator in the item. It has to be recognised as a delimiter and must not be confused with a comma operator.
comma operator	The code snippet contains a comma serving as a binary operator. To solve this item, one has to know that the first element will be evaluated and discarded, and then the second element will be evaluated and returned. This operator must not be confused with a comma separator.
AND operator	Subjects need to know that the overall statement evaluates to true only if both operands before and after a AND operator (&&) evaluate to true.
OR operator	In order to solve this snippet, one has to know that a logical statement with an OR operator ($\parallel$) evaluates to true, if at least operand evaluates to true.
comparison operator	Within a logical expression in the atom, there is a numerical comparison ($\ge ,>,\le ,<$). Subjects have to be familiar with these symbols and need to know their differences.
NOT operator	To solve this item, one needs to know the NOT operator ($!=$) and apply it correctly.
equal operator	One has to apply an equal operator ($==$) within a logical expression correctly and must not confuse it with an assignment operator ($=$).
modulo operator	Solving the atom requires subjects to know the modulo operator and to use it correctly.
ternary operator	Solving the item requires the subject to be familiar with the ternary operator ($?:$).
maths	To solve the item one is required to apply some mathematical operations going beyond simple addition or multiplication (e.g. operator precedence).
while-loop	One has to know how a while loop works and be able to apply the logic.
for-loop	One has to know how a for loop works and be able to apply the logic.
post in-/decrement	The post increment and decrement operator has to be known, applied correctly, and it must not be confused with a pre increment/decrement operator.
pre in-/decrement	Subjects have to know pre in- and decrement operators and must not confuse them with post increment/decrement operators.
macro	The item contains a macro, which subjects need to understand and correctly apply to the code.
preprocessor	There is a preprocessor statement within the code, which has to be identified as such and subjects have to know that these are evaluated at first.
hexadecimal numerals	The code contains numbers in hexadecimal numeral system, which have to recognised as such and have to be translated to decimal.
octal numerals	The code contains octal numerals, which have to recognised as such and have to be translated to decimal.
type conversion	Some variables in the code will be (implicitly) converted to a different variable type by the compiler, which subjects need to anticipate to master the item.
variable type	The code snippet contains a variable type other than integer. One has to be familiar with other variable types, recognise them as such and treat them correspondingly.
semicolon terminator	Subjects need to be aware that the semicolon terminates the previous statement and that after the semicolon a separate statement begins.
zero based numbering	The atom contains indexing and subjects have to be aware of that the sequence starts with zero.
reversed subscripts	Subjects need to know that the subscript operator is commutative in order to solve these items.

Table 2. Results of the conducted LRTs for the split criteria of median of person score, population, and median of experience. The columns refer to the ${\chi }^2$ test statistic (LR), the degrees of freedom $\mathit{df}$, the critical ${\chi }^2$ value for a corrected significance level ${\alpha }^{\ast }=0.01/3=.0033$ of the respective test distribution, as well as the associated $\mathit{p}$-value.

Split criterion	${\chi }^2$	$\mathit{df}$	${\chi }_{\mathrm{crit}({\alpha }^{\ast })}^2$	$\mathit{p}$-value
Person score (median, high/low)	82.41	51	82.63	0.0035
Sample (US, Chinese)	73.28	62	96.45	0.1547
Experience (median, high/low)	70.05	61	95.20	0.1999

Figure 2. Graphical model checks for the three LRTs. Each plot depicts item difficulty parameter estimates (“betas”) for the subgroups of each LRT (person score, population, and experience). Blue dots represent item difficulty estimates, grey ellipses represent the two-dimensional confidence intervals (${\alpha }^{\ast }=.0033$) associated with the item estimates.

Figure 3. Person-item map of our RM. The upper histogram reflects the distribution of our subjects’ ability parameters. The lower part of the figure depicts the item difficulty for every item in our dataset. The $\mathit{k}=71$ items are marked by horizontal ticks on the left and are ordered by difficulty (from easy items on top to difficult items on the bottom).

Table 3. Estimates of $\mathit{\eta }$ parameters (basic components) of our LLTM, their standard errors ($\mathit{S}.\mathit{E}.$), the respective 99% confidence interval, as well as the number of atoms containing the respective component ($n_{\mathit{atoms}}$).

	Estimate	$\mathit{S}.\mathit{E}.$	$99\mathrm{\%}$ CI	$n_{\mathit{atoms}}$
macro	$5.224$	$0.432$	$[4.11,6.34]$	$1$
octal numerals	$5.051$	$0.335$	$[4.19,5.91]$	$1$
comma operator	$4.278$	$0.337$	$[3.41,5.15]$	$1$
variable type	$2.250$	$0.206$	$[1.72,2.78]$	$11$
semicolon terminator	$2.096$	$0.142$	$[1.73,2.46]$	$11$
equal operator	$1.910$	$0.148$	$[1.53,2.29]$	$15$
reversed subscripts	$1.678$	$0.419$	$[0.60,2.76]$	$1$
pre in-/decrement	$1.648$	$0.146$	$[1.27,2.03]$	$11$
while-loop	$1.379$	$0.279$	$[0.66,2.01]$	$8$
modulo operator	$1.359$	$0.522$	$[0.02,2.70]$	$1$
OR operator	$1.278$	$0.199$	$[0.77,1.79]$	$6$
condition equals zero	$0.768$	$0.169$	$[0.33,1.20]$	$19$
AND operator	$0.695$	$0.198$	$[0.18,1.21]$	$3$
ternary operator	$0.618$	$0.333$	$[-0.24,1.48]$	$2$
type conversion	$0.608$	$0.239$	$[-0.01,1.22]$	$3$
post in-/decrement	$0.420$	$0.173$	$[-0.03,0.87]$	$15$
comparison operator	$0.406$	$0.247$	$[-0.23,1.04]$	$14$
for-loop	$0.023$	$0.345$	$[-0.87,0.91]$	$2$
maths	$0.008$	$0.347$	$[-0.88,0.90]$	$3$
if-else statement	$-0.049$	$0.176$	$[-0.50,0.40]$	$30$
hexadecimal numerals	$-0.147$	$0.323$	$[-0.98,0.69]$	$1$
NOT operator	$-0.294$	$0.295$	$[-1.05,0.47]$	$5$
preprocessor	$-0.505$	$0.293$	$[-1.26,0.25]$	$3$
logical expression	$-0.834$	$0.250$	$[-1.48,-0.19]$	$29$
comma separator	$-0.979$	$0.199$	$[-1.49,-0.47]$	$8$
zero based numbering	$-1.187$	$0.387$	$[-2.18,-0.19]$	$2$

Table 4. Model fit of the estimated RM and LLTM as well as results of a ${\chi }^2$ difference test for model comparison. $\mathit{LL}=$ logLikelihood; $n_{\mathit{par}}=$ number of model parameters; $\mathit{LR}({\chi }^2)=$ likelihood ratio, also denoting a ${\chi }^2$ value; $\mathit{df}=$ the difference in degrees of freedom between the models.

Model	$\mathit{LL}$	$n_{\mathit{par}}$	$\mathit{LR}({\chi }^2)$	$\mathit{df}$	$\mathit{p}$-value
RM	−1861.3	70
LLTM	−2143.0	26	281.71	44	$<.001$

Figure 4. Estimated RM item difficulty parameters plotted against the (mean-centered) item difficulty parameters reconstructed by LLTM’s estimated basic parameters. The 45° line indicates the hypothetical line the dots would be at if parameters were perfectly equal.