Applying the significant-digit law to simplify grading of chemical engineering students design projects

ABSTRACT

The grading of chemical engineering capstone design project student reports represents a considerable burden on educators, due to the detailed calculations undertaken by students. The reports represent the accumulation of calculations into material and energy balances, detailed process and mechanical designs, process control as well as economic analysis. To facilitate these calculations, most students utilise computational programs and detailed spreadsheets. Hence, the verification of the resulting mathematics behind the generated data requires significant time commitment. To simplify this verification for grading, the significant-digit law was applied here to chemical engineering student design project reports. The use of this law enables large-scale data sets to be quickly analysed to establish if the data has been manipulated or errors exist. The significant-digit law was successfully shown to apply to chemical engineering students’ design project reports, with four abnormal student reports identified that deviated from the law due to fraud/errors in their design calculations. Importantly, applying the significant-digit law enabled a rapid approach to verifying the mathematics within the design project reports. This reduces the time burden on educators, enabling them to focus on ensuring the correct design equations and procedures have been applied.

KEYWORDS:

• chemical engineering design
• grading
• significant digit law
• calculations

1. Introduction

The capstone design project is a significant task in the final year of a chemical engineering degree, which involves the development of an integrate chemical process that consists of multiple major process units as well as a multitude of minor units (Kentish and Shallcross 2006). To accomplish their design project, students work in groups but undertake design and associated calculations individually on separate unit operations, with the results shared amongst team members (Scholes 2021). The assessment tasks of the capstone subject are critical to the overall success, because the large scope of the project is an indicator of the wider requirements for the chemical engineering degree (Scholes 2021). Importantly, the assessment tasks need to be ill-defined to ensure the students demonstrate creativity and judgement as part of their engineering training (Kentish and Shallcross 2006), which then makes the assessment tasks grading rubrics correspondingly ambiguous for the examiners (Bishop, Nespoli, and Parker 2012). This places a lot of flexibility on the examiners in what they emphasise as important within the submitted assessment. One core component of any assessment tasks is the need to undertake detailed calculations, utilising design equations and engineering principles to answer complex engineering questions. In chemical engineering, students need to undertake a significant number of calculations to achieve a satisfactory outcome. This includes mass, material and energy balances around all unit operations, detail process design of all major unit operations as well as minor equipment, mechanical design of major process units and ancillary minor equipment associated with their chosen major process, process control as well as high level economic analysis of their entire process (Pekdemir, Murray, and Deighton 2006; Scholes 2021). The resulting calculations that arise from this work can be hundreds of pages of hand calculation, spreadsheet tables and computer coding, all of which the student submits for evaluation and grading.

The quantity of calculations provided by the students places a significant time burden on educators to grade. Firstly, they must identify that the students are using the appropriate and correct design equations and procedures, with the resulting calculations followed to ensure the students have done the mathematics correctly. This becomes incredibly challenging when grading spreadsheet tables and computer coding, because of the number of calculations behind what is presented can be overwhelming. The grading of one student design project report can take several days to go through to ensure their submitted calculations are correct. For a large class, the subsequent grading can represent several weeks of work by the educators. At the University of Melbourne, this grading burden is spread amongst all the academics in the chemical engineering department, but still represents a significant time demand on individual academics. This burden increases the chance of error in the marking or mistakes by the examiners, given the large documents they must review in detail. Therefore, there is need for additional tools to assist examiners in reviewing student design projects, in particular support is required to analyse the design calculations, as these represent a large time commitment for examiners. As such an alternative strategy is proposed to analyse chemical engineering student’s design project submission to verify that the submitted calculations and resulting numbers being presented are correct. One strategy that demonstrates potential for speeding up the evaluation of students’ design project reports is significant-digit law analysis.

The significant-digit law, also known as Benford’s law or the phenomenon of significant digits, is the finding that the first numeral of numbers found in series of related records do not display a uniform distribution (Benford 1938). Rather, the numeral ‘1’ is the most frequent first-digit in a series of numbers, followed by ‘2’, then ‘3’ and so on in a successively decreasing frequency down to ‘9’. Importantly, this law holds for calculations of number sets, resulting in the numeral ‘1’ being the most frequent first-digit in any number outcome, ‘2’ being the second most frequent and so on. A corresponding relationship exists for the frequency of the numerals in the second digit of a number and so on for the third digit, etc. This law has been known since 1881, but only proven in 1995, and appears in a wide variety of settings, from utility bills, distances between towns, stock prices and accounting (Drake and Nigrini 2000; Newcomb 1881; Nigrini and Mittermaier 1997). The accounting example is of relevance to chemical engineering design project, as the significant-digit law is used in forensic accounting to identify fraud and the falsification of data (Durtschi, Hillison, and Pacini 2004). Deviation in the significant-digit law for large-scale data sets are generally the result of manipulation and/or fabrication of data and is therefore cause for further investigation. This approach has shown promise at identifying misrepresentation in macroeconomics (Cerioli et al. 2018; Shi, Ausloos, and Zhu 2018; Todter 2019), within business and individual financial reporting (Long et al. 2004), as well as in government programmes (Azevedo et al. 2021). Similar analysis has been applied to scientific data to detect fraud (Horton, Kumar, and Wood 2020). In this application, the fraud being detected is associated with large data sets that are published by the same author(s) across multiple peer-reviewed papers, rather than one discrete publication, to enable the numeral frequency trend to be established (Diekmann 2007). The law does not apply to data sets that are predisposed with a limited set of digits, such as telephone numbers, nor does it apply to designation numbers, such as instrument codes. Hence, the significant-digit law has application to many large datasets and those that undergo calculations, and deviations from the law provide reasons for further investigation. This will be especially true given the rapid increase of digitisation in all aspects of life, including engineering, and the need to ensure quality within the established data sets (Geyer and Williamson 2007; Kossler, Lenz, and Wang 2021). The same approach is applied here to chemical engineering student design project report calculations, which are analysed by the significant-digit law to identify cases where the student(s) have not undertaken the necessary design calculations and instead presented false or erroneous data. Currently, there are no tools available to educators to analyse large-scale calculations independently for errors or plagiarism, as there exists for text-based assessments (Foltynek et al. 2020; Naik, Landge, and Mahender 2015). Therefore, the aim of this research was therefore to establish the suitability of significant-digit law analysis for chemical engineering design projects, and chemical engineering calculation-based assessment tasks in general.

2. Theory

The first-digit law is satisfied for a set of numbers of base ‘b’, if the leading digit (d) occurs with a probability of (Varian 1972):

(1) $\mathrm{P}\left(\mathrm{d}\right)=\mathrm{l}\mathrm{o}{\mathrm{g}}_{\mathrm{b}}\left(1+\frac{1}{\mathrm{d}}\right)$(1)

The proof of this law is based on the number set being the combination of distributions (Hill 1995). For a base of 10 (e.g. 1, 2, … ., 9) the first-digit law probability is:

(2) $\mathrm{P}\left(\mathrm{d}\right)=\mathrm{l}\mathrm{o}{\mathrm{g}}_{10}\left(1+\frac{1}{\mathrm{d}}\right)$(2)

The resulting frequency of base 10 numerals by the first-digit law is provided in Table 1, highlighting that the numeral ‘1’ will be first digit 30.1% of the time, the numeral ‘2’ being the first digit 17.6% and so on. The law has been extended beyond the first-digit and can be applied to any given digit in a number, with the probably of encountering a number starting with a string of digits (n) being:

(3) $\mathrm{P}\left(\mathrm{n}\right)=\mathrm{l}\mathrm{o}{\mathrm{g}}_{10}\left(1+\frac{1}{\mathrm{n}}\right)$(3)

Table 1. Probability of base 10 numerals being the first and second digit of a number, as defined by the significant-digit law.

Digit	1st	2nd
0	-	0.120
1	0.301	0.114
2	0.176	0.109
3	0.125	0.104
4	0.097	0.100
5	0.079	0.097
6	0.067	0.093
7	0.058	0.090
8	0.051	0.088
9	0.046	0.085

This enables the second-digit frequency to be established and is also provided in Table 1. For example, the numeral ‘0’ will be the second digit 12.0% of the time. Higher digit analysis is also possible, with each numeral probability approach 10% for the third-digit and fourth-digit.

3. Experimental details

Chemical engineering student design project reports and supporting documents were analysed for their first-digit and second-digit numeral frequency, to determine if the data set followed the significant-digit law. The reports were all processed by optical character recognition software (Adobe Acrobat Pro) to identify numbers in both printed and handwritten format, with the subsequent numbers correlated and the first digits and second digits recorded. This process is straightforward, as students must submit their various design project reports electronically through the university’s learning management system. As such, the reports are already digitised and in a format that is compatible with the software. The analysis included all calculations (hand, spreadsheet outputs and computer coding) as well as reporting of numbered results in the main body of the reports’ text. Number designations (such as valves) in drawings, control and operation schemes were ignored as these do not represent calculations. Similarly, the inclusion of quoted technical information from standards and references sources were also ignored, as these do not represent calculations. The removal of this information is necessary to ensure the subsequent analysis fits within the significant-digit law criteria. The removal of this information must occur to prevent the detection of false positives.

The time required to generate the numeral frequency charts for each report was less than 10 minutes. Most of this time was associated with selecting the paragraphs and pages that must be excluded from the analysis, as these contain drawings, technical schemes and quoted designation data that do not represent calculations. The follow-up analysis of calculation procedure and ensuring the correct algorithms were followed can be conducted in <2 hours per report. This contrasts with the >8 hours per report usually required by examiners to properly analyse technical calculations and associated discussions.

The data set consisted of four years of students reports on four different design projects (helium extraction from natural gas, vitamin production from natural sources, rare earth element refining and camel milk product processing). In total 356 student reports were analysed for their first-digit and second-digit frequency in calculations, representing over 75,00 pages of hand calculations, spreadsheet calculated tables and computer coding outputs.

4. Results and discussion

The aggregated first-digit outcome from the entire cohort of chemical engineering design project reports analysed is provided in Figure 1 as a histogram. Also included in the figure is the corresponding first-digit law probability for each numeral, and there is a very good agreement between the reports analysis and the law, with the numeral ‘1’ having a frequency of 31% (compared to 30.1% for the first-digit law) and the numeral ‘2’ having a frequency of 17% (compared to 17.6% for the first-digit law). There is some discrepancy between the analysis and first-digit law at higher numerals, with ‘9’ having a frequency of 5.8% (compared to 4.6% for the first-digit law). To determine if the deviation was significant the z test was applied:

(4) $\mathrm{Z}=\frac{\left|\mathrm{A}\mathrm{P}-\mathrm{E}\mathrm{P}\right|}{\sqrt{\frac{\mathrm{E}\mathrm{P}\left(1-\mathrm{E}\mathrm{P}\right)}{\mathrm{n}}}}$(4)

Figure 1. Aggregated first-digit numeral frequency from chemical engineering design project reports as a histogram, as well as the corresponding probability (points) based on the first-digit law.

where AP is the actual proportion, EP the expected proportion from the first-digit law and n the data set. The corresponding z score is 1.068, which indicates that the deviation observed is not significant.

The aggregated second-digit outcome from the entire cohort of design project reports analysed is provided in Figure 2 as a histogram, also included are the second-digit law probability for each numeral. The trend of the lower numerals having a higher frequency as the second digit compared to the higher numerals is present, but there is more deviation between the report analysis and the corresponding law probability than observed in Figure 1. For example, the numeral ‘0’ has a frequency of 14% (compared to 12% for the second-digit law) and the numeral ‘1’ has a frequency of 10% (compared to 11.4% for the second-digit law). This variability between data set and the second-digit law probability is common, and therefore only large variations in the second digit frequency are used to identify fraud. Hence, chemical engineering design project calculations are hereafter assumed to follow the significant-digit law. Working from this assumption enables individual student reports to be analysed to determine that students have undertaken the appropriate calculations throughout their design project.

Figure 2. Aggregated second-digit numeral frequency from chemical engineering design project reports as a histogram, as well as the corresponding probability (points) based on the second-digit law.

During the analysis of the all the student design project reports there were four abnormal reports in terms of the first- and second-digit frequencies, compared to the significant-digit law, that had not previously been identified by examiners as having questionable content. All other reports were classified as observing the significant digit laws. All 356 reports analysed had previously received passing grades from examiners. Hence, the significant-digit law analysis indicated that 1.1% of the reports had abnormal distributions, which under closer examination had significant calculation errors what should have received a failure grade. No report was classified as having abnormality significant digit law behaviour that under subsequent examination had no underlying problem with the design calculations presented. Hence, the significant-digit law analysis had a false positive rate of zero. The false negative rate is expected to be very low, as all the reports that demonstrated normal significant-digit law behaviour had previously been given pass grades by examiners. However, there remains the possibility that some misconduct undertaken by students will produce a Benford’s law distribution.

Two of the four abnormal reports consisted of numeral distributions that had similar frequencies across the numerals in the first- and second-digit places, which are provided in Figures 3 (Report A) and 4 (Report B). Both data sets clearly deviate from the expectation of the significant-digit law and the behaviour of their fellow students’ reports. There is a slight increased frequency for the numeral ‘1’ and ‘2’ for both data sets for the first-digit, but not to the extent expected from the significant-digit law, and there is no difference for the second-digit. On deeper analysis of both reports, it was established that majority of the spreadsheet tables provided in both reports consisted of essentially random numbers. This covered material and energy balances calculations and detailed calculations on the internal process conditions of their respective major unit operations. That is, the author could not determine the mathematical connection between numbers on the same rows and across columns, which is what would be expect in spreadsheet tables. The obvious conclusion is that the students fabricate the design numbers used in their reports and presented the meaningless tables to obtain a pass grade, relying on the hope that the marker would not examine the presented data in depth. Report B (Figure 4) distributions are muted forms of the significant-digit law, which indicates that some sections with the student’s report have data that has been calculated (in this case the material balances), while other sections have been fabricated. Hence, the numerical frequency profile may provide a guide to the degree of fabrication and error within the design calculations. However, a larger data set of student reports showing such behaviour will need to be analysed to understand this phenomenon in more detail.

Figure 3. Numeral frequencies for report A: a) first digit and b) second digit arising from a chemical engineering student’s report that consisted of unconnected numbers, provided also is the expected probability based on the significant-digit law.

Figure 4. Numeral frequencies for report B: a) first digit and b) second digit arising from a chemical engineering student’s report that also consisted of unconnected numbers, provided also is the expected probability based on the significant-digit law.

The third report (Report C) consisted of a first-digit and second-digit distributions that strongly favoured the higher numerals ‘7, 8 and 9’ much more than would be expected from the significant-digit law, as shown in Figure 5. Analysis of the corresponding report determined that the student had at two different stages of the design completed altered the data set they were using for their calculations. Specially, the energy balances presented were based on different material balances than those that were established in the mass balance section of the report. Similarly, their economic costings and evaluation were based on a completely different sized process with additional ancillary equipment that were not present in their process description, process design and drawings. Hence, the calculations being presented where essentially a confusion between sections that did not present a coherent story, representing poor design ability by the student. Importantly, the significant-digit law approximation clearly breaks down for this scenario and may therefore be used to evaluate the quality of the design calculations.

Figure 5. Numeral frequencies for report C: a) first digit and b) second digit arising from a chemical engineering student’s report with a disconnect in the data sets being used for calculations, provided also is the expected probability based on the significant-digit law.

The fourth report (Report D) consisted of a first-digit distribution that corresponded well with significant-digit law, but the second-digit distribution heavily emphasised the numerals ‘0’ and ‘5’, as shown in Figure 6. Analysis of the student report found the submitted code for their detailed process design had an error in their calculation, which multiplied key components in the process by a factor of 5. This led to their design being significantly larger than necessary and the corresponding numbers being produced were multiples of 5, which appeared strongly in the second digit. The actual error could only have been determined by examining the submitted coding in detail, but the corresponding oversized design was obvious to an experienced marker.

Figure 6. Numeral frequencies for report D: a) first digit and b) second digit arising from a chemical engineering student’s report with an error in their design code, provided also is the expected probability based on the significant-digit law.

For all four examples, deviation from the significant-digit law indicated an underlying issue with the chemical engineering student reports’ calculations. Deviation from the law does not in itself represent fraud or a mistake, as correlation does not equal causation. Rather deviation provides guidance to markers to probe more deeper into the student(s) calculations to verify the design produced, as the four examples demonstrate. Equally important, the significant-digit law only provides insight into the calculations, the law has no guidance on the right type of calculations are being undertaken. Hence, it is up to the examiner to ensure that students are using the right design equations and procedures for their process.

There is potential to extend the significant-digit law to detect large-scale calculation discrepancies in other chemical engineering assessment tasks. The limitation of the law is the need for a large set of data to establish the numeral frequency distribution. The literature’s general lower limit is a sample size of at least 50 to 100 numerical values required, with a sample size of over 500 more appropriate for proper analysis (Varian 1972). This prevents significant-digit law analysis for standard chemical engineering practical reports and hand-calculation based assessment tasks (Flach 1999; Scholes and Hu 2021), but does highlight the potential for the analysis in computational based assessments. This aligns well with the pedagogy changes currently underway in many chemical engineering courses, with a focus on computer program and digitisation analysis (Chiu 2021; de las Heras et al. 2021; Feise and Schaer 2021; Wong and Barford 2010).

5. Conclusions

Engineering student capstone projects grading represents a significant time burden on educators, given the large number of calculations and procedures that need to be reviewed and assessed. Tools are needed that can simply and speed up the grading time of these assessments’ items, as well as be an additional check for errors and misconduct undertaken by students. The significant-digit law is a strong analysis tool of large data sets that can be used to verify calculations outcomes and the misrepresentation of data. In this study, the significant-digit law was applied to chemical engineering students’ design project reports to simplify the grading of the mathematics behind their process designs. It was established that design project reports do follow the significant-digit law very closely and that of the 356 student reports analysed four deviations from the law were identified. Subsequent analysis of those deviations identified two cases of fraud by students to justify their final design, one case of a student being inconsistent across their design basis and the final case of a coding error that influenced the final design. To conclude, chemical engineering educators can use significant-digit law analysis to better grade the calculation outcomes of their students, but the educators still need to determine for themselves that the students have used the appropriate design equations and procedures.

Disclosure statement

No potential conflict of interest was reported by the author.

Additional information

Notes on contributors

Colin A. Scholes

Colin A. Scholes is an Associate Professor in the Department of Chemical Engineering at the University of Melbourne. He is an expert in clean energy processing and membrane science. He is also a passionate engineering educator to people from disadvantage backgrounds.