Developing a framework to re-design writing assignment assessment for the era of Large Language Models

KEYWORDS:

• assessment design
• authentic assessment
• critical thinking
• Large Language Models (LLMs)
• formative assessment

Introduction

Since November 2022, AI tools based on generative Large Language Models (LLMs), like ChatGPT (OpenAI), Bing AI chatbot (Microsoft), Bard (Google), and LLaMa (Meta), have gained significant interest globally. Their impact on higher education has sparked debates and discussions in Dutch universities (e.g., ScienceGuide and Surf Communities). These text-based LLM tools produce human-like outputs that are hard to distinguish from human-made content. Concerns have arisen about the potential misuse of such chatbots for ghostwriting student academic assignments, leading to unreliable assessment decisions (Perkins, 2023; Rudolph et al., 2023a).

In response to these concerns, many universities have identified the impact of LLM tools on coursework assessment and established guidelines for teachers to enhance assessment quality in writing assignments. Our current project focuses on assisting teachers at a Dutch university in redesigning their assignment assessments, guided by a memo that addresses the impact of LLMs and the university working group’s established guidelines.

Since our project is solution oriented under high time pressure, this project follows a Design Science Research (DSR) approach, with two goals of developing practical solutions for real-world challenges and thoroughly testing these solutions in the field (Van Aken & Berends, 2018). Specifically, our implementation of the project aligns with the six sequential steps of DSR proposed by Peffers et al. (2007): (1) recognizing and clearly defining the problem; (2) establishing specific objectives and requirements for the proposed solution; (3) developing the solution, which may involve creating a framework, model, or other applicable entities; (4) providing a demonstration of the solution’s effectiveness; (5) evaluating the solution by comparing expected outcomes with observed results; and (6) effectively communicating the problem, the solution, and its practical value to relevant audiences.

To achieve the first goal of DSR, this brief report focuses on the third (i.e., the framework) and fourth (i.e., workshops) steps of the project. In addition to using our conventional toolkit approach for teacher professionalization, we have pursued a solution that guides teachers to analyse their course context within the program and effectively cater to various student groups. Consequently, we have developed a framework to identify design dimensions for redesigning writing assignments while considering the benefits and challenges presented by LLMs. This report aims to describe the prototype of our framework and share the lessons learned from its implementation in the course redesign workshops. Furthermore, we outline future studies to further enhance our understanding and application of these innovative approaches.

Brief literature review

To establish a framework for the design of assessment in writing assignments, we conducted a brief literature review in two phases.

In the initial step, our focus was on getting familiar with the relevant literature. We made an inventory of diverse publication types on assessment design for writing assignments, particularly in response to the influence of LLM tools. This compilation encompassed a variety of sources including blog posts, magazines, instructional guides (e.g., Gimpel et al., 2023), scholarly journal articles based on review studies (e.g., Lo, 2023; Perkins, 2023; Rudolph et al., 2023b), and university education support websites (e.g., Koenders & Prins, 2023; Walden University, 2023). We observed that a substantial number of recommendations for designing writing assignment assessments in the LLM tools era drew from literature on contract cheating and the impact of previous AI tools (e.g., Grammarly, DeepL, Atlas.ti) on assessment practices. Secondly, we identified shared references from these publications and synthesized their recommendations for design principles of writing assignment assessments.

What can we learn from these earlier studies? Several large-scale empirical studies conducted in Australian higher education institutions have examined the relationship between assessment design and students’ self-reported propensity to engage in contract cheating (e.g., Bretag et al., 2019; Ellis et al., 2020). In their book “Re-imagining University Assessment in a Digital World”, Bearman et al. (2020) have clarified the distinction between human and machine intelligence, the impact of AI on assessment design, and the design principles for incorporating cognitive offloading (the use of tools for efficient work) into students’ future professional and personal lives. Both human ghostwriters and machine offloading assistants contribute to the understanding that there are no quick fixes; however, student-centred assessment design, aimed at promoting academic integrity and learning, holds the potential to decrease students’ inclination towards fraudulent behaviour. The recurring themes in these recommendations are:

• prioritizing critical thinking over mechanical writing skills;

• implementing authentic assessment that connects to real-life experiences and current societal issues;

• incorporating cognitive offloading as part of academic skills;

• extending course-level assessment to programmatic assessment (i.e., integrating AI and digital literacy into the critical thinking curriculum and utilizing LLMs);

• balancing formative and summative assessment and emphasizing student learning; and

• the necessity for teacher professionalization in assessment literacy with LLM tools.

Though many of these suggestions align with standard course assessment design, the presence of LLM tools – akin to free ghostwriters – has exacerbated academic misconduct, more so than commercial essay mills (Eaton, 2022). LLM tools add intricacies to design, demanding teachers’ grasp of their functionalities, strengths, weaknesses, and impact on learning and integrity (Perkins, 2023; Perkins et al., 2023). Crucially, they raise questions about the validity of testing higher-order and critical thinking skills as intended in learning goals (Lo, 2023). Additionally, universities’ policies on detecting (e.g., oral follow-up checks) (Gimpel et al., 2023) and penalizing unauthorized LLM use (e.g., unacknowledged generated texts) introduce further complexity. These challenge teachers to balance instructional and examination roles, potentially overwhelming their limited resources.

Framework

To help teachers deal with the complexity, we identified six design dimensions to structure the suggestions.

Purpose – course context and learning goals (LGs)

Course context analysis is an essential step in the design of any assessment. While the integration of LLMs is widely advocated, teachers should carefully consider how the use of digital tools (i.e., cognitive offloading) aligns with program-level learning outcomes and whether it may hinder students’ acquisition of foundational knowledge and skills. Dawson (2020) suggests adopting a “reverse-scaffolding” approach to program curriculum design, where students are allowed to use LLMs only after mastering the foundation knowledge and skills, similar to how primary school pupils first learn times tables before using a calculator. Additionally, teachers must be mindful of the limitations of LLMs, such as flaws and biases, which differ significantly from using calculators or statistical programs like SPSS, where we can have a high level of confidence in their accuracy when performing assigned tasks.

Choosing an appropriate assessment type depends on what thinking skills, subject content and attitude are targeted in the course LGs (Biggs & Tang, 2007). As critical thinking is often emphasized in assessment design (e.g., Gimpel et al., 2023; Rudolph et al., 2023b), we use Paul and Elder’s universal standards of critical thinking (Paul & Elder, 2001), which are also crucial in the era of LLMs. In Table 1, we provide a summary of the significant weaknesses and limitations of LLMs (Cotton et al., 2023; Crawford et al., 2023; Farrokhnia et al., 2023) as well as their potential implications for formulating course LGs and example LGs (Kuhn, 1999). When formulating course LGs, it is important to consider the appropriate level of critical thinking based on the course’s position within the program curriculum. For instance, information literacy may be suitable for first-year bachelor’s courses, while complex problem solving may be more appropriate for senior students.

Table 1. Weaknesses and limitations of LLMs and their implications on course LGs.

Weaknesses and Limitations	May not meet the quality standards of critical thinking in terms of …	Implications on formulating course LGs
Overconfident tones Hallucinations, or mix of partially true facts	Accuracy	Enhance information literacy -Determine the truthfulness and credibility of the sources
“What most people said” claims and examples	Precision	Transfer to the context -Identify important details and specifics to the question in the context and the negative consequences that resulted without details and specifics.
Irrelevant information to the prompt	Relevance	Develop metacognitive strategies to think on track -Differentiate ideas/thoughts directly and indirectly related to the issue
Lack deep understanding of “the concepts behind the words” and “nuanced understanding” of a certain subject	Depth	Complex problem solving -Analyze the complexity of the problem and determine the most significant solution to solve it.
Biases and discrimination	Balance	Consider multiple perspectives – Articulate each point of view objectively, regardless of your personal views
No justified structure how claims and evidence are presented	Logic	Epistemological understanding and Evaluative judgement – Evaluate and weigh contrasting accounts while respecting the right to hold opinions, without assuming that all opinions are equally valid and immune to examination.

In the following dimensions, as a worked example, we use the LG on information literacy, determine the credibility of the sources (see Table 1), with three underlying steps: 1) list academic databases for credible sources, 2) categorize characteristics of credible sources based on a rationale, and 3) employ search strategies to find credible sources.

Function & focus

Function as formative assessment

In general, once a clear set of course LGs has been formulated (e.g., determine the credibility of the sources), the next step is to develop and monitor student progress towards these targeted knowledge and skills (e.g., the three underlying steps). To facilitate this process, teachers should provide clear expectations for each underlying level, enabling students to generate effective prompts (e.g., list 10 open access academic databases in the discipline of “artificial intelligence” and provide their URLs) and critically examine LLM generated texts (Mollick & Mollick, 2022).

Since LLM tools offload the task at the underlying level, formative assessment should guide students how to assess the quality of LLM outputs using critical thinking criteria (e.g., accuracy) (Crawford et al., 2023; Gimpel et al., 2023).

Function as summative assessment

The most challenging decision for teachers centres on determining whether LLMs can be utilized for summative assessment. In general, grading should primarily focus on evaluating the higher levels of LGs. In the era of LLMs, their authorized uses involve student content refinement, whereas unauthorized uses entail the direct use of generated texts without proper acknowledgements or processing (e.g., Gimpel et al., 2023; Perkins, 2023).

For instance, let’s consider the LG that aims to evaluate the credibility of sources cited in a given article. Relying solely on LLMs to determine the credibility of sources would be an unauthorized use, as it hinders the assessment of whether students have achieved the highest level of LGs. However, utilizing LLMs in a more limited and controlled manner can be considered an authorized use. For example, teachers can have students use LLMs to process three underlying steps of the LG and acknowledge this use or include the generated texts as part of deliverables.

Focus: balance both process and product

In the era of LLMs, solely grading the final product without monitoring the process poses a significant risk of fraudulent behaviour (Gimpel et al., 2023). Therefore, it is crucial to divide the final product into multiple process deliverables for formative or continuous summative assessment, considering the cognitive processing required by writing assignments, such as outline, draft, review, revision (Mills, 2023). These process deliverables should be sequenced in a logical order, enabling students to optimally utilize feedback to enhance their performance in subsequent tasks and to ensure student accountability. In order to gain a comprehensive understanding of student learning outcomes, both the process deliverables and the final product for summative assessment can be compiled into a portfolio, accompanied by students’ critical reflections on the relationships between the process and the products (Gimpel et al., 2023; Rudolph et al., 2023b).

Grading criteria

With the advent of LLMs, there has been a shift in the focus of writing, moving from a mechanical emphasis (primarily concerned with form and presentation) to a more sophisticated approach (prioritizing meaning and critical thinking) (Bishop, 2023; Mills, 2023). Consequently, grading should place greater emphasis on assessing the criteria related to the critical thinking process of the content (Gimpel et al., 2023; Rudolph et al., 2023a), such as argumentation, reasoning, and the alignment among research questions, theoretical framework, methodology, results, discussion, and conclusion. Conversely, less emphasis should be placed on mechanical aspects of writing, such as language, organization, style, and layout.

For example, when evaluating the criterion of “evaluate information and its sources critically”, the grading should focus on whether students select a variety of information sources that are appropriate to the scope and discipline of the research question. Additionally, students should be assessed based on their consideration of multiple criteria, such as relevance to the research question, currency, authority, audience, and bias or point of view (Association of American Colleges and Universities [AAC&U], 2009).

Modes

Teachers have the flexibility to incorporate formative assessments in blended modes. For instance, the task of “categorizing characteristics of credible sources based on a rationale” can be assigned as an off-campus activity, allowing students to develop their categories and rationales. Subsequently, during in-class discussions, students can share their categorizations and engage in collaborative discourse. LLM tools can serve as valuable facilitators in these small group discussions by providing real-time answers to questions and offering elaborations of key concepts, enhancing the learning experience (Gilson et al., 2023; Lo, 2023).

In terms of summative assessment, for instance, in addition to evaluating the written product that demonstrates the determination of credible resources, teachers can employ online or in-person oral follow-up inspections to verify the authenticity of student work on the attainment of targeted course LGs (Gimpel et al., 2023).

Authenticity

Authenticity in writing assignments encompasses several key aspects, including relevance to real-world and current events, consideration of audience, purpose, genre, impact, student interest, and student choice (Wargo, 2020). Such authentic writing assignments play a crucial role in empowering students to take an active role in their own learning and engaging them in transforming their knowledge and skills towards problem-solving and decision-making, ultimately fostering critical thinking. As a result, students are less likely to commit academic misconduct (Bretag et al., 2019).

For example, have students compare LLMs generated texts to a published article of their choice, focusing on a similar topic. This comparative exercise not only creates a more captivating and interactive learning experience (Rudolph et al., 2023b) but also mitigates the temptation for unauthorized use of LLM tools. By incorporating student choice and connecting assignments to their areas of interest, teachers foster a sense of ownership and encourage active participation in the learning process. This, in turn, has the potential to prevent fraudulent behaviour and academic misconduct.

Administration

In higher education, teachers play a dual role as educators and examiners, bearing the responsibility of identifying instances of fraud. Research on training teachers to detect student work produced by ghostwriters has shown significant progress in effectively identifying contract cheating (Dawson & Sutherland-Smith, 2018). However, the empirical examination of whether similar training on detecting LLMs-generated texts yields comparable results remains lacking.

We propose the need for training both teachers and students in understanding the characteristics and vulnerabilities of LLMs generated texts (refer to Table 1). By equipping teachers and students with this knowledge, we can promote responsible usage of these tools, ensuring academic integrity and enhancing the quality of their work (Cotton et al., 2023; Farrokhnia et al., 2023). Such training will enable teachers to better identify any misuse or inappropriate use of LLMs, while empowering students to make informed decisions when engaging with these tools.

Workshops

To exemplify the utilization of the framework for (re)designing writing assignments, we conducted a series of three workshops involving 30 teachers from two schools within two academic fields at a Dutch university. Although participation in these workshops was voluntary, the school’s education support team strongly encouraged teachers engaged in writing assignment assessments to take part.

Workshop descriptions

Each workshop followed a three-step approach:

1. Prior to the workshop, we requested participants to submit learning questions about designing assessments for writing assignments with LLM tools.

2. During the workshop, we explained the six dimensions of the framework and applied them to a domain-specific course, as well as addressed the learning questions aligned with the relevant dimensions.

3. Upon workshop conclusion, we collected verbal feedback from participants regarding the framework’s utility.

To prepare for the workshop, we reviewed the learning questions, categorizing them into three overarching categories: elucidation of university policies pertaining to the utilization of LLM tools in assessment practices (i.e., stated as the memo in Introduction), finding necessary changes, and presenting good examples of practices that involve using LLMs with or without permission. The learning questions displaying in the presentation slides were subsequently deliberated within the context of the relevant dimensions of the framework.

An exemplar question that pertains to the dimension of Purpose is as follows: “I guess that regarding Chat GPT our role is to guide students into using it in a smart way that actually benefits them, but I do not fully know how we can do that. For instance, to what extent do we allow them to use it? Should we support them using it for some tasks and not for others? If the exam/assignment is about them writing an academic paper (so showing that they have acquired the necessary academic writing skills), how can we draw the line between what they can do and what they cannot do with Chat GPT?”

To illustrate the application of the framework, for instance, in the social and behavioural sciences workshop, we utilized a Research Master’s program course in psychology. This course featured a high-stakes blog assignment, accounting for 75% of the overall grade, and aimed to foster critical thinking skills. However, it’s important to point out that the writing task posed a vulnerability, as it left room for potential misuse of LLMs for ghostwriting. The assignment merely mandated the submission of final blog posts through a platform curated by the program.

We reviewed the LGs and the assignment description of this specific course by employing a flow chart (Moore & Stanley, 2013). The aim was to assess to what extent the original design aligned with the intended assessment of critical thinking skills outlined in the LGs. Notably, this flow chart was created prior to the emergence of LLMs, yet its depiction of critical thinking resonates remarkably well with the capabilities of LLMs.

Subsequently, we recommended adjustments in alignment with the dimensions of the framework:

• Purpose: We made the four essential cognitive processing of blog writing explicit in the assignment goals, namely triggering (initiation of critical inquiry), exploration (brainstorming, questioning, and exchange of information), integration of ideas to connect, describe the issue or problem under consideration. We further delineated which processes could be augmented by authorized used of LLMs, such as brainstorming.

• Function & Focus: To reach a balance between process and product, we introduced a draft version stage and incorporated peer feedback.

• Grading criteria: We placed emphasis on grading criteria pertaining to critical thinking. Students were mandated to provide peer feedback guided by these criteria (refer to Table 1).

• Modes: Students were instructed to incorporate both visual elements and textual content within their blog posts.

• Authenticity: We implemented a mechanism wherein peer feedback is harnessed for continuous enhancement in subsequent work, mirroring real-world applications where utilizing feedback is pivotal for improvement.

• Administration: Comprehensive instructions were introduced, distinguishing between authorized LLM uses (e.g., generating blog ideas using ChatGPT) and unauthorized applications (e.g., employing ChatGPT to evaluate peers’ blog posts and offering suggestions).

Our observations and participants’ feedback

At the outset of the workshops, we noted three primary issues. Firstly, participants were frustrated by their perception that LLMs had the capacity to demonstrate critical thinking, leading them to consider abandoning writing assignments in favour of traditional sit-in exams. One participant voiced this sentiment: “ChatGPT can write better than my students: the generated texts shows a high level of comprehension and synthesis, and there is no error in the APA style”.

Secondly, they believed that their writing assignments were primarily intended for assessing critical thinking skills. However, upon using the flow chart to examine the blogs or their assignment descriptions, doubts regarding the original design arose. One participant’s viewpoint reflected this change: “I used to think that summarizing various theories was a form of critical thinking. However, my certainty is now gone when I understood the strengths of LLMs and used the flow chart”.

Thirdly, they were unaware of the relationship between the limitations of LLM tools and critical thinking criteria, causing them to doubt their ability to identify LLMs generated texts. After showing them Table 1, we used one sample undergraduate’s essay question and answers generated by ChatGPT version 3.5. By applying critical thinking criteria (see Table 1), participants gained confidence in detecting flaws in examples of LLM generated texts, enabling them to approach their role as examiners with greater assurance and proficiency. As expressed by one participant: “I did not think of applying these basic critical thinking criteria in examining student work. It seems to me that ChatGPT says so many common things like senior high school students. Its answer goes obviously against the precision criterion that is emphasized by our academic higher education”.

At the end of the workshop, the overall response was highly positive. The framework proved invaluable in helping them make crucial decisions for re-designing their writing assignments. They particularly appreciated the clarity of LGs and underlying steps, which guided them in determining the appropriate extent to which students could use LLMs. Moreover, the authenticity dimension of the framework prompted them to recognize that their writing assignments often prioritized knowledge telling over promoting knowledge transformation. The framework empowered them to reach a balance between leveraging LLMs’ benefits and ensuring the integrity of their writing assignments, ultimately enhancing the overall learning experience for their students.

Limitations and future studies

This report shares our experience of swiftly developing a framework to support teachers in structuring their re-designing approach to promote critical thinking while preventing academic misconduct under high time pressure.

Our primary goal was to address the impact of LLMs on student fraudulent behaviour in writing assignments. The framework was constructed based on a brief literature review of suggestions from various sources, some of which lack empirical validation. To further enhance the framework’s effectiveness, it is crucial to conduct an empirical study using this framework and complete steps 5 and 6 of the DSR process. This future study will help validate the framework and adapt it to the rapidly evolving landscape of LLM tools.

Acknowledgment

We would like to express our sincere appreciation to Professor Dr. Yvonne Brehmer and Dr. Mercedes Almela Zamorano for granting us permission to use their courses to demonstrate the application of our framework in re-designing their course writing assignments at the workshops.

Disclosure statement

No potential conflict of interest was reported by the author(s).