A Module Designed with Chaos and Complexity in Mind

Abstract

This case study focuses on the conceptual framework, implementation and evaluation of a ‘chaotic learning’ based module which utilised an ‘assessment for learning’ rather than an ‘assessment of learning’ approach (Black & William 1998). The development entailed combining different types of teaching and learning methods (face-to-face learning, eLearning and project-based learning) to create an Object Oriented Analysis and Design module for second year computing students at Plymouth University. Initial results show that students gained valuable workplace skills and that attainment results were improved compared to those of the previous year’s cohort.

Keywords:

• chaotic learning
• non-linear learning
• assessment for learning
• project-based learning
• extended blended learning

Introduction: A background based on chaos and complexity

There is a lack of empirical evidence in educational literature about chaotic learning and indeed, very little has been written about chaotic learning at all. It is against this paucity of chaotic learning literature that Lee’s contribution stands out (Lee 2000). Elements of Lee’s work have recently informed the development of an Object Oriented Analysis and Design (OOAD) second year undergraduate module at Plymouth University. Lee stated that “the young who have grown up using the Internet have developed a particular learning style, one which is chaotic and non-linear with a central ‘play’ theme.” (Lee 2000).

The following definition of chaotic learning is based, in part, on Lee’s work. Chaotic learning is “playful, exploratory and collaborative learning that takes place within a system which is complex and non-linear, whose outcome may be unpredictable and difficult to determine in advance, but whose whole may be greater than the sum of its parts.” (Schoenborn & Neal 2010).

The possibilities inherent in chaotic learning led the Plymouth OOAD module designers to consider whether and how they might employ teaching methods using Web 2.0 that could exploit chaotic and non-linear elements to improve learning. (Web 2.0 is the second generation of the World Wide Web that incorporates new features and functionality, particularly in terms of the possibilities for collaborative work, to that which was possible using the original World Wide Web.)

This case study paper describes the subsequent development, implementation and evaluation of an OOAD ‘chaotic learning’ based module, which utilised an ‘assessment for learning’ rather than an ‘assessment of learning’ approach (Black & William 1998). The development entailed combining different types of teaching and learning methods (face-to-face learning, eLearning and project-based learning) within the OOAD module. The paper focuses on three main elements: an introduction to the conceptual framework for module delivery, implementation of the module, and multi-staged evaluation – the latter of which showed that students gained valuable workplace skills and that attainment results were improved compared to those of the previous year’s cohort.

Theories of chaos, complexity and chaotic learning: the conceptual framework

Chaotic learning derives from mathematics-based chaos theory as well as science-based complexity theory and loosely describes non-linear learning.

Chaos theory is the study of non-linear dynamic systems, first pioneered by Lorenz (1963) and according to Bloom (2001), is characterised by unpredictability and emergent properties in chaotic or random phenomena. Organic systems are largely non-linear, including the brain and ergo, the way we learn.

Complex systems are also non-linear and can exhibit holistic properties that are greater than the sum of their parts. Bloom (2001) characterises complexity theory as addressing complex relations within (chaotic) systems and notes their self-generating/self-maintaining properties. If one accepts the premises of chaos and complexity theories, one can deduce that thinking, learning and behaviour may therefore be characterised as intrinsically chaotic and complex, unpredictable and non-linear.

Despite this unpredictability, employing a chaotic learning design does not negate the possibility of planning and preparing learning outcomes and an outline of material to be covered in a learning module, based on an assumption that some form of student engagement will take place, even if the exact ways in which learning is manifested may themselves be unpredictable.

Such a chaotic learning design, when constructed with student collaboration in mind, can be based on concepts of social and communal constructivism and, in recent years, many learning materials and activities (including eLearning) have been designed based on those concepts. In social constructivism, learners construct knowledge through social interaction with others (Vygotsky 1978), whilst in communal constructivism (Holmes et al. 2001) learners actively build knowledge beneficial to other students, with help from wider communities and in collaboration with their peers. Schoenborn (2008) has described the latter as providing a potentially chaotic learning environment in which tutor engagement might aid learning through the provision of support and appropriate scaffolding.

Phelps (2003) asserts that when using the World Wide Web, itself a chaotic learning environment as noted by Lee (2000), students and others adopt online learning approaches consistent with complexity theory and indeed, communal constructivism. Their learning is non-linear, mirroring the structure of the Web. The types of activities that they engage in, including online discussions, wikis, reflective blogs and webcasts, enable and support group collaboration and interaction and are controlled and developed by the authors themselves. It might therefore be argued that there is scope for tutors to relinquish a certain degree of control and cede it to students. According to Bloom (2001), greater student control would result in a dramatic increase in the potential for learner development of complex understanding, meaning making and the integration of content and process involved in learning discourses. Recent pedagogical developments in higher education, such as Pedagogy 2.0 (McLoughlin & Lee 2008) have embraced this philosophy.

The project: An exercise in student autonomy

It is against such a background that the OOAD module was developed. A project-based learning approach was adopted in which, as Jonassen et al. (1999) suggest it should, the task became a process of learning that focussed on questions, problem-solving, decision-making and generating solutions in both structured and unstructured learning situations. The use of authentic professional scenarios is important for such learning (ibid) as described below. The OOAD module incorporated such a scenario.

In addition to scheduled class time, online delivery of project-based learning in a shared, distributed environment was used to enable learners to collaborate in virtual communities of practice that related to situated learning (Lave & Wenger 1991), whereby learning occurred unintentionally, as a function of an activity and the context and culture in which it took place (ibid). The potential pedagogical advantages include greater learner and learning autonomy, a sense of ownership by students and increased learner motivation (Simon, cited in Wheeler et al. 2005).

The finished chaotic-learning design was situated within an extended blended learning environment. Extended blended learning is conceived of as a mix of face-to-face learning, eLearning and project-based learning (Schilke et al. 2006 ) and the redesign of the Object Oriented Analysis and Design module provided an ideal opportunity to incorporate this and to thus develop an innovative module that would not only incorporate elements of chaotic learning and extended blended learning, but would also incorporate assessment for learning centred around a single authentic case study rather than assessment of learning (Black & William 1998).

In previous years, the case study for the group assignment was written by a tutor and based on a fictitious garden centre, and although students worked on the assignment intermittently throughout the year, tutorial exercises frequently featured different scenarios to that used in the assignment. For this study, the design for the module was based on an authentic project, whereby students had to develop an I.T. solution for a local industry partner. Only one scenario was used, which was written and introduced by the industry contact and based on a real project entitled ‘Resources Interface Project’ – a new system to be developed for verifying and transferring data between departments/different systems and Web services. Students had to analyse and design a system using the Unified Software Development Process, Unified Modelling Language and Structured Query Language (USDP, UML and SQL respectively). As part of this, students were required to deliver ‘Use Case’ diagrams and descriptions, ‘Class’, ‘Sequence’, ‘Communication’, ‘Deployment’ and ‘State’ machine diagrams, operations specifications and an overview of the entire interface project.

By working on an authentic project over the course of the module and with assessment for learning integrated into it, rather than as a separate element tacked onto the end of a module, it was felt that students might not only learn more deeply as they engaged with a real-life scenario on which such engagement was assessed, but that they might also accrue evidence that they could include in their personal development portfolio which may increase their employability.

Elements of ‘self-organised learning’ were incorporated to enrich students’ learning experience and the Moodle Virtual Learning Environment (VLE), because of its inherent social constructivist approach, was used to provide a suitable platform for the project.

The traditional weekly structure of a one-hour lecture followed by a one-hour tutorial for the module was abandoned and instead, students participated online and in face-to-face seminars and progress meetings with the tutor, as well as with an industry contact who introduced the assignment. The acquisition of subject knowledge was achieved through self-organised learning and peer support. The 51 students of the first cohort undertaking this revised module were divided into two seminar groups and each of these groups were divided into teams of three or four. For each week of the module, two seminar sessions were held (one for each seminar group). Each seminar was of two hours duration. In total each group had nine weekly seminars, of which two were tutor-led; the remainder were led by the students themselves. Additionally, each week a two-hour block was timetabled for the entire cohort to come together for team progress meetings. The progress meetings were used by the groups to check on weekly objectives, as set out in the Moodle VLE, and to work together on either seminar preparation or on the assignment. In these sessions, students discussed their strategy and progress with each other and the tutor would talk with teams and individuals, note progress on a progress chart, and provide continuous formative feedback on work done. Should a team need more specific instruction to fully ‘grasp’ a concept, the tutor would encourage peer-support from (an)other team(s) for additional help and support. This continuous contact and support combined with the tutor’s record keeping helped to address and resolve any problems in the teams. Professional conduct and communication were expected; for example, not every team member had to attend every progress meeting as long as they sent apologies to their team and completed the work as planned. A small proportion of the module grade was attributed to ‘module engagement’ (see Table 1) and although the maximum increase achieved by a student was only three marks and the maximum reduction was two marks, it seemed to provide sufficient motivation to uphold team communication and cooperation. The sessions were also used for progress discussions with the industry contact.

Table 1 Coursework marks explained.

	Group Coursework Mark	69.5
	Total Attendance %	Completion of Blogs %	Seminar Mark %	General Attitude %	Overall Module Engagement (ME) %	ME rounded %	Coursework Mark/80 %	Peer Assessment Share from pool	Peer Assessment Actual	ME (+ or −1 for each 10 % above or below 50)	Final mark
Student1	68.18	45.45	50	45	52.16	50	55.6	11	7.65	0	63.25
Student2	86.36	63.64	50	60	65.00	70	55.6	25	17.38	2	74.98
Student3	72.73	36.36	60	55	56.02	60	55.6	22	15.29	1	71.89
Student4	77.27	36.36	60	50	55.91	60	55.6	22	15.29	1	71.89

Each team was allocated a specific topic of the curriculum, e.g. class diagrams. Each team had to set up and populate a group wiki on their topic as well as prepare and run a seminar for their peers. Every team had access to all other teams’ materials and wikis to help develop the assignment, which encompassed all topics of the curriculum. Furthermore, teams were expected to collaborate on the assignment online through their wiki and face-to-face. Every student was asked to reflect on their experiences throughout the module by means of personal blogs. This number, type and combination of elements would, it was felt, provide the structure in which some chaotic learning, as defined by Schoenborn (2008), might occur.

As an incentive for students to fully engage with the task, an element of competition was incorporated into the assignment which, as stated, was based on a real project provided by a local industry partner. A prize was offered for the best team solution to the task, which was to be presented at an award ceremony.

To summarise, the activities undertaken by learners, tutor, and industry contact are outlined in Figure 1.

Figure 1 Activities of learners, tutor, and industry contact.

Although the tutor’s role may seem to involve a considerable amount of work, in total, apart from the ‘up front’ time to design and set up the module, the tutor time allotted to the course differed little to that allocated to modules which use more traditional teaching formats.

Similar to previous years, the assessment design incorporated tutor assessment as well as an element of peer assessment. The weightings for these were 80% and 20% respectively.

The main difference between the marking of previous years’ group projects and the ones produced during this study was that individual marks assigned by the tutor included a small element for ‘module engagement’ (ME). This component mark was calculated based on attendance, completion of blogs, seminar performance and general attitude. The average ME was rounded and only one mark per 10% above or below 50% was included in the calculation. For example, a final mark of 56% for a student with overall ME of 30% would be reduced by two to 54%; if her ME had been 50%, the mark would remain the same; if her ME had been 60%, the mark would be increased to 57%. Peer assessment entailed team assessment of the relative contribution that each member made to the overall assignment. Each team was allocated a mark pool of 20 multiplied by the number of team members (i.e. 80 marks for a group of four and 60 marks for a group of three). If all members were deemed to have contributed equally, each member would receive a peer assessment mark of 20. The marks allocated to each member were used with the percentage gained by the group as a whole, to arrive at an individual percentage mark for each member as follows: individual mark = (overall group percentage x 0.8 ) + (peer group mark x overall group percentage / 100). It was felt that this would help to ensure each team member had a vested interest in contributing fairly. To illustrate more clearly, Table 1 explains the marks achieved by members of one of the groups.

Results: Assessment outcomes and evaluation

As with any new innovative enterprise, evaluation is important and so a number of evaluations were devised for the module. As well as the evaluation provided by student achievement results for the module, three forms of student evaluation were undertaken: an ‘initial thoughts’ evaluation by students, a mid-term evaluation and an end-term evaluation which took the form of a ‘stop, start, continue’ evaluation. The results of each of the evaluations are shown below and discussion of the results follows on from these.

Achievement results for students undertaking the module

As can be seen in Table 2, more than three times as many students achieved a mark of ‘very good’ compared to the previous year’s cohort. Additionally, there were no ‘fails’ and those achieving a mark of ‘sufficient’ almost halved compared with the previous year’s cohort. Clearly there were significant gains in terms of student levels of achievement.

Table 2 Breakdown of student achievement at the end of the module (n = 51).

Marks	Year prior to new design	Year of new design
Very good	8	26
Good	6	5
Satisfactory	19	16
Sufficient	7	4
Fail	5	0

Initial thoughts evaluation

Students were informed about the innovative nature of the module and in order to identify potential areas of difficulty, students were asked what their initial thoughts were after the introduction to the module. This brought very mixed responses, although concern about others ‘pulling their weight’ seemed to stand out clearly. Indicative views expressed were as follows:

“I think this learning style has potential, although I am concerned about other students being responsible for content I need to learn. As long as [the tutor] polices the content and ensures that group members pull their weight, all should be okay. I am quite excited about this approach and as long as the above is addressed, it has the potential to work well.”

“Should be good, learning on own and helping group learn for assignment. More independency (sic). Weekly blog could be annoying. Worried about people not pulling their weight.”

“At first I was sceptical as I like order and being told what to do but after listening I am more confident. The only thing I worry about is whether other groups will do their work. If they don’t, then the assignment will be harder.”

“Still concerned that it’s too chaotic! I like the idea but I don’t think everything will run smoothly as people will have trouble making sure everyone does their bit.”

“Sounds scary, every other team is relying on us and we’re relying on everyone else.”

These concerns about reliance on others were anticipated at the module design stage and were addressed in several ways: firstly, through inter-team dependency as every team developed expert knowledge in a particular curriculum topic and therefore teams communicated and collaborated to help each other with the assignment; secondly, in the peer assessment and ‘module engagement’ elements of the assessment process, and finally, through continuous tutor support and progress discussions.

Overall, the initial responses ranged between those that showed excitement at the way the module was structured and those that revealed a measure of trepidation. Opinions varied about the requirement to use blogs and wikis, with some students questioning whether they were necessary, whilst others appeared to look forward to using them. Of those who mentioned industry involvement, the majority were extremely positive. Apart from the concerns noted above about others ‘pulling their weight’, the initial responses, although mixed, contained more positive responses than negative.

Mid-term evaluation

The mid-term general evaluation was administered as a questionnaire which included a combination of open and closed questions. The analysis revealed a ‘happiness index’ whereby the majority of students responded positively to 60% or more of the questions.

As Table 3 illustrates, there were significantly more positive responses than negative, and of the 38 respondents from a potential population of 51, only five gave more negative than positive responses.

Table 3 Breakdown of student responses by percentage of ‘happiness’ for mid-term evaluation (n = 38).

Positive	Negative
No students	% happiness	No students	% unhappiness
8	100%	1	100%
9	90%	1	90%
4	80%	1	80%
4	70%	2	70%
7	60%	1	50%

Questions were also asked about various elements of the module. The two elements eliciting the highest total of positive responses related to teamwork, which was perceived as being productive or very productive, and materials provided by the tutor, which were available on Moodle and perceived as useful or very useful. Students who expressed negative responses regarding their experience of teamwork tended to express more negative responses generally than those by the students who had given positive responses about their experience of teamwork.

The element eliciting most negative responses was the student-led seminars, which 17 respondents deemed only partly valuable. However, it should also be noted that 16 respondents found these valuable and five found them very valuable.

Students were also given the opportunity to say what elements they might like changed. Of the 38 who responded to the evaluation, 15 did not suggest any changes. Of those who did suggest changes, the most frequently mentioned change was for greater tutor ‘direct’ teaching input, which ten students suggested.

In the ‘any other comments’ section, of the few students who responded (11), six said that they were enjoying the module, four respondents appeared to want more tutor input and one student expressed their dislike of the Moodle learning environment.

End-term evaluation

This was based on ‘stop, start, continue’, a widely used change management technique. Students were asked to record on post-it notes (one suggestion per note) which elements of the module they thought should be eliminated (stop), what could be introduced to improve it (start) and what was good about the module (continue). The results showed that students appeared to be dissatisfied with student-led seminars and this was consistent with the mid-term evaluation responses. Other notable ‘stop’ categories included the use of blogs, Moodle and calendars.

Figure 2 Breakdown of responses to ‘start’, ‘stop’, ‘continue’ evaluation task.

More positively, there were no suggestions about what might be introduced to improve the module, i.e. no ‘start’ responses, and there were considerably more ‘continue’ responses than ‘stop’ ones. Notably, three areas that elicited no negative responses were: lecture slides, tutor seminars (of which there were two), the group assignment, and progress chats, which the tutor held with groups and one-to-one.

Discussion

For the most part, the evaluations were encouraging, particularly in terms of achievement, which is consistent with a study by Sorbral (1997), cited in Gatfield (1999), which demonstrated that the students working in groups achieved a higher grade point average than those working in a control situation involving individual students. Other authors, such as Johnson and Johnson (1994) and Baloche (1994) have also argued that a majority of research indicates that group work promotes greater academic success. Whilst this may be true, at this stage, the possible link between group work and higher achievement can only be tentatively explored, particularly as the previous year’s assessment tasks were also in form of group assignments. Furthermore, the OOAD student achievement results could simply reflect an exceptional year/cohort and this needs to be monitored in successive years. One additional consideration is that one of the authors of this paper was a module tutor and assessor.

The ‘initial thoughts’ evaluation brought very mixed responses, as indicated in the previous section. The last student response in the ‘initial thoughts’ section reflects the fact that each team was responsible for delivering a seminar to the rest of their cohort about a specific part of the system and that within each group, each member would need to contribute fully to the success of the whole. In total, seven out of 26 respondents were concerned about relying on others ‘to do their bit’. This is consistent with the literature about students’ fears about assessed group work (Johnson & Johnson 1994, Gatfield 1999, Falchikov & Goldfinch 2000, Burdett 2003). Interestingly, the end-term evaluation showed that all the respondents felt that the group assignment should continue, which would seem to indicate that their initial misgivings were overcome during the actual process in which, with few exceptions, each student was seen to contribute fairly to group activities.

There was no ‘plan B’ to revert to a more traditional method for teaching and learning for the module, and it was therefore important that the mode of engagement expected by students was outlined very clearly at the start. For ethical reasons, the tutor was honest with the students from the outset about the newness of the approach used in the module and some of the students’ initial thoughts alluded directly to this fact. Some were excited by it whilst others evinced some trepidation. In any future runs of the module or for similar modules, it may be possible to somewhat allay student fears by referring to the success of the delivery of the module described here.

Many of the respondents were initially excited about the authenticity of the task, particularly that a real industry contact was involved, and this remained a positive aspect for students, as reflected in the end-term evaluation responses. Given the demands by policy makers and industry for greater links between higher education and industry (Committee of Vice-Chancellors and Principals of the Universities of the United Kingdom, Department for Education and Employment, Higher Education: Quality and Employability Section, CVCP/DfEE/HEQE 1998, DTI/CIHE 1990), this would appear to be an aspect of the module that meets these demands in a very cogent way.

For the mid-term evaluation, the happiness index showed that whilst six out of 38 respondents rated their unhappiness with the course at 50% or more, 32 respondents rated their happiness with the course at 60% or more, and indeed the data shows that at least 21 of the 38 respondents found it a very positive experience, scoring their happiness with it at 80% or over. It would appear therefore, that despite some initial fears about the course structure, by the mid-point of the course, many students had adjusted well to it.

The mid-term evaluations showed that just over half of the responses were favourable about the student-led seminars. However, this leaves almost half that did not and this concern by students about student-led seminars is consistent with findings by Daniel (1991).

Potential reasons for this level of dissatisfaction, also expressed in the end-term evaluation, might include students believing they could rely on the seminars for detailed information about a particular aspect of the assignment task and that the delivery fell short of their expectations, or that there were still residual concerns about team members not contributing a fair share of the work for such seminars. Alternatively, it may simply be that students were unsure about their own and their peers’ knowledge and, viewing themselves as novices, questioned their ability to teach each other. For any future runs of this or similar modules, more careful and gradual scaffolding of support should be implemented, e.g. interim deadlines for submission of seminar materials with subsequent feedback to groups, possible practice runs, monitoring of satisfaction with student-led seminars and the provision of more seminar subject resources online after each seminar so that reliance on information provided by students in the seminars is not seen to be as crucial from a student perspective.

This might also help to meet some students’ wishes for greater tutor input, an aspect that is mentioned further below. To ensure students learn through the process of preparing and running seminars, these contributions should, however, continue to be (at least formatively) assessed to provide useful feedback to students and to encourage quality presentations. One further avenue to explore is for module tutors to build confidence in students by reviewing their work even more closely during the preparation and delivery of these seminars. This will help staff to gauge students’ knowledge and confidence levels and to provide support as necessary, but nevertheless continue to gradually withdraw support as students’ confidence levels, ability and autonomy increase.

The end-term evaluations showed that a majority of students, as well as disliking student-led seminars, also disliked the use of reflective blogs, Moodle and calendars. Calendars may have been seen by students as somewhat redundant because of the use of regular progress discussions with the tutor throughout the course. Similarly, the value of blogs was not felt by students and it may be that some work needs to be done in order to demonstrate their value in terms of developing critically reflective skills that are valued in the workplace.

One concern for the course designers was the cohort’s negative response to Moodle. However, the designers feel with hindsight, that the layout of the VLE was somewhat over complex and that a redesign may be required, which will need to be subsequently tested.

Areas showing least support included those elements that students viewed as being unrelated to the actual task set, and it would therefore be useful to focus more clearly on the most appropriate elements only and explain or demonstrate the relevance of these to students.

Despite the aforementioned negative responses, the end-term evaluation showed more positive (‘continue’) responses than negative (‘stop’) responses and no suggestions were made about what other elements might be introduced. Importantly, the strongest support was for aspects that entailed significant tutor input and/or guidance. These aspects included resources such as previous year’s lecture materials, book chapter slides, weekly information, and tutor presence via tutor seminars and progress discussions. Additionally, the industry introduction, award and related ceremony appeared to be highly valued by students, as were team meetings. What the data therefore appears to indicate is that students involved in learning that incorporates elements of chaotic learning, whilst achieving higher assessment scores despite their concerns, nevertheless value aspects of the course that have strong guidance and reward elements. It may be that this serves as reassurance to students that they are ‘on track’ or it may be that such guidance is necessary because it is vital to imparting certain information that enables students to fulfil the tasks, which is itself rewarding. This would appear to resonate with Phelps’ (2003) view that students tend to expect what they are used to in terms of more directive approaches to teaching and learning, but is somewhat antithetical to the concepts underpinning the module’s chaotic learning design. However, as stated earlier, it may be possible to provide more subject resources and a more considered level of support in order to help students feel sufficiently supported. Overall, the findings resonate with the Vygotsky’s zone of proximal development (Vygotsky 1978), and although there are many benefits relating to participative and less structured pedagogies (e.g. Holmes et al. 2001, McLoughlin & Lee 2008), some of which emerged from this study, careful scaffolding and confidence building seem crucial elements in designing effective learning situations. Thus, elements of chaotic learning could be designed to increase in size, difficulty and importance according to students’ increasing abilities and confidence levels.

It would seem that there is a case for a more detailed and qualitative follow-up piece of research in order to gain some understanding that might help explain the reasons given by students about what they liked or didn’t like about the module. Whilst it is not possible with the cohort that undertook the module discussed in this paper, it may be possible with a future cohort.

A further avenue for research might be to explore the extent to which chaotic learning and similar collaborative and less structured pedagogies enhance the development of broader life and employability skills, and how student perceptions and decision making are used in order to de-risk their projects within this new learning mode.

Conclusion

The OOAD module incorporated a chaotic learning design element in order to provide a more authentic approach to learning, and represented a move away from didactic teaching approaches. It provided an opportunity for students to experience working on a real project, whilst developing highly transferable skills, such as the ability to work as part of a team and to take responsibility for a task, as well as presentation skills and other skills deemed valuable in the workplace.

The redesigned module challenged students who are more used to directive approaches, to take greater control of their own learning. The improved student attainment results however, appear to show that despite these challenges, student learning benefited from the module redesign, although it is too early to tell whether this will be replicated with other cohorts.

Many aspects of this study suggest that students require reassurance and signposting. It seems sensible, therefore, to carefully consider how students’ work is reviewed and showcased and the amount and frequency of tutor support and scaffolding to increase confidence in students’ own and each other’s work.