Assessment in Systems Analysis and Design: Simulation, Constraints and New Foundations

Abstract

Over recent years, we have attempted to design summative assessments for a final year undergraduate module in Systems Analysis and Design that engage students in stimulating, practical tasks that are based upon real-world requirements. This paper evaluates the progression of these ambitions and draws attention to the constraints that inevitably limit attempts to simulate software development environments. The paper critically reflects on lessons learned and anticipates improvements in the model, particularly the idea that student solutions can form a repository of material representing an advancing body of in-house knowledge, and the prospect of moving towards a more real-world learning and assessment environment. Current work is focused on analysing system requirements among a community of voluntary carers dealing with elderly residents. This has the advantage of forging valuable external links as well as providing a live user environment. It is suggested that such projects could also be integrated into other modules.

Keywords:

• Systems Analysis and Design
• Formative Assessment
• Summative Assessment
• Systems Modelling
• Real-world Scenarios

1. Introduction

This paper is concerned with designing a summative assessment in a Systems Analysis and Design module for final year undergraduate computer science students. The module has been running for the past few years during which time we have continued to modify the structure and content of the assessment. The emphasis throughout has been on requiring students to assess a particular scenario and to derive and justify data flow models (US Department of Commerce, 1993), entity relationship models (Chen, 1976), or object-oriented models (Booch, 1994; Booch et al, 1998) of the specified environments. A significant challenge has been to provide a context in which students can interact with a potential system user to get a sense of how to identify requirements and to convey the need for enquiry and interpretation in the process. The principal difficulty lies in providing a consistent setting for two groups of fifteen to twenty students as well as a group of some ten students located remotely in the Isle of Man who follow the same academic programme.

The simplest approach is to provide a written specification from which students can work. However, this approach lacks interaction. Successfully engaging students in purposeful projects is acknowledged as a challenge for those teaching modules in Systems Analysis and Design, as discussed in Section 2.

This paper describes how we utilised a former student’s experience in developing a web-based application for an external client to simulate user interaction. Attention is then focused more specifically on how feedback from that process has informed our current approach which involves the specification and design of a real-world system.

2. Pedagogical Approaches

A number of authors discuss the issues surrounding the teaching of systems analysis and design techniques; this is within the wider context of software engineering (Rajilich, 2012); from the more specialised perspective of requirements engineering (Laplante, 2009); and from a business analyst’s view of management information systems (van der Heijden, 2009). An important factor highlighted by Dathan and Ramnath (2010) is that there should be a practical component to students’ learning and that selected project tasks should be of a size and complexity that challenge learners yet enable them to complete a solution within the time and resource constraints of their academic programme. Many textbooks provide examples of structured tasks and projects enabling students to engage with a variety of scenarios in tackling systems-related activities: (Bennett et al, 2002; Dennis et al, 2010a, 2010b).

In addition to stressing the importance of practical engagement with appropriate tools and techniques, there is recognition that students can lack enthusiasm for projects that seem too artificial or contrived. Various approaches have been adopted to address this. Chen (2006) discusses collaboration with the university IT department in specifying and designing a new live system; Hasan (2002) has groups of students fulfil the roles of both user and developer to enhance their insight into the overall systems development process; Schatzberg (2002) introduces Lego-based classroom activities to provide a vital experiential learning dimension to introductory analysis and design modules. The latter approach aims to foster a wider understanding of the multifaceted demands of systems development.

The idea of linking systems development learning to the real demands of the outside world is addressed much more radically and comprehensively by Holcombe et al (2001) and Holcombe and Gheorge (2003). In establishing a student-managed software development environment, the authors aim to capture the demanding and potentially volatile nature of developing projects for real business clients with all that entails in terms of fulfilling requirements, meeting deadlines and delivering high quality systems. Given these aspirations, student teams are in a strong position to learn, from experience, which approaches to systems development deliver effective and reliable solutions. The authors maintain that this degree of autonomy and responsibility contributes significantly to learners’ sense of enterprise and their employability through the development of personal, technical and teamwork skills.

While Holcombe and Gheorghe (2003) are justifiably enthusiastic about the success of their initiative, it is evident that tutors need to work carefully to foster a suitable environment to stimulate enthusiasm among students with often limited scope to capture the range and intensity of demands that help learners acquire essential skills in a business-focused environment.

Work-Based Learning (WBL) initiatives also provide contexts within which students can acquire real-world development experience although securing relevant, stimulating placements may not always be guaranteed (Mason et al, 2003; Little and Harvey, 2006). Another alternative is to adopt Virtual Reality (VR) as a means of utilising simulated learning environments (Huang et al, 2010; Burdea and Coiffet, 2003; Sherman and Craig, 2003). Our approach, which sits somewhere between WBL and VR, represents an attempt to introduce some real-world vitality into a final year undergraduate module, notwithstanding the constraints of the module structure (see Section 3).

3. The Module

Advanced Systems Analysis and Design is an optional final year computer science undergraduate module. Currently, the module takes a holistic view of the software development lifecycle (Dennis et al, 2010a). This entails an overview of the key development stages and covers issues such as project planning, project management and resource allocation (Nicholas and Steyn, 2012). Technical considerations such as systems modelling, program design, systems architecture and system procurement strategies are covered. Within this framework alternative methods such as Rapid Application Development (RAD) (McConnell, 1996; Stapleton, 1997) and Agile Development (Martin, 2003) are evaluated.

Assessment comprises an end of module examination, worth about a third of the overall marks, which covers the full range of topics introduced. A single coursework assessment accounts for the remainder of the marks. In designing this summative work we have aimed to engage the students in specifying the user requirements, analysing the results of this step and developing a system model using a suitable formalism. Finally, they are required to critically evaluate these activities in the light of current approaches to software development.

We currently follow a well-established format in which course material is delivered in weekly sessions. This is complemented by practical activities, discussion, individual and group investigation and student presentations. Formative project work has been introduced into this format. These formative activities are meant to provide practice in the various techniques that are required by the summative assessment. It is debatable, however, whether this approach is as successful as it might be. This is one of the questions raised by our ongoing experience, which will contribute to our discussion here and may inform future modifications to the module, as discussed in Section 6.

4. Utilising Existing Expertise and Resources

Recent departmental developments led to the formation of a specialist centre in which a group of students, staff and computer professionals worked on external projects (Kerins, 2011). This work provided a potentially rich source of evolving expertise and experience which could be used to enhance teaching and learning throughout the academic programme. We decided to select one of the projects as a basis for our assessment. This presented some practical difficulties: we wanted to let students get a sense of what faced the system developers when they embarked upon the project. It was impossible to provide access to the original client so we sought to simulate the process by having the developer, by now a graduate who was still contributing to the work of the centre, present the requirements as they had initially been presented to him.

Although the developer agreed to volunteer his time to present the scenario, some practical considerations remained. The aim was to enable students to listen to a description of the user requirements and then to let them formulate questions to clarify their understanding of the environment before devising their own system specification. While it was infeasible to support individual interviews, one option was to have meetings with class groups at which questions could be posed. Apart from the impracticality of engaging colleagues in the Isle of Man in these sessions, the reality of student attendance led to the likelihood of some absences from these sessions. This would potentially disadvantage absentees. Another option would have been to set up the session by videoconference. This could potentially accommodate all interested parties but it would have been difficult to coordinate timetables to find a suitable time slot.

As a compromise, we decided to create a video clip in which the developer posed as his former client and presented details about the client’s business, the limitations of his existing and somewhat outdated web site, and his requirements for a more advanced solution to enhance business processes. The developer presented a non-expert view of the system, which gave the students flexibility in suggesting potential features with which a genuine client may well be unfamiliar.

The next step was to facilitate interaction between the students and the Client without making the process unduly onerous for the volunteer. We decided to use e-mail given the infeasibility of face-to-face communication.

4.1 Structuring the Simulation

Once the short video was complete, we made it available through the module web pages so students could view it as and when it suited them, so there was no difficulty in coordinating access to the material. We posted instructions on the module assessment page stipulating that students should view the video and formulate a system specification as the first task in their assessment. They were encouraged to clarify their understanding of the requirements by e-mailing queries to the developer. This device was used for pragmatic reasons: it was impractical to facilitate direct contact between a sizeable number of students and a busy volunteer. The latter agreed to respond to individual queries. He waited until a number of queries had been received so that he could compose generic replies where appropriate.

At this stage we considered summarising the responses so that everyone would benefit from the questions that had been posed. We decided against this on the premise that individual queries should remain private so that individuals could exploit their own ideas. In fact we have taken a different approach in our current iteration of the process, as discussed in Section 5.

A time limit was imposed for submitting initial queries. Some two weeks were allowed beyond the deadline for the developer to respond to queries. An additional deadline was set by which students could engage in further dialogue with the developer to follow up their queries. The developer continued to simulate the business requirements of the client with whom he had worked. He also sought to replicate the client’s level of technical knowledge; this meant that a number of design decisions were open to student recommendations.

4.2 Student Reflection

When the specified deadlines for interaction with the developer had expired, the students’ task was to use the information to develop suitable use cases for the system and to produce associated system design models. They were also required to provide a rationale for their approach and to critically evaluate the process.

The evaluations provided useful insight into their reflection on the process. A critical observation was that they were frustrated that they had been unable to continue to interact with the Client. Many of them viewed the task as being best tackled with a RAD approach and they recognised the artificial constraints imposed by the simulation.

Of course, this was a consequence of the compromise of trying to exploit the knowledge, expertise and experience of the developer without placing undue demands on his time. We considered how the approach could be improved and an opportunity emerged from a genuine requirement to investigate existing community activity.

5. Reviewing the Format

By utilising a completed project we had managed to link the assessment task to a real-world scenario which helped to make it more tangible and relevant. Subsequent discussion with a retired colleague revealed an opportunity to introduce a live scenario into the process. He was involved with a network of volunteers who provided support to elderly residents in a local community. A system had emerged in which volunteers collected residents from their homes and escorted them to weekly sessions where they would meet, have lunch and take part in activities. There was clearly an opportunity to introduce IT resources to help with scheduling and coordinating the process and in making relevant information more readily available to both volunteers and to the elderly participants in the scheme.

This project presented a different perspective. No IT system was in place and the requirements seemed potentially broad and not particularly well defined. However, there was a genuine client in this scenario who had an interest in discovering what kind of solutions might meet his requirements. He was also prepared to contribute to the process and to accept that the outcome would be a series of designs rather than a working system that the volunteers could deploy.

Many of the practical constraints outlined earlier still applied. Therefore we agreed to produce a video recording which would be available for students to consult independently, as before. This time we decided that a collective response would be provided taking account of all the queries submitted. Again three separate groups of students were contributing and they were given an initial deadline by which preliminary queries should be submitted. However, in view of the previous limitations of students not being able to clarify requirements or check details as their ideas developed, the Client agreed to continue to respond to individual queries throughout the process. He also agreed to update the collective response during this period if appropriate.

The rationale for producing a collective response was that the students would benefit from all the additional material generated by individual queries. The assessment required students to briefly explain how they had clarified system requirements. For those who had formulated additional questions after viewing the video this required them to think carefully about the system and to compose their questions accordingly. We would also expect to see the results of this enquiry informing their designs. They could also evaluate ideas generated by other students and incorporate them where they might enhance their own thinking. We would expect this kind of beneficial cross-fertilisation of ideas in a team environment. We considered that these advantages outweighed any sense of unfairness that students who made no contribution to the generation of additional ideas were, nevertheless, able to utilise the collective response as freely as those who had.

6. Formative Opportunities

Given that we had established a new scenario for the current summative assessment, we decided to reuse the previous material by introducing formative tasks based upon small project teams within the groups. Essentially, the web application video was made available and groups were required to devise a specification, as before, and to use the scenario in undertaking various modelling techniques as they were introduced during the module. They had no access to the Client to clarify the requirements but they were encouraged to make their own assumptions and decisions about how the environment might be interpreted.

The aim of this initiative was to provide a test bed in which students could practice the tasks which they were required to carry out in their summative assessment. It was also designed to introduce an element of team working which would encourage collaboration and provide some insight into the way members of software development teams operate interdependently.

The results of this initiative have been somewhat desultory. There may be a number of reasons for this, and no careful analysis has been carried out. It may be that students understandably gauge the assessment requirements for their modules. They determine where their effort must be focused in order to optimise their results. These are significant considerations in circumstances where students often hold down part-time jobs.

We may also need to modify the structure of the module if we want to integrate this activity. Principally, we deliver material formally during weekly sessions and then set individual or group tasks which represent a mixture of briefly summarising key conceptual points, consulting external sources, utilising software tools and building system models, for example. The outcomes from the sessions contribute to a personal portfolio. In the past the portfolio has been used as a basis for summative assessment but the option of using real-world projects such as we describe in this paper is more stimulating. Some sessions are set aside for working specifically on these practical tasks. It is here that we have linked the work to the team project. The aim has been to augment these sessions with brief presentations evaluating work in progress.

Doing this well requires motivation, persistence, co-operation and perhaps most significantly, incentive. The approach presented here places responsibility on learners to work independently, to undertake challenging tasks and to be prepared to present their findings. It seems that a number of factors work against these aims. One is time: we have limited class contact time so if these tasks are to be achieved then the students need to do some of the work elsewhere and this places additional demands on them. Some students do not feel confident presenting work publically, even though this is among people whom they know. Poor attendance also undermines this kind of initiative. Introducing the idea of a team project has been experimental; the fact that no formal assessment marks are at stake means that it will not have any impact on student results. Indeed, this may account for its limited success. (Although an optional question on the end of module examination paper focuses on providing a critical insight into working as a member of a team on a software development task). Students recognise that there are many demands on their time and they must, of necessity, focus on the activities for which there is sufficient incentive to do well.

We can reflect on whether and how we might modify module delivery and assessment to integrate project work. Our current aim has been to do this as a means of enhancing the student experience and helping prepare them for the summative assessment which runs in parallel. This may not be a feasible aim in view of the observations above.

There is nothing new in the idea of group work and group assessment. We have probably been overambitious in trying to use it with limited time and resources as a formative exercise in a demanding final year schedule. If we were to introduce group work into the summative coursework then we would need to think carefully about group management and assessment methods: anecdotal evidence suggests that difficulties can arise in summative assessments where students are required to work in groups to produce specified deliverables. Concerns are expressed that individuals are absent from class, can be difficult to contact, and are perceived not to be making sufficient contribution to the project. However, these issues can be managed successfully, as Chen (2006) advises.

There may be an argument for focusing explicitly on doing the summative assessment in project groups during classroom sessions. Taking account of the significant literature in Organisational Learning (Prusak and Matson, 2006; Easterby-Smith and Lyles, 2011), it may be advantageous to treat the module as a live project with project groups responsible for managing the overall process and for the quality of the project deliverables as they would be in a professional environment. This shift in emphasis might also enhance students’ employability (CHERI and KPMG, 2006; QAA, 2010).

7. Current Progress

The Client’s response to the initial e-mail queries acknowledged the range of questions that had been posed. As he had chosen not to reply to individual queries, he composed a comprehensive and informative summary of the points that had been raised, which served as a collective repository of client information.

This feedback identified the kind of exceptions and exclusions that were inherent in the real system to be modelled. Specific conditions were clarified, such as the need to handle circumstances where volunteers were unavailable, or where routines were interrupted by holiday periods, or where premises for activities had to be changed, for example. Questions concerning data protection and system security were also raised: in some cases, details of medical conditions need to be made available for volunteers, for example. Effective communication was identified as a significant problem facing participants in the system and this provided students with an opportunity to evaluate the kinds of technological solution that might best address this issue.

The Client defined the scope of the project, excluding considerations such as finance, catering responsibilities and any branding of the charitable organisation. The Client’s response provided a clear and comprehensive account of the key system requirements. The students acknowledged during classroom sessions that they found this very helpful and seemed confident that they had enough information to complete the task.

7.1 Evaluating the Solutions

From the students’ perspective, they were generally satisfied that they had sufficient understanding of the system requirements to meet the assessment criteria, which entailed the production of relevant use cases and system models. They used the context to develop solutions of varying levels of complexity and incorporating technologies ranging from web applications to a comparatively simple yet effective spreadsheet-based system. The student submissions were assessed on the quality of the deliverables and the accuracy with which they had utilised appropriate techniques. The results were generally good, with a few exceptional pieces of work featuring comprehensive, accurate models. Overall, they were consistent with the academic profile of the cohort and the standard of work was comparable to that of the previous cohort.

The involvement of a client with a genuine interest in the solutions has added a valuable dimension to the process. On reflection, this offers insight into what has been achieved and more particularly into some of the limitations of the approach. A central issue, raised by the Client, concerns the audience for the solutions and illustrates a possible tension between the requirements of the Client and those of the assessment. The latter focused on formal, accurately produced deliverables that were consistent with a given set of requirements. These may not necessarily match the requirements of the Client and associated system users and stakeholders.

We want students to develop skills in producing and interpreting system models. This is an iterative process which enables them to refine their representations. Ideally, such models will inform their discourse with development team members and with clients. Selecting an appropriate level of detail for communication demands practice and experience, and the Client’s observation highlights the real-world importance of effective interaction. In fact, he goes on to acknowledge that, “…many issues would have been ironed out in a real life case with face-to-face meetings.”

A number of specific issues were also highlighted in the feedback:

“…few students offered solutions to the key problems of re-arranging rotas at short notice or engaging volunteers who were not IT proficient.”

“There was often no distinction made between key volunteers, who are ‘key’ in running the club, and other volunteers who generously help out but who are not involved (in) organising (the club).”

On the other hand, potential benefits were also acknowledged:

“I was intrigued to see how some students had expanded the project to include a general website generating newsletters etc. This was a useful idea.”

The Client’s conclusions acknowledged the inherent constraints in the process:

“Summing up, I thought that the students provided a good set of solutions, recognising that they only had a brief problem-setting video and limited, albeit structured, feedback. Ideally, it would be advantageous to meet up with students when solutions start to emerge. However I can see serious logistic problems in achieving this with a large group on two campuses.”

7.2 Comparing the Simulations

As noted in Section 4.2, when critically reflecting on their work, students involved in the initial simulation remarked, as in the following example, on the constraint of having, “…no continuing opportunity to contact and discuss ideas and assumptions with a client.” In fact, 54% of the students’ critical reflections commented similarly on the limitations imposed by restricted access to the Client.

In contrast, only 25% of the recent reflective reports referred to client accessibility. There was also a qualitative difference between the two sets of comments. The recent cohort’s observations generally acknowledged the limitations imposed by their inability to engage more directly with the Client and other system users. Unlike their predecessors, they were free to e-mail the Client at any stage. They also benefitted from the fact that structured responses derived from all the questions put to the Client were made available to all the students (see Section 5). Arguably, these responses might have prompted additional questions to be posed to the Client. In the event, students considered that they had enough information upon which to base their system requirements.

It would appear that the recent cohort was more satisfied with client interaction and with the information available to carry out the assessment tasks then the previous cohort was. Nevertheless, the students recognised the artificiality of the simulation and gained insight into the true nature of acquiring a deeper understanding of system requirements, as the following observation indicates:

“In the future I would most definitely send out a small questionnaire to all the users of the system and ask them exactly what they think about the new computer based system and also what changes they would like to see. I would also attend one of the weekly meetings and personally ask the members, volunteers and drivers what they think of the new system and also what they would like to see differently. Asking the users face to face will more likely give a clearer and better answer as to what the users actually want … ”

Finally, one student acknowledged the contextual constraints:

“The contact limitations with [the Client] were regrettable but understandable due to the number of separate individuals also working on this scenario.”

The idea of enhancing the simulation presented in Section 4 by introducing a current system requirement and extending e-mail access to the Client has not eliminated the fundamental desirability of closer collaboration between system designers and system users. Some of the students’ reflections indicate that, in engaging with the task, they have grasped its limitations. Arguably, this has enhanced their awareness of the range and complexity of real-world development.

Despite the limitations of the approach discussed in Section 7.1, the Client response adds very useful detail and we now have an opportunity to package the video, the Client summary in response to student queries, selected solutions, and the Client appraisal of the solutions into a comprehensive case study. Significantly, students’ attention can be drawn to the shortcomings of the solutions to raise their awareness of what the complete process entails.

8. Continuing Review

The initiative reported in this paper reflects our continuing attempts to link the curriculum to real-world scenarios. We have acknowledged other authors’ comments on the importance of stimulating students’ motivation and interest. We have been concerned with a specific final year undergraduate module; while we do not want to introduce the demands of working on live commercial projects, we do want to foster the sense of purpose that such developments entail.

The scenario presented is linked to real-world requirements. We have evaluated the advantages of handling the task as a video recording with interaction and feedback among three separate groups of students. The combination of video, collective response, selected requirements specifications, use cases and system models potentially form the basis of a repository of student-generated materials that can serve as exemplars for subsequent assessment tasks. In this particular case, such resources could also be utilised in other modules. For example, one of our final-year undergraduate modules, Intelligent Technologies, could explore the scenario and its associated solutions in investigating the design and implementation of intelligent software to enhance services for elderly members of the community and those responsible for their care.

The accumulation of case-studies offers scope for students to develop in-house styles and standards which could be adopted or refined by subsequent cohorts. These ideas fall short of the commercially oriented enterprise of Holcombe and Gheorge (2003) but they offer potential to simulate these activities and to move towards a more interactive learning environment where students can assume more autonomy in acquiring systems analysis and design skills. As such an environment emerges it presents opportunities to address its structure and to investigate areas such as knowledge management (Hislop, 2009), organisational learning (Brandi and Elkjaer, 2011; von Krogh, 2011), and automated support for learning (Dimitrova, 2003; Schulze and Penner, 2008).

By engaging students in this way, and by continuing to link developments to external system requirements, we can enhance students’ knowledge and understanding by practical investigation, and we can increase their employability.

9. Summary and Potential Development

This paper articulates our evolving experience of teaching Systems Analysis and Design in which we aim to balance the practical constraints of handling diverse student groups with the aim of providing a stimulating and purposeful assessment task. We have used internal resources to simulate external links and more recently we have engaged directly with the community to introduce a more realistic and dynamic scenario.

These initiatives have produced some impressive solutions which offer potential to develop an in-house repository of resources which can support subsequent cohorts in developing in-house standards and practices. The possibility of enhancing our evolving model with a wider range of communication technologies such as social networks, for example, remains to be explored, and we can also reflect on the potential benefits of moving to a team-based project. The feedback from the client with whom we have worked on real-world system requirements has added valuable insight to the process. This has contributed significantly to the module’s evolving resource base and helps inform our view on how we might continue to enhance student assessment.

We are building the foundations of an environment in which students can utilise their predecessors’ knowledge and expertise and take more responsibility for managing their own project planning and delivery.