Jitse van Ameijde, Martin Weller and Simon Cross, The Open University, UK
With the implementation of student fees in many countries, increased focus on accountability of universities and metrics such as the Teaching Excellence Framework (TEF) in the UK, student retention has become a topic of greater interest to both institutions and students. Student retention varies considerably by country and discipline (OECD, 2014). Whereas vocational courses which are highly selective in entry, for example medicine and veterinary studies have low withdrawal rates, open entry study, as practised by the Open University (OU), has much higher rates.
The OU has a large student population of 170,000, so small improvement can have potentially significant benefits for the University, for example enhancing student retention across all the University’s large population courses (>1000 students) by 3% is estimated to increase University income by £2,195,100 per annum.
The OU operates an open entry mode of study, without requiring any prerequisites. This inevitably has a higher level of drop-out than filtered entry. Some students might realise that OU study is simply not for them or be unable to complete a programme of study for a variety of personal and professional reasons. In such cases taking a break in study could be the best option for the student with the possibility of students retaking a course at a later stage, something which has been made easier with the introduction of assessment banking. Similarly, there will be students who will pass and succeed largely on their own. However, there is also a subset of the student cohort for whom drop-out is not desirable and who could have been retained under the right circumstances. It is for these students that possible improvements in design strategies could improve retention. The aim of the project was to devise a set of design principles that could improve retention across all disciplines.
Enhancing student retention is important then for various reasons. Not only does student withdrawal have adverse consequences for students, for whom it can have both financial and emotional implications, but also for the university where high levels of student withdrawal will adversely affect both the financial and the reputational position of the university. With online and distance education study, retention can be a particular issue as students do not have localised face to face support. For students, failing to complete their first online course can adversely impact on their levels of self-confidence and deter them from engaging in further online study (Poelhuber, Chomienne, & Karsenti, 2008). For institutions of higher education, low student retention implies ineffectiveness of online courses and poor quality (Willging & Johnson, 2009). However, it is funded, there is also an impact nationally, with OECD countries spending an average USD 13 958 per tertiary student per year.
Many factors can influence retention, for example Australia reported an increase in retention when the fee system shifted to some financial contribution from the student, rather than all state funded, which implies that some form of financial contribution on the part of the student may well increase commitment (Australian Dept of Education and Training, 2017). The level of selection prior to study can also influence how many students will persist with study, as well as many personal factors. This paper will explore the issue of retention by focusing on the design-related factors, in contrast to the post-design intervention models of retention which have dominated much of the relevant literature. The research was conducted at the UK Open University, so there is particular reference to distance education students, although the model is more broadly applicable.
Various authors have observed that student retention is lower for online courses as compared to courses utilising traditional methods of delivery (Hiltz, 1997; Dutton, Dutton, & Perry, 1999; Terry, 2001). In a study comparing an online and a campus-based version of the same computer programming course, Dutton et al. (1999) found that online students showed a 72.2% completion rate whereas the campus-based course saw a completion rate of 90.3%. In a similar comparison at the programme level, Simpson (2013) observes that in a programme that has both a distance and a face-to-face version, graduation rates for the distance version are around one quarter of the graduation rates for the face-to-face version. Simpson (2004) argues that some level of dropout is inevitable and beyond the control of institutions of higher education, especially those cases where dropout is directly linked to issues of illness or family crises. However, the poor retention rates observed in distance education cannot be justified as merely a product of inevitable drop-out and research needs to focus on what can be done to enhance retention in a distance education context.
Simpson (2013) speaks of a “distance education deficit” (p. 106) with a large number of distance education institutions having graduation rates less than a quarter of those of traditional face-to-face institutions. Simpson attributes this deficit to the “category error” (p. 111) of confusing teaching with learning. He argues that many distance education institutions have traditionally focused predominantly on the provision of online teaching materials at the expense of motivating students to learn. He observes that student dropout and retention are the main focus of less than one-fifth of articles published in distance education journals, indicating that this issue is not at the forefront of the literature.
The most commonly cited model of student retention is Tinto’s (1975) Student Integration Model. Tinto argues that the process of dropping out of a higher education institution characterised by a set of social structures and values is analogous to the process of committing suicide in wider society. The underpinning social conditions leading to dropout are argued to be insufficient social integration and insufficient congruency with the culture and value system of an institution of higher education.
Tinto differentiates between the academic and the social domain of higher education institutions, and argues that dropout results from a failure to properly integrate with either or both of these. A failure to integrate with the academic domain involves issues in aligning with the academic values characteristic of the institution and a failure to achieve the academic standards set by it. Failure to integrate with the social domain involves a lack of alignment with and commitment to the social life of an institution.
Tinto argues that initial levels of commitment to the goal of completing a programme of study (goal commitment) and initial levels of commitment to the particular institution (institutional commitment) are driven by factors related to family background, individual attributes and former education. These input variables are largely outside of the control of institutions of higher education. The subsequent engagement with the academic and social domains of an institution will determine the level of academic integration and social integration, which in turn lead to revised levels of goal commitment and institutional commitment. It is these levels of goal commitment and institutional commitment which ultimately determine decisions to drop out.
Tinto’s model suggests that in order to enhance student retention, a higher education institution needs to be effective at facilitating both the social and the academic integration of students to minimise the likelihood of students deciding to withdraw. Rovai (2003), building on Tinto’s (1975) Student Integration Model and Bean and Metzner’s (1985) Student Attrition Model, proposes the Composite Persistence Model which aims to specifically address retention in the context of distance education. Rovai argues that distance education students are qualitatively different from face-to-face students in a number of respects which need to be taken into account when trying to understand student retention in a distance education context. He argues that for these non-traditional students, the academic and social integration emphasised in Tinto’s model are perhaps less important, whereas other external factors important for non-traditional students are ignored in Tinto’s model. Such external factors include family and work responsibilities and support structures outside of the higher education institution.
Kember (1995) developed a model of student progress in distance education and like Rovai based his work on Tinto’s Student Integration Model. Kember argues that students can follow one of two pathways which lead to either continuing study or dropout. The first pathway involves students successfully integrating in the social domain of an institution followed by effective integration in the academic domain through an acceptance of and adherence to the academic norms and values of an institution. The alternative pathway involves external attribution followed by academic incompatibility. Kember argues that less successful students who fail at effectively integrating academic demands with social and other demands on their time, often attribute their failure to integrate to external factors beyond their control, which tend to be followed by academic incompatibility. A failure to integrate academic and other demands is more likely for distance education students due to the fact that they often have to juggle family and work responsibilities along with study.
The literature on student retention indicates a wide range of factors which can influence it. In a large-scale study of 8,500 students in 33 colleges, Martinez and Munday (1998) found a number of factors which were associated with higher dropout among students, including where students:
Chyung, Winiecki and Fenner (1999) interviewed students (both withdrawals and continuing) on a distance master’s programme and found that the main factor which contributed to the decision on whether to continue or withdraw constituted the student’s level of satisfaction with the first or second course in the programme. Specific reasons for withdrawal included:
dissatisfaction with the learning environment
divergence between professional and personal interest and the structure of the course
low confidence in distance learning
hesitations about successfully communicating online
lack of competence in utilising distance education software
feeling overwhelmed by the amount of knowledge and information
Thompson (1997) found that the level of satisfaction of students with the quality of communication with their instructor was one of the key factors that differentiated between students who dropped out and students who continued with their studies. Astleitner (2000) found that not only communication and social interaction between students and instructors but also the level of social interaction between students and peers was an important factor in students’ decision to withdraw from an online course. Park, Perry and Edwards (2011) make similar observations and argue that online students face potential feelings of isolation as they do not have conventional opportunities to regularly meet face-to-face with educators and peers. Feelings of isolation puts students at risk of early dropout and regular contact with students, for instance through short motivational messages, as well as meaningful online opportunities for communication and collaboration, can mitigate the risk of isolating students.
A study by Glogowska, Young and Lockyer (2007) of the factors impacting on nursing students’ withdrawal decisions articulated four ‘pull’ factors which kept students on courses and six ‘push’ factors which contributed to decisions to leave. The pull factors were described as determination, commitment to the profession, informal support mechanisms and formal support mechanisms. The push factors entailed challenges of academic work, the load of other responsibilities, financial strain, lack of effective support mechanisms, early negative experiences and illness or injury.
A qualitative study by Perry, Boman, Care, Edwards and Park (2008) of dropout among online graduate students reported that personal reasons and programme reasons were the most commonly stated reasons for withdrawal. Personal reasons here included life or work commitments whereas programme reasons revolved around whether personal and programme learning styles matched and fit of the programme with career goals.
Lee and Choi (2011) conducted a review of the literature on online course dropout, and looked in detail at 35 empirical studies which were published over a 10-year period between 1999 and 2009. They found 65 factors which were associated with dropout on distance education courses. They divided these factors into three categories: student factors, course / programme factors and environment factors. Table 1 below outlines the 65 factors found by Lee and Choi.
|
Student factors |
|
|
Academic background |
· GPA · Previous academic performance · SAT math score |
|
Relevant experiences |
· Educational level · Number of previous courses completed online · Number of previous distance learning courses · Previous experience in the relevant field · Involvement in professional activities in relevant field |
|
Skills |
· Time management skills · Underestimation of the time required to balance their academic and professional obligations · Ability to juggle roles / balancing multiple responsibilities · Strong coping strategies · Resilience · Relevant prior computer training · Computer confidence |
|
Psychological attributes |
· Locus of control · Motivation · Goal commitment · Love of learning · Self-efficacy · Satisfaction |
|
Course / Program factors |
|
|
Course design |
· Team building activities · Program quality |
|
Institutional supports |
· Administrative support · Student support infrastructure · Orientation · Tutorial attendance |
|
Interactions |
· Inter-student interaction · Faculty interaction with students · Student participation |
|
Environment factors |
|
|
Work commitments |
· Employment status · Work commitments · Increased pressure of work · Changes in work responsibilities and environments |
|
Supportive environments |
· Financial aid · Support from family, work, friends · Emotional Support · Supporting environments allowing study time · Life circumstances · Life challenger · Life events |
Table 1 – factors impacting on student retention on online courses. From Lee and Choi (2011)
As Willging and Johnson (2009, p. 108) conclude, “…research has shown that the reasons for dropping out of a distance education course or program are complex, multiple and inter-related”.
From this literature and considering an institutional perspective the authors propose that factors influencing retention can be placed in four categories:
Design – factors in the design of the course, such as workload, introduction of topics, activities, etc
Presentation – factors during the presentation of any course, including communication with the institution, support from the educator and technical or environment issues
Personal – relating to the individual student, this can cover issues such as the student motivation, their existing knowledge and skills, as well as issues in their personal life such as home life and work pressures
Contextual – the context within which education is occurring, such as fee structure and employment market
Learner retention has become a topic of renewed interest with the advent of Massive Open Online Courses (MOOCs). While these free, open entry courses attract large numbers of initial learner registrations, their completion rate is low, typically less than 10% (Jordan, 2014). This area therefore offered another potential source of retention data, although learner motivations and commitment are different from those students in formal higher education (Yuan & Powell, 2013). While there is interaction between these categories, for instance personal factors may become problematic if a course has heavy workload designed into it that does not allow for time to address these. However, the focus of the Design for Retention project at the Open University was to concentrate on the design element only.
In order to devise the design principles, four approaches were adopted:
1 Review of relevant retention literature, and extraction of methods for enhancing retention
2 Analysis of Massive Open Online Courses (MOOCs) interventions and designs to improve retention
3 Analysis of student comments from the OU student experience survey
4 Interviews with experienced course designers at the OU focusing on retention.
The review of both the external literature and internal reports on student retention, which is summarised in the two sections above, highlighted several design related aspects which previous research had found to have a positive impact on student retention. Following Tinto’s Student Integration Model, these factors were grouped into those which enhance student motivation and goal commitment, those which enhance students’ academic integration, and those which support students’ social integration. This analysis was used to structure the model developed below.
MOOCs have an open entry model, and little or no support, so represent a particularly problematic area for retention design. Much of the attention on MOOCs has focused on their low retention rates, typically around 10% or less. They therefore offered a potentially new area for study where retention might be a particular focus. However, while retention has probably been the biggest topic in MOOC research, there was very little literature on design aspects that influence retention. Many of the factors identified relate to personal aspects of the learner or are inherent in the nature of MOOCs, such as lack of time, motivation, feelings of isolation, lack of interactivity in MOOCs, insufficient background and skills, and hidden costs (Khalil & Ebner, 2014). The use of discussion forums, and regular weekly activities were found to have a positive influence on retention (Adamopoulos, 2013). Interaction with the instructor (not always possible on a large course), and general interest in the content (Hone & El Said, 2016) were also influential factors. The data generated by large cohorts in totally online environments has led to interest in predictive analytics, which analyse behaviours that can predict dropout for example, fast forwarding videos can be an indicator of partial engagement, which predicts dropout (Kizilcec, Piech, & Schneider 2013). This may be of interest in integrating learning analytics into the design phase of development, which will allow for intervention.
The open comments data for six undergraduate courses (two level 1 Maths, Computing & Technology (MCT) courses, two level 2 Science courses and two level 2 Social Sciences courses) was analysed to distil design related factors which students comment on in relation to their study experience. A thematic analysis (Braun & Clarke, 2006) was conducted on a total of 458 student comments to elicit the key themes. Thematic Analysis is a method for identifying, analysing, organising, describing and reporting on themes found within a data set (ibid). An initial pass was made over the comments to familiarise the researchers with the data. An Excel spreadsheet was created with student comments on the rows and emerging codes in the columns. Each comment was analysed and where the comment contained material pertinent to the nature of the module or its features, an existing code was assigned or a new code was generated. During subsequent passes the coding scheme was refined and grouped into themes. The themes which were found were: ‘Out of date materials’, ‘Engagingly written materials’, ‘Usefulness of having print materials’, ‘Alignment between materials and assessment’, ‘Usefulness of Self-Assessment Questions’, ‘Timing of Tutor-Marked-Assignments and proximity to End of Module Assessment / Exam’, ‘Usefulness of formative assessment’, ‘Poorly executed communication / collaboration activities and Variety in activities’. Example comments such as “This module was far too diverse and unconnected for me, it tries to cover too much and the materials were not very engaging which limits effective learning” and “This module has been really engaging and enjoyable. Both the online resources and module book have been really interesting” were coded under ‘Engagingly written materials’ while comments like “All the teaching materials, activities and assessment were very useful, however there was not enough time to go through everything. The most negative part was that the link of the different materials and subjects was not clear. Although the points were explained it was not clear how it all fitted together, making it difficult to have a good understand of the subjects” were coded under ‘Alignment between materials and assessment’.
A total of 12 interviews were conducted with experienced academic staff who have had responsibility for leading the writing and delivery of modules. The staff sample was obtained by asking each of the six faculties to put forward two experienced module authors to be interviewed. The design of the project was classified as low-risk and small-scale which meant that no ethics approval was needed. The interviews focused on eliciting the design-related factors which the course chairs perceived as enhancing and inhibiting student retention on courses and qualifications using a process called cognitive mapping. Cognitive mapping focuses on developing a diagrammatic representation of the means and ends relationship for a particular domain, in this case supporting student retention. Cognitive maps were created for each interview life during the interview process, allowing participants to check and expand on the map as it was created. For instance, where a participant mentioned the factor “Getting the assessment strategy right” the interviewer would ask what contributes to getting the strategy right, to which the interviewee would add “Having a gradual assessment pathway.” In this way, each of the factors were unpacked and links added to reveal the connectivity between different statements. Based on these maps, key themes were derived for inclusion in the design principles.
Analysis of the data covered above followed an approach akin to grounded theory. Grounded theory (Glaser & Strauss, 1967) constitutes an inductive method for theory formation which is grounded in the context of a particular data set. Through progressive stages, categories of meaning are derived from the data set until theoretical saturation has been achieved (ibid). Each relevant statement was coded according to the underlying design-related aspect that it pertained to and the resulting codes were grouped according to thematic similarity into concepts. The resulting concepts were placed in an aggregate map with links between the concepts added. Seven clusters emerged within the aggregate map which were named Integrated, Collaborative, Engaging, Balanced, Economical, Reflective and Gradual. A model was then developed based around these clusters, with an emphasis on transforming them into practical advice for course designers.
The model for improving retention through design was given the acronym ICEBERG, based around the seven key clusters that emerged in the analysis. Although there is some overlap between these clusters, they represent distinct areas of practical advice.
A well-integrated curriculum constitutes a coherent whole where all the parts work together in a meaningful and cohesive way. This means that there is constructive alignment between learning outcomes, assessments, activities and support materials which all contribute effectively to helping students to pass the course.
Meaningful student collaboration and communication aids students in engaging in deep learning and making concepts and ideas their own (e.g., Garrison, Anderson, & Archer, 2001; Johnson & Johnson, 1999). It also serves as a mechanism for social support where students feel part of an active academic community of learners (see Tinto, 1975) which makes it more likely that they are retained. Where collaborative activities are well integrated with the study aims of a piece of curriculum and are effectively structured to aid the collaborative process, many students tend to enjoy opportunities to work together with others, although it is also important to recognise that some students find collaboration difficult and challenging.
An engaging curriculum draws students in and keeps them interested, challenged and enthusiastic about their learning journey. Where the curriculum matches student interests and aligns with their educational and career aspirations, students are more likely to be retained. Using relevant case studies and readings and keeping these up-to-date as well as including a variety of different types of activities contribute to an engaging curriculum.
Balanced in this context refers to the workload that students face when studying the curriculum and the extent that this workload is well-paced and evenly distributed. Research has pointed out a negative correlation between average weekly workload and student outcomes (Whitelock, Thorpe, & Galley, 2015) including satisfaction and pass rates, making it particularly important that students don’t feel overloaded whilst keeping the workload appropriate for the level of study. Also, an unevenly distributed workload where there is significant variance in the workload between different study weeks has been found to have a negative impact on student outcomes (van Ameijde, 2014).
Economical refers to the extent to which a course or qualification is efficient in delivering the learning outcomes without providing too much additional material which is not key to achieving the defined learning outcomes.
There is often a temptation to provide students with an abundance of interesting facts, ideas, theories and concepts in a given subject area. However, students are guided in their studies by what is relevant for their learning and career aims as well as what they are assessed on. Providing students with a clear critical path which delivers the required learning aims without unnecessary digressions will make it more likely for them to be retained. Making it clear to students exactly what they will learn and providing them with a clear set of learning activities and learning documents that enable them to achieve the required learning aims will contribute to students’ success.
For students to effectively pass a course and engage in deep learning, it is important that they are able to reflect on their learning and study progress and have the time and space to do so (Thorpe, 2004; Clegg, Tan, & Saeidi, 2002). Course materials that continually reflect back to the student what they are learning and how this learning helps them develop the knowledge and skills helps to place content in the broader context of their study. This includes regular opportunities for students to test their understanding through for instance self-assessment questions and formative quizzes and iCMAs. Such opportunities for reflection and feedback help keep students engaged with the curriculum and makes it more likely for them to be retained.
In an effective learning journey, students will gradually encounter increasingly complex and challenging concepts, ideas, materials, tasks and skills development. Where knowledge, skills and assessments all occur over a manageable gradient which builds on acquired knowledge, provides timely opportunities to learn and practice study skills and prepares them achieving the defined learning outcomes, it is more likely that students will not be overwhelmed and therefore more likely be retained.
The OU has a strategic approach to Learning Design, with an accepted methodology and standardised tools and representations. All new courses that are produced are required to undertake a learning design session, and a team of four learning designers run regular workshops for courses and faculties, in conjunction with media designers. Having this strategic approach to learning design in place provided an ideal route for implementing the ICEBERG model. In conjunction to the model a set of ‘10 tips to improve retention’ was devised, along with a standardised workshop with activities addressing retention. In conjunction with existing learning design suite of workshops and activities, this provided a means of promoting retention as an issue that merits a specific focus.
The model was developed in 2016, and course teams throughout the OU have started to use it as a framework to discuss and promote retention as a key factor in the design of new curriculum. This is usually in conjunction with other retention related initiatives, such as changes to the support model during presentation, and the use of learning analytics to aid intervention by tutors.
It should be noted that this model has been limited in scope, prioritising retention, learning design and distance education. Although the model presented here was developed in a distance-education context and focusing predominantly on retention, the resulting principles are to a large degree relevant to a wider education context and speak to issues of student engagement and satisfaction as well as retention. For instance, a reflective curriculum which provides learners with regular opportunities for meaningful reflection is as relevant in a traditional face-to-face context as it is in a distance education context.
There are many perspectives to consider when designing a course, and retention is just one of these. What may benefit retention may have negative consequences from other perspectives, for instance retention improves when students are charged a fee to study, since this makes switching or dropping courses an expensive option. It might be that for society more broadly, free higher education is more desirable, or for an individual student, discovering that a particular course is not suited for them then non-continuation is their best option. Similarly, retention is only one aspect that should be considered by course designers, and should not be at the expense of addressing complex topics, or implementing challenging pedagogy. However, it is the authors’ contention that retention is rarely given sufficient attention as a design principle in its own right, and it is a matter of increasing significance to students, educators, institutions and society. The proposed model then is a means of considering any course from the perspective of retention.
There are a variety of ways in which interested readers could apply the ICEBERG model in their own practice. For instance, educators could use the factors as a basis for a benchmarking or self-assessment framework where a piece of curriculum can be scored on each of the seven factors and improvements noted to enhance the degree to which each factor is exemplified. Another way in which the model can be used is by using the factors as a basis for a feedback mechanism for students to feed back on their curriculum as a means to inform improvements.
The complex inter-related nature of retention makes it difficult to isolate one factor, so although designing for retention can have an impact, this could be diminished by changes elsewhere, such as the support model, or external climate. It also has a long time frame to investigate impact, particularly with the OU module production cycle where courses can take two years to develop, and then several years of presentation data would be required to ascertain its effectiveness. However, given the significance of retention both to the learner and the institution, foregrounding it during the design phase is an important step in addressing this complex issue.
Jitse van Ameijde leads the Learning Design team within the Open University, whilst being a part time Associate Lecturer in Systems Thinking. He has a broad interest in how students learn and how to design effective pedagogies which students experience as engaging and meaningful.
Martin Weller is Professor of Educational Technology at the Open University. He was Academic Director of the Learning Design project, and Director of the OER Hub. His research interests are in digital scholarship, open education and learning design. He blogs at edtechie.net
Simon Cross is a Lecturer at the Open University and was involved in early Learning Design initiatives across the University. Current research interests include the Learning Design of assessment and student choice, MOOCs and open learning in India, VR and Learning Gains.