Skip to content

Toggle service links
  1. eSTEeM
  2. Category
  3. Theme
  4. Other
Subscribe to RSS - Other


Refining a framework for measuring qualification effects

Project leader(s): 
Steven Self and Mark Slaymaker

This project follows on from our previous project1 which focused on a new pedagogical approach (Hall and Rapanotti, 2015) implemented in three post-graduate computing modules for a recently introduced Computing qualification (F66 MSc in Computing), where the students’ own professional context of practice, rather than fictitious case studies, is used to assess their understanding of and ability to apply what is taught in those modules, as well as to develop a wide range of research and employability skills as they progress through the qualification.

The aim of the previous project was a preliminary evaluation of the pedagogical approach and the definition of a framework which could be used to evaluate its effectiveness within the qualification, with particular attention to cumulative effects along different pathways students may take, and culminating in the capstone research project module, where skills acquired through this type of pedagogical approach were assumed to be particularly relevant.

The previous project was successful in achieving the following outcomes:

  1. identify a selection of relevant quantitative and qualitative data, including specific student journeys as representative case studies
  2. analyse the available quantitative data for cumulative qualification effects
  3. quantify the workload required by module team and tutors
  4. define proof-of-concept semi-automated techniques for the extraction of relevant qualitative data from unstructured data sets, to facilitate follow-up manual analysis
  5. apply the techniques to specific data sets extracted from module student fora
  6. trigger pedagogical adjustments within the modules under study aimed at improving student performance and retention
  7. disseminate the research via a school seminar

Moreover, we are in the process of writing the final project report and submitting an abstract to the 6th eSTEeM Annual Conference.

The combination of techniques we have developed and applied to identify, extract and analyse data for this project can be seen as contributing to an overall evaluation framework, which we aim to refine and put to test in this follow-up project, whose specific objectives include:

  1. incorporate new quantitative data from more recent module presentations, and data from other modules used as benchmarks
  2. expand the range of qualitative data sets with student assignments, including aggregating data for specific student journeys through the qualification, to be used as representative case studies
  3. customize the classification and selection scheme used by the semi-automated techniques to the range of research and employability skills under consideration
  4. apply the customized framework to the new data sets
  5. establish the wider applicability of the framework

Outcomes will provide an evidence-based framework with direct benefits to:

  • the teams of the modules under study, for further development and fine-tuning of the pedagogical approach;
  • other module teams wanting to implement or capitalise on the pedagogical approach; or to perform similar analysis on different pedagogy;
  • qualification leads, as part of wider qualifications assessments.

The aim of both the previous and this project is to develop tools and processes that enable us to evaluate this new pedagogical approach (and other pedagogical approaches) by looking at the impact on student success – student performance, retention, progression and acquisition of transferable skills.

1 “Measuring qualification effects of a new pedagogy which embeds learning and assessment activities within each student’s rich professional context of practice”.

Hall, J.G. and Rapanotti, L. (2015), Masters-level software engineering education and the enriched student context, in 'Software Engineering (ICSE), 2015 IEEE/ACM 37th IEEE International Conference on', Vol. 2, IEEE, pp. 311–314.


Engineering qualifications at the OU – what motivates women to study?

Project leader(s): 
Carol Morris and Sally Organ

A project was carried out aimed at gaining a better understanding of the motivations and career aspirations of mature women studying engineering qualifications at the OU and to find out more about how their study experience differs from that of their male counterparts.

Key findings:

  • The majority of OU engineering students are working, with men more likely to be working full time and significantly more likely (58% men vs 19% women) to be already working in an engineering role; although high, the proportion of men currently working in engineering is lower than has traditionally been the case and suggests a change in the demographic of these students.
  • Significantly more women than men (44% women vs 17% men) have never worked in engineering but want to enter the profession. As a consequence, men were significantly more likely to be studying to progress their current career, while women are more likely to be interested in changing career direction or entering the engineering profession. Women were also more motivated by studying something useful.
  • Many female engineering students are well qualified: 46% of the female students already had another degree, compared to just 16% of the men. Female students tend to be aiming higher in both terms of qualification aim and professional registration intention.
  • When choosing their qualification, female students were more likely than men to have used the OU website or prospectus, or to have spoken to an OU adviser. Men were more likely than women to have seen advertisements and were more likely to have had advice from others, including friends or family, work colleagues, employers and engineering institutions.
  • The male students were more likely to have been encouraged to do engineering by others. Several female students reported a lack of information about engineering as a career option both at school and afterwards, but they were more likely than the men to report a family connection with engineering and found networking opportunities very helpful.
  • The most common source of funding was a student loan, but a high proportion of students were paying the fees themselves and significantly more men than women received funding from their employer.
  • Students were happy with the subject content of the OU engineering qualifications, its level and its delivery. The Open University, and especially their tutors, were identified as being consistently supportive and flexible and the importance of support from family was also recognised.
  • Interests in sustainability and environmental engineering featured highly for both sets of students and are reasonably well catered for in the OU engineering curriculum. However, civil engineering, which emerged top of the list for women but was a much lower priority for men, is not currently offered as a specialism.
  • Whilst most students reported positive attitudes towards their study of engineering, there were instances of negativity from managers and work colleagues towards women. All of the interviewees with experience of working in engineering had seen, or been subject to, sexist attitudes and behaviour. Male students interviewed were aware of the gender imbalance in most workplaces and of the sexism faced by their female colleagues.
  • Although a higher proportion of women strongly agreed that they were confident to succeed on their qualification compared to the men (48% women, 35% men) the overall confidence levels (strongly agree + agree) are higher for men (84% women, 94% men), possibly related to the educational background of the women. A similar but less pronounced effect was seen for confidence in mathematics. Men were more likely to believe they have a similar level of previous knowledge to others on the qualification’ (22% of women strongly agree or agree, compared to 47% men), but assumed prior knowledge in OU module materials was not considered a major issue.
  • Women were far more likely to be aware of being in a minority (64% strongly agree, or agree), mainly due to their gender, but a smaller number of men (18% overall) were also aware of being in a minority for reasons that included being good at maths, being a distance learner, not working in an engineering context, being non-British and being older.
  • The biggest challenge to studying for most students was the demands of their work schedule. The demands of home life and health issues were also mentioned. The most common requests for improving their OU study experience were for more flexibility in study patterns and more tutorials.

The project has led to a more detailed understanding of the backgrounds and prior educational experiences of female engineering students and the challenges they face, not only in their study but in their work and home lives. We have also gained an understanding of male students’ work experience, which is significantly different from that previously assumed. The knowledge gained from this project will inform future curriculum developments and current presentation practices.

Related resources

Morris, C. and Organ, S. (2019) Engineering qualifications at the OU – what motivates women to study? eSTEeM Final Report (PDF)

Morris, C and Organ, S (2018) Women in Engineering at the Open University: motivations and aspirations. Best poster winner at the 7th eSTEeM Annual Conference, 25-26 April 2018.


Factors affecting student success in the workplace

Project leader(s): 
Hilary MacQueen
  • We undertook a survey of graduates of a distance-taught work-based learning qualification (the Foundation Degree in Paramedic Sciences) to identify aspects of support required by students to succeed in this context. We were looking not only at pass rate and degree classification, but on less measurable factors contributing to the overall student learning experience.
  • Using a combination of quantitative, qualitative and open text data questions, we identified factors important for student success. Some of these factors were easily quantifiable (e.g. protected study time) but others were less easy to measure (e.g. students’ feelings of being supported).
  • The first part of our investigation involved collecting anonymised demographic data for all graduates of the Foundation Degree in Paramedic Sciences. In the second part of the project we invited all graduates (8 student cohorts) to participate in the survey. The number of respondents was small (n = 30; 8.9% response rate), and because our study was limited to one profession (Emergency care), our results may have limited relevance to other professions.
  • Dedicated study time was not a significant predictor of student success as assessed by pass rate, but was related to the quality of achievement as measured by pass grade. More important for a perceived successful experience were less tangible factors such as relationship with a mentor, and being part of a supportive peer group.
  • For maximum impact, the support measures we have identified should be overtly built into new course design. In particular, placements undertaken away from the primary workplace should be well organised, and students proactively supported while undertaking them.
  • The value of these findings is that they inform, and can be used to enhance, the experience of students on work-based learning programmes and also on degree apprenticeships.

Related resources

MacQueen, H. (2019) Factors affecting student success in the workplace. eSTEeM Final Report (PDF)

MacQueen, H. and Aiken, F.J. (2019) Supporting distance-taught students in the workplace, in Higher Education, Skills and Work-Based Learning, DOI 10.1108/HESWBL-04-2019-0048