Development and evaluation of a software tool for automated Java specification marking

• Project leader(s): Anton Dil
• Theme: Technologies for STEM learning
• Faculty: STEM
• Status: Archived
• Dates: October 2017 to October 2018

Highly Commended at the 2nd eSTEeM Scholarship Projects of the Year Awards 2019 under the category - Innovative/Original Approach to Teaching.

Students studying M250, our second year object-oriented programming module using Java, are required to complete Java programs according to detailed syntactical, structural, functional and stylistic specifications.

Although software tools exist for code syntax, functionality and style checking, tools for structural specification checking are not widely available. The long-term goal of this project is to raise awareness of these various aspects of correctness in our assessment of students’ code and to support automated assessment of these aspects of code quality for tutors and students.

The project focused particularly on the development and evaluation of a structural specification tool (known as CheckM250), deployed in the 2017J presentation of M250, to allow tutors to check to what extent students’ code met a specification. The tool was provided for use in the module IDE, BlueJ, alongside traditional tutor marking notes. The project also explored the use of automated marking in the module Virtual Learning Environment (VLE), for quick feedback to students, and overcame technical obstacles in this context. 

Tutor surveys and interviews were used to gather feedback on CheckM250 and on other kinds of marking tool support and traditional resources.

Automated structural checks on code were found to have multiple use cases:

• supporting human markers in assessing students’ code;
• for markers to assess their own marking (as a kind of e-monitoring tool);
• in online assessment for automated marking of students’ code;
• as a step in determining if software functional tests can proceed;
• for students to perform checks on their code before submitting it for marking;
• for question setters to check completeness of questions set for students. 

There was evidence of tutors favouring the use of marking tools, or of their distrusting them, or finding them an obstacle. This appeared to depend less on the tool itself than on a predisposition for or against the use of tools. Similarly, tutors’ comparative rating of tools as aids to themselves versus as aids to students appeared to depend on the tutors’ disposition towards tools.

Most tutors using CheckM250 found it to be useful, and some reported that it increased their accuracy in marking. Tutors not using the tool cited lack of time and the simplicity of the assignment it was trialled on. Some reservations were expressed about reliance on automated marking tools, both for markers and for students.  The marking software was also shown to be useful in the VLE for automated student feedback.

The results provided indicators of topics that should be discussed with tutors and students in this context:

• how automated code marking tools may best be used in tutor and student workflow;
• how the outputs of the tools should be interpreted;
• the potential benefits and pitfalls of automated marking;
• the relationships between the outputs of various automated marking tools.

The project has also suggested ways forward in developing automated marking tools for Java code. 

Related Resources: 

Attachment	Size
Dil, Truby and Osunde, Tool for Java. eSTEeM final report.pdf	438.24 KB

eSTEeM final report.

Attachment	Size
Dil, Truby and Osunde, Appendix A.pdf	532.59 KB

Appendix A - Java specification checking: software notes. 

Attachment	Size
Dil, Truby and Osunde, Appendix B.pdf	1.13 MB

Appendix B - Java specification checking: survey and interview results. 

Attachment	Size
Anton Dil.pptx	404.11 KB

Project presentation.

Attachment	Size
Dil, A., Osunde, J. (2018) ICETC Evaluation of a tool for Java structural specification checking.pdf	76.32 KB

Conference paper.