The topic of the Teaching Lunch held on 21^st of January focused on the aims, benefits and implementation of the School of Informatics Moderation Policy and process.

This topic was prompted by analysis of how we manage marking and moderation for many of our larger courses. We may employ multiple markers for examination and coursework marking, or we may use a mixture of auto marking and human intervention.  This may result in some inconsistency because of variation in the interpretation of marking schemes.

Stuart Anderson outlined that the goal of this lunch was to give a brief outline of our Moderation Policy and how it implements Taught Assessment Regulation 31, so that the School is compliant, consistent and effective. The moderation process must be appropriate to the discipline and is applied to examinations and coursework components weighted from 25%.  The School must conform to the Taught Assessment Regulations guidelines 1.31.1 to 1.31.8. The lunch was an opportunity for a conversational discussion on the way forward from Semester 2 2019/20.

We heard contributions from Iain Murray, Sharon Goldwater and Volker Seeker, outlining how they manage the marking and moderation processes for their courses, in order to best manage the number of Markers, to ameliorate inconsistencies and to ensure that the process is fully compliant with the policy.

Iain Murray outlined the process he uses in managing marking and moderation for the course Machine Learning and Pattern Recognition. This includes an advance pre marking meeting with Markers, to discuss and outline the marking guide, and set standard and specific marking instructions for questions/solutions. It was noted that MLPR coursework and exam papers have a specific structure which lends itself to a specific marking guide as opposed to, for example, an essay. The Course Organiser also checks a marking sample of 10. Moderation replicated the process with a further sample of 10 checked for consistency by a second nominee. This enabled identification and management of inconsistencies and Marker feedback at all stages. Iain also noted that communication with the students is key, including personalised feedback and information on the marking and moderation process, in order to manage student expectations.

Sharon Goldwater outlined that the course Accelerated Natural Language Processing has been managed along similar structures for the past 2 years, with short assessment deadlines and clear communication and feedback to students. The marking process includes a marking instructions guide, and clear guidance to Markers. The Course Organiser employs batch sampling at marking and moderation stages, thereby identifying and managing inconsistencies and giving constructive Marker feedback early in the process.

Volker Seeker discussed use of automarking for Extreme Computing, in addition to a detailed marking scheme set up in advance of the diet.  Running the automarker on a sample enabled him to identify deviations from expectation, and issues where code did not compile. Volker has been trialling Codegrade which includes interface with Blackboard, continuous feedback, mitigation of codes, and unit test functions including basic and advanced tests. Assignments have a 3 stage process: submission, feedback which includes marking of peer feedback and advice on 2 random anonymous submissions, and moderation management, which includes a marking guide and a Marker meeting.

Attendees then discussed the finer detail of the presentations and Q and A with the presenters.

Stuart Anderson and Alex Burford gave a brief overview of the Gradescope tool we have been piloting in Informatics.  Chris Williams presented the pros and cons from his experience of Gradescope, and took questions. Gradescope will also be covered in more detail at the next Teaching Lunch.

Stuart Anderson summarised that having a clear, robust and consistent moderation process is key, with clarification that moderation samples should be 10% or 25 papers maximum to reduce an onerous workload. It was clear that pre preparation is key, to include managing student expectations and feedback, a detailed marking guide, a Markers meeting in advance of marking, and rigorous sample moderation and feedback to Markers.

Stuart Anderson then invited Claudia Chirita to discuss the process of moderation at Royal Holloway University. This is a very rigorous, controlled and consistent process, including marking restricted to Course Organisers, and double marking restricted to Academic staff. However due to the increasing numbers of students, the process currently requires some updates to continue to be manageable and effective.

The Teaching Lunch was then brought to a close.

The next Teaching Lunch will be held on the 18^th of February, and the topics will be Online Learning and further presentation of Gradescope features and facilities.