Background
Pre pandemic, supporting STEM teaching and assessment at scale had become a strategic focus for the College of Science and Engineering at the University of Edinburgh. However, as the world went into lockdown in Spring 2020, the University was forced to move at a speed not usually seen in such large institutions. Thankfully, a successful pilot of Gradescope by the School of Informatics meant we were able to identify an immediate solution to moving our assessment online.
Before bringing a third party tool in to service, a number of checks and processes are required. A Data Protection Impact Assessment (DPIA) has to be carried out. This has to be run past Legal Services and Information Security. In addition, an accessibility assessment needs to be carried out. As The Smiths once wrote, these things take time. Thankfully, as due diligence had been completed prior to the pilot in Autumn 2019, we were in a position to adopt a new service, and new ways of working, very quickly.
As we already had a relationship with Turnitin (who had recently bought Gradescope), expanding the contract was relatively smooth. The College could look at this as an opportunity to include a wider group of academics and professional service colleagues — from whom to gather feedback. We already knew from feedback from markers — in particular moderators — that marking at scale in Blackboard Learn was extremely cumbersome — and, at times, unworkable.
A hiccup
And so, we seemed primed to roll out Gradescope across the School of Informatics, as well as a selection of other Schools within the College. However, there was a bump in the road. Changes to the Safe Harbour rules meant we had to revisit the DPIA and work with Legal again to ensure we could continue to use a third party service based in the US, without compromising the privacy of our students’ data. All of this had to be completed by Schools, which were facing unprecedented workloads caused by the pandemic. Thankfully, we were able to negotiate a clause which satisfied Legal Services, and allowed us to continue to use Gradescope.
December 2020 Exams
We rolled out Gradescope as the default assessment platform for our exams in the December 2020 diet. The Homework Assignment assessment type was the preferred option, as we were keenly aware that many of our students were living under lockdown rules and may not have exclusive access to a device for the entire exam window. We wanted to ensure students could sit and submit their exams with the least amount of tech possible. They would need internet access at the start of the exam, to allow them to download their exam paper, and then again at the end of the exam, to allow them to upload their answers. A smart phone would be the most likely device they would have access to that would allow them to do this and, using one, they could easily scan their answers and upload them to Gradescope.
Students were able to access their exam papers and submit their answers without issue. Markers could enter their marks in an intuitive interface. Marks could be accessed without issue by the teaching office. The only thing that was missing was a decent anonymity feature. We worked around this by changing the names within the Roster from <Jane Doe> to <Anonymous User 1>. But this was an additional administrative burden we did not want to keep supporting. We took advantage of our close working relationship with the Gradescope team and requested a proper anonymity feature. Within a few months, it had been rolled out and worked pretty much exactly as requested. Anonymity can be toggled on and off with ease — with a consistent anonymous ID applied to each submission. Blackboard and Turnitin — please take note!
2021 Exams
The next exam diet to support was Spring 2021. We also widened the assessment types available to our academic colleagues. After a successful pilot of the Online Assignment by two academic colleagues, this was now seen as a potential format for both our online exams and coursework. Markers were becoming more and more became familiar with the Gradescope marking interface, and students could have confidence that their submissions had been received.
During this period, Victoria Dishon, Head of IT for the College of Science and Engineering, had also recruited a project manager to evaluate the implementation of Gradescope across the College. The report can be found here (link accessible to UoE staff only). In short:
- 65% colleagues in Informatics reported spending less time marking each paper, with 11% reporting an increase. This was principally down to a couple of courses wishing to mark vertically.
- 65% colleagues in Informatics reported spending less time on the administrative aspects of marking too.
2022 Hybrid Exams preparation
With Gradescope now the default platform for online exams, we were keen to keep the affordances of marking horizontally online, but to trial this with a return to students sitting invigilated exams on campus. This was primarily driven by concerns around instances of academic misconduct in online exams, but with no appetite to introduce online proctoring. To this end, we applied to College to request two exams be sat this way in the Spring 2022 diet (Probabilistic Modelling and Reasoning & Operating Systems).
Once College approved this, we began to work with the course organisers and teaching office colleagues to identify potential issues with this approach. To avoid confusion in the exam hall itself, the exam question papers were printed separate to the answer booklets. Personalised answer booklets were printed with the student name and exam number on the front cover (the name was included to make it easier for exam invigilators), and the exam number in the header of each subsequent page. The name would be blacked out with a sharpie prior to scanning for upload to Gradescope. The answer booklets were bound with a staple in the top left corner. It is this corner which would be cut prior to scanning. Stickers for identifying information were discounted as there were concerns this would jam the scanner. The exam answer booklets needed to have sufficient space for students to write their answers and some spare pages, as the template in Gradescope could not cope with pages being uploaded that it didn’t recognise. The reverse of each page of the answer booklet had a large note saying “Do not write on this page; Anything written on this page will not be scanned for marking” to ensure that students did not write any part of their answers on the back. This meant that the answer booklets could be scanned single-sided, which would help cut down on jams. Colleagues scrunched up, folded, and spilt water on sample answer booklets — anything we could think might happen in the exam hall — to help us prepare for any issues with the scanning process.
Gradescope was set up as a Homework / Problem Set type assignment with the option for the Instructor, rather than the Student, to upload the submissions. Then the answer booklet template was uploaded and used to create the assignment outline, using the exam number space at the top of the first page for their answers as the “ID Region”. A student roster was manually uploaded to Gradescope with the exam number as the Student ID. This meant that Gradescope should automatically be able to match the exam number indicated in the assignment outline to the correct student once the answers were uploaded.
Hybrid exams — what happened?
The student experience of sitting the hybrid exams has been straightforward. Sufficient blank pages were left in the answer booklets which meant that no additional pages were required. The Course Organisers were positive about the experience, as it meant they could continue to use Gradescope for marking, but also the way Gradescope was set up — with a template to auto-match the student submissions to each question on the exam — meant that there were no problems with mis-tagged questions, as there had been when students uploaded the exam themself.
The Teaching Office came up with a clear process for preparing the answer booklets (crossing out the name, cutting off the staple, etc.) and scanning them before uploading them to Gradescope. The large, general use Xerox printer / scanners were used for scanning; these send the scanned document through as a PDF via email.
There were mechanical problems, which meant that the scanning process ended up taking a lot more Teaching Office staff time than anticipated. The machines jammed which made the scanning process slow. Also, the lack of proper paper cutting implements meant that it was very difficult to cut through large page count booklets and left staff with blisters. Ideally, dozens of answer booklets could be scanned together into one document and then uploaded to Gradescope. Using the student exam numbers and the assignment outline, Gradescope would then be able to split all answer booklets contained into the file into individual submissions. However, due to jamming issues, only two answer books were able to be scanned together. This meant more time spent waiting over the scanner and more time selecting and uploading files to Gradescope.
What did we do?
These were not the biggest courses in the School and were only two of the 32 exams run in the School during the Spring 2022 diet. While the process for hybrid exams is now in place and does not need to be drastically changed for future diets, the Teaching Office staff time required to get the answers ready for marking was unsustainable.
Considering both the student experience, and the markers’ experience remained positive, and the frustrations experienced by the teaching office staff were mechanical related, we took the decision to invest in a dedicated scanner. Able to scan c150 pages / minute, we bought a Kodak i4000 series machine in time for the December diet.
As of the time of writing (August 2023) we have now run two full successful exam diets using the approach outlined above. Not only is this the experience of the School of Informatics, but many Schools across the College of Science and Engineering have reported similar results. Auto matching a script to a user has also dramatically increased in accuracy, after Gradescope responded to our feature request to hide the student name field to markers. This meant we could include two unique identifiers per script, with only one (non identifiable to the marker) being visible to the marker.
