Cutting Corners: An Informatics Case Study

Background

Pre pandemic, supporting STEM teaching and assessment at scale had become a strategic focus for the College of Science and Engineering at the University of Edinburgh. However, as the world went into lockdown in Spring 2020, the University was forced to move at a speed not usually seen in such large institutions. Thankfully, a successful pilot of Gradescope by the School of Informatics meant we were able to identify an immediate solution to moving our assessment online.

Before bringing a third party tool in to service, a number of checks and processes are required. A Data Protection Impact Assessment (DPIA) has to be carried out. This has to be run past Legal Services and Information Security. In addition, an accessibility assessment needs to be carried out. As The Smiths once wrote, these things take time. Thankfully, as due diligence had been completed prior to the pilot in Autumn 2019, we were in a position to adopt a new service, and new ways of working, very quickly.

As we already had a relationship with Turnitin (who had recently bought Gradescope), expanding the contract was relatively smooth. The College could look at this as an opportunity to include a wider group of academics and professional service colleagues — from whom to gather feedback. We already knew from feedback from markers — in particular moderators — that marking at scale in Blackboard Learn was extremely cumbersome — and, at times, unworkable.

A hiccup

And so, we seemed primed to roll out Gradescope across the School of Informatics, as well as a selection of other Schools within the College. However, there was a bump in the road. Changes to the Safe Harbour rules meant we had to revisit the DPIA and work with Legal again to ensure we could continue to use a third party service based in the US, without compromising the privacy of our students’ data. All of this had to be completed by Schools, which were facing unprecedented workloads caused by the pandemic. Thankfully, we were able to negotiate a clause which satisfied Legal Services, and allowed us to continue to use Gradescope.

December 2020 Exams

We rolled out Gradescope as the default assessment platform for our exams in the December 2020 diet. The Homework Assignment assessment type was the preferred option, as we were keenly aware that many of our students were living under lockdown rules and may not have exclusive access to a device for the entire exam window. We wanted to ensure students could sit and submit their exams with the least amount of tech possible. They would need internet access at the start of the exam, to allow them to download their exam paper, and then again at the end of the exam, to allow them to upload their answers. A smart phone would be the most likely device they would have access to that would allow them to do this and, using one, they could easily scan their answers and upload them to Gradescope.

Students were able to access their exam papers and submit their answers without issue. Markers could enter their marks in an intuitive interface. Marks could be accessed without issue by the teaching office. The only thing that was missing was a decent anonymity feature. We worked around this by changing the names within the Roster from <Jane Doe> to <Anonymous User 1>. But this was an additional administrative burden we did not want to keep supporting.  We took advantage of our close working relationship with the Gradescope team and requested a proper anonymity feature. Within a few months, it had been rolled out and worked pretty much exactly as requested. Anonymity can be toggled on and off with ease — with a consistent anonymous ID applied to each submission. Blackboard and Turnitin — please take note!

2021 Exams

The next exam diet to support was Spring 2021. We also widened the assessment types available to our academic colleagues. After a successful pilot of the Online Assignment by two academic colleagues, this was now seen as a potential format for both our online exams and coursework. Markers were becoming more and more became familiar with the Gradescope marking interface, and students could have confidence that their submissions had been received.

During this period, Victoria Dishon, Head of IT for the College of Science and Engineering, had also recruited a project manager to evaluate the implementation of Gradescope across the College. The report can be found here (link accessible to UoE staff only). In short:

  • 65% colleagues in Informatics reported spending less time marking each paper, with 11% reporting an increase. This was principally down to a couple of courses wishing to mark vertically.
  • 65% colleagues in Informatics reported spending less time on the administrative aspects of marking too.

2022 Hybrid Exams preparation

With Gradescope now the default platform for online exams, we were keen to keep the affordances of marking horizontally online, but to trial this with a return to students sitting invigilated exams on campus. This was primarily driven by concerns around instances of academic misconduct in online exams, but with no appetite to introduce online proctoring. To this end, we applied to College to request two exams be sat this way in the Spring 2022 diet (Probabilistic Modelling and Reasoning &  Operating Systems).

Once College approved this, we began to work with the course organisers and teaching office colleagues to identify potential issues with this approach. To avoid confusion in the exam hall itself, the exam question papers were printed separate to the answer booklets. Personalised answer booklets were printed with the student name and exam number on the front cover (the name was included to make it easier for exam invigilators), and the exam number in the header of each subsequent page. The name would be blacked out with a sharpie prior to scanning for upload to Gradescope. The answer booklets were bound with a staple in the top left corner. It is this corner which would be cut prior to scanning. Stickers for identifying information were discounted as there were concerns this would jam the scanner. The exam answer booklets needed to have sufficient space for students to write their answers and some spare pages, as the template in Gradescope could not cope with pages being uploaded that it didn’t recognise. The reverse of each page of the answer booklet had a large note saying “Do not write on this page; Anything written on this page will not be scanned for marking” to ensure that students did not write any part of their answers on the back. This meant that the answer booklets could be scanned single-sided, which would help cut down on jams. Colleagues scrunched up, folded, and spilt water on sample answer booklets — anything we could think might happen in the exam hall — to help us prepare for any issues with the scanning process.

Gradescope was set up as a Homework / Problem Set type assignment with the option for the Instructor, rather than the Student, to upload the submissions. Then the answer booklet template was uploaded and used to create the assignment outline, using the exam number space at the top of the first page for their answers as the “ID Region”. A student roster was manually uploaded to Gradescope with the exam number as the Student ID. This meant that Gradescope should automatically be able to match the exam number indicated in the assignment outline to the correct student once the answers were uploaded.

Hybrid exams — what happened?

The student experience of sitting the hybrid exams seems to have been straightforward. Sufficient blank pages had been left in the answer booklets that there were no additional pages required, which would have been problematic with the assignment template in Gradescope. The Course Organisers were positive about the experience, as it meant they could continue to use Gradescope for marking, but also the way Gradescope was set up — with a template to auto-match the student submissions to each question on the exam — meant that there were no problems with mis-tagged questions, as there had been when students uploaded the exam themself.

The Teaching Office came up with a clear process for preparing the answer booklets (crossing out the name, cutting off the staple, etc.) and scanning them before uploading them to Gradescope. The large, general use Xerox printer / scanners were used for scanning; these send the scanned document through as a PDF via email.

There were both mechanical and technical problems, which meant that the whole process ended up taking a huge amount of Teaching Office staff time. The machines jammed and the scanning process itself was quite slow. Also, the lack of proper paper cutting implements meant that it was very difficult to cut through so many pages (one booklet had 29 pieces of paper) and left staff with blisters. Ideally, 10+ answer booklets could be scanned together into one document and this uploaded to Gradescope. Using the student exam numbers and the assignment outline, Gradescope would then be able to split all answer booklets contained into the file into individual submissions. However, due to jamming issues, only two answer books were able to be scanned together. This meant more time spent waiting over the scanner and more time selecting and uploading files to Gradescope. Each exam took days of time for multiple staff member to scan and upload to Gradescope.

What next?

These were not the biggest courses in the School and were only two of the 32 exams run in the School during the Spring 2022 diet. While the process for hybrid exams is now in place and does not need to be drastically changed for future diets, the Teaching Office staff time required to get the answers ready for marking is unsustainable, if this were to be rolled out across more or larger courses. Therefore, the issues discussed above with scanning will need to be addressed for this to happen.

One way to do this is to buy out time from the University’s Digital Library team to get them to do the scanning. However, this option would be limited again by the number and size of courses using the hybrid model, as it would be unfair and unfeasible to expect their team to take on scanning tens of thousands of pages in the very short time required to get exams ready for marking.

There are dedicated scanners, like those used by the Digital Library team, which are estimated to be able to scan 100 pages / minute. These types of machines are used to scan unusual documents and should be able to handle the A4 pages used in the exam answer booklets with less jamming, which means that someone could get on with other work while sitting next to the scanner, rather than having to dedicate all their time to the scanner, as happened this time. These would save the files to a selected folder, rather than sending them by email, which would make it easier to keep track of them. However, these scanners are expensive and also require regular upkeep. This would only be an viable option if a significant number of exams moved to the hybrid model or if it was a shared resource across the College.

With the current set-up and technology, it is not feasible to continue running the hybrid exams as they are, even just a couple per exam diet, as it requires far too much resource from Teaching Office staff, who are already stretched at that time of year. However, there is a significant appeal of hybrid exams as a way to ensure academic integrity, so it is likely that there will be continued interest in them. As mentioned above, markers do find that marking in Gradescope is quicker, so they will not want to return to marking by hand, if students are sitting their exam in an exam hall. Other possibilities are being considered, such as invigilated computing labs where students would type their answers into a document or use the Online Assignment in Gradescope as discussed above.

Share

Leave a Reply

Your email address will not be published.