Background

In 2019 I was contacted by a colleague in the Student Disability Service (SDS) regarding how best to support a profoundly deaf Informatics student with live captions. They wished to employ the services of AI Media to provide the captions, but needed some help in how best to support this with the technology available. At the time, they had provided the student with a wireless mic. The student was required to hand the mic to the lecturer at the start of each lecture, and the receiver would be plugged in to the student’s laptop. From there, the student would initiate a Zoom call with AI Media and receive a caption track in response by accessing a given web link.

Results for this approach were patchy. Not only was it undesirable to make the student hand over the mic at the start of each lecture, and sit within a suitable range to receive the audio signal, but there were issues with the mic receiver having to be directed via various adaptors for the student’s laptop which had one USB-C port only.

An alternative option was to share access with AI Media (via a functional account) to the Media Hopper Replay live stream. However, the lag between the lecturer speaking and the audio being received by the captioner could be anywhere in the region of 10-30 seconds. When you add on the small lag in the captioner repeating the words to create the caption track (c. 5 seconds) this meant that the student was receiving captions at a significant enough delay to make affect their understanding of the material being discussed.

Online teaching

When the Covid 19 global pandemic closed campuses across the world, and the University of Edinburgh pivoted to online teaching overnight, all students began to access their lectures via a variety of online streaming services the University had previously procured.

Media Hopper Replay (Echo360) could still be used to record and deliver online lectures via the personal capture application lecturers could install on their local machine. Media Hopper Replay can provide an automated transcript for corresponding video / audio assets. However, without the affordances of the built-in timetable scheduler many lecturers opted for a service pushed by Central as the default video streaming service – Media Hopper Create.

Media Hopper Create (Kaltura) – often referred to as the University’s Youtube – can provide automated captions for corresponding video / audio assets. As with Media Hopper Replay, and MS Stream (see below) a user with appropriate privileges can manually correct the automated captions. However, this work is non-trivial and can take anything in the region of a ratio of 1:3 to 1:15 to perform.

The University of Edinburgh also widened access to MS Teams during lockdown. Lectures / meetings could be recorded and published to MS Stream. Anecdotal evidence from our students at this time reported that the automated transcripts produced by Stream were the most accurate.

Return to on-campus teaching

After the hybrid approach to teaching for 2021/22 academic session, we knew that all teaching would return to on-campus for the 2022/23 session. With this in mind, I was back in touch with the SDS to try and iterating and improve on our pre-pandemic practice. I approached ISG Learning and Teaching Spaces to leverage their technical expertise regarding how centrally supported teaching spaces are equipped, to trial a new approach.

Aim:

• One single mic for the lecturer to wear
• Excellent quality audio for the captioner – with minimal lag
• minimal additional work for the lecturer

After considering the standard AV set up in the teaching spaces, the optimal solution presented as follows:

• lecturer launches Zoom call (as co-host) from the teaching PC (this receives audio input from the same microphone lecturer uses for both in-room amplification and for the lecture recording service)
• AI Media receive audio by logging in, as participant, to Zoom call
• Student receives close to real-time captions by logging in to previously shared AI Media URL

What happened?

I asked the relevant student support officer to share with me the student’s timetable and list of course organisers for those classes for which the student was enrolled. I then scheduled one recurring Zoom session and added all course organisers as co-hosts. Course organisers were emailed outlining what was required of them – including notifying us of any additional lecturers who we may need to include.

The first two weeks resulted in a slightly patchy service (mostly down to incorrect course organisers being recorded) however,  when it worked, it worked well. Week 3 to week 6 saw much more consistency and the student reported excellent captions.

The main challenges (from my perspective) are not technical, but human and organisational:

• the lecturer has an additional piece of work to conduct (launch the Zoom session)  at the start of the lecture when they are already time pressured*
• the lecturer must remember (as always) to ensure the mic is switched on. Again – a visual indicator on the desk for this would be most welcome
• the lecture must remember to end the Zoom call at the end of the lecture. This could be avoided if – rather than creating a single recurring Zoom session – individual sessions were created for each course and and / or each class. However, this would create additional work and potential confusion for AI Media.

there is no one single point of contact for co-ordinating everything. The main stakeholders are:

• Student
• Student Disability Service
• Lecturers
• Learning Technologists
• Student Support Officers

The work to be co-ordinated involves:

• confirming the timetable
• communicating with course organisers and other members of the teaching team
• communicating with AI Media
• communicating with the student.

This often leads to duplication of effort and vital communications going missing.

* this can be somewhat shortened by creating a shortcut on the PC account desktop, which references the Zoom session. When selected it launches Zoom and connects to that meeting.

What next?

I have asked the Informatics student support team to contact the student to request feedback on their experience of live captioning this semester. We will use that feedback to inform our practices going forward. I also want to share our experience with the wider University community to see if and where we can improve.

Additional Resources

Accessibility and Lecture Recording

Updates to existing lists

Please review and update your existing lists for Semester 2 and send them to the Library for review by Monday 24 October using the ‘Request Library review’ button at the top of the list.

New Resource Lists

If you would like the Library to set up a new Resource List for your course, please provide the information requested on the online form and submit it by Monday 24 October.

Prioritise list citations

Remember to prioritise items on your lists using ‘Essential’, ‘Recommended’ and ‘Further reading’. The priority tags and student numbers inform the action the Library will take. You may also identify one ‘key text’ per course for which the Library will take additional steps to provide e-book access. Guides and short videos are available to help you update existing list and set up new ones.

List visibility

I would like to remind you that Resource Lists can be accessed without a University login via http://resourcelists.ed.ac.uk/ Subscription resources (e-books, e-journal articles etc), any uploaded PDFs and scans provided by the Library can only be accessed by staff and students. If you would like to restrict access to your list to staff and students only, please use the ‘Library Discussion’ on the list to add a note before you send your list for review or contact Library.Learning@ed.ac.uk If you plan on using the online form to submit a new list, there’s an check box you can use to let us know your visibility preference.

Help and support

Please contact Library.learning@ed.ac.uk if you have any questions or would like to arrange a Resource Lists training session.

With the new release of Noteable in August 2022, ISG have configured a new way to connect your Learn course to Noteable –  Noteable LTI 1.3.

Step 2. Check the settings and then click on Submit.

Step 3. The Noteable link will appear in the Learn course page. Click on it, it will launch on a new tab.

Background

Pre pandemic, supporting STEM teaching and assessment at scale had become a strategic focus for the College of Science and Engineering at the University of Edinburgh. However, as the world went into lockdown in Spring 2020, the University was forced to move at a speed not usually seen in such large institutions. Thankfully, a successful pilot of Gradescope by the School of Informatics meant we were able to identify an immediate solution to moving our assessment online.

Before bringing a third party tool in to service, a number of checks and processes are required. A Data Protection Impact Assessment (DPIA) has to be carried out. This has to be run past Legal Services and Information Security. In addition, an accessibility assessment needs to be carried out. As The Smiths once wrote, these things take time. Thankfully, as due diligence had been completed prior to the pilot in Autumn 2019, we were in a position to adopt a new service, and new ways of working, very quickly.

As we already had a relationship with Turnitin (who had recently bought Gradescope), expanding the contract was relatively smooth. The College could look at this as an opportunity to include a wider group of academics and professional service colleagues — from whom to gather feedback. We already knew from feedback from markers — in particular moderators — that marking at scale in Blackboard Learn was extremely cumbersome — and, at times, unworkable.

A hiccup

And so, we seemed primed to roll out Gradescope across the School of Informatics, as well as a selection of other Schools within the College. However, there was a bump in the road. Changes to the Safe Harbour rules meant we had to revisit the DPIA and work with Legal again to ensure we could continue to use a third party service based in the US, without compromising the privacy of our students’ data. All of this had to be completed by Schools, which were facing unprecedented workloads caused by the pandemic. Thankfully, we were able to negotiate a clause which satisfied Legal Services, and allowed us to continue to use Gradescope.

December 2020 Exams

We rolled out Gradescope as the default assessment platform for our exams in the December 2020 diet. The Homework Assignment assessment type was the preferred option, as we were keenly aware that many of our students were living under lockdown rules and may not have exclusive access to a device for the entire exam window. We wanted to ensure students could sit and submit their exams with the least amount of tech possible. They would need internet access at the start of the exam, to allow them to download their exam paper, and then again at the end of the exam, to allow them to upload their answers. A smart phone would be the most likely device they would have access to that would allow them to do this and, using one, they could easily scan their answers and upload them to Gradescope.

Students were able to access their exam papers and submit their answers without issue. Markers could enter their marks in an intuitive interface. Marks could be accessed without issue by the teaching office. The only thing that was missing was a decent anonymity feature. We worked around this by changing the names within the Roster from <Jane Doe> to <Anonymous User 1>. But this was an additional administrative burden we did not want to keep supporting.  We took advantage of our close working relationship with the Gradescope team and requested a proper anonymity feature. Within a few months, it had been rolled out and worked pretty much exactly as requested. Anonymity can be toggled on and off with ease — with a consistent anonymous ID applied to each submission. Blackboard and Turnitin — please take note!

2021 Exams

The next exam diet to support was Spring 2021. We also widened the assessment types available to our academic colleagues. After a successful pilot of the Online Assignment by two academic colleagues, this was now seen as a potential format for both our online exams and coursework. Markers were becoming more and more became familiar with the Gradescope marking interface, and students could have confidence that their submissions had been received.

During this period, Victoria Dishon, Head of IT for the College of Science and Engineering, had also recruited a project manager to evaluate the implementation of Gradescope across the College. The report can be found here (link accessible to UoE staff only). In short:

• 65% colleagues in Informatics reported spending less time marking each paper, with 11% reporting an increase. This was principally down to a couple of courses wishing to mark vertically.
• 65% colleagues in Informatics reported spending less time on the administrative aspects of marking too.

2022 Hybrid Exams preparation

With Gradescope now the default platform for online exams, we were keen to keep the affordances of marking horizontally online, but to trial this with a return to students sitting invigilated exams on campus. This was primarily driven by concerns around instances of academic misconduct in online exams, but with no appetite to introduce online proctoring. To this end, we applied to College to request two exams be sat this way in the Spring 2022 diet (Probabilistic Modelling and Reasoning &  Operating Systems).

Once College approved this, we began to work with the course organisers and teaching office colleagues to identify potential issues with this approach. To avoid confusion in the exam hall itself, the exam question papers were printed separate to the answer booklets. Personalised answer booklets were printed with the student name and exam number on the front cover (the name was included to make it easier for exam invigilators), and the exam number in the header of each subsequent page. The name would be blacked out with a sharpie prior to scanning for upload to Gradescope. The answer booklets were bound with a staple in the top left corner. It is this corner which would be cut prior to scanning. Stickers for identifying information were discounted as there were concerns this would jam the scanner. The exam answer booklets needed to have sufficient space for students to write their answers and some spare pages, as the template in Gradescope could not cope with pages being uploaded that it didn’t recognise. The reverse of each page of the answer booklet had a large note saying “Do not write on this page; Anything written on this page will not be scanned for marking” to ensure that students did not write any part of their answers on the back. This meant that the answer booklets could be scanned single-sided, which would help cut down on jams. Colleagues scrunched up, folded, and spilt water on sample answer booklets — anything we could think might happen in the exam hall — to help us prepare for any issues with the scanning process.

Gradescope was set up as a Homework / Problem Set type assignment with the option for the Instructor, rather than the Student, to upload the submissions. Then the answer booklet template was uploaded and used to create the assignment outline, using the exam number space at the top of the first page for their answers as the “ID Region”. A student roster was manually uploaded to Gradescope with the exam number as the Student ID. This meant that Gradescope should automatically be able to match the exam number indicated in the assignment outline to the correct student once the answers were uploaded.

Hybrid exams — what happened?

The student experience of sitting the hybrid exams has been straightforward. Sufficient blank pages were left in the answer booklets which meant that no additional pages were required. The Course Organisers were positive about the experience, as it meant they could continue to use Gradescope for marking, but also the way Gradescope was set up — with a template to auto-match the student submissions to each question on the exam — meant that there were no problems with mis-tagged questions, as there had been when students uploaded the exam themself.

The Teaching Office came up with a clear process for preparing the answer booklets (crossing out the name, cutting off the staple, etc.) and scanning them before uploading them to Gradescope. The large, general use Xerox printer / scanners were used for scanning; these send the scanned document through as a PDF via email.

There were mechanical problems, which meant that the scanning process ended up taking a lot more Teaching Office staff time than anticipated. The machines jammed which made the scanning process slow. Also, the lack of proper paper cutting implements meant that it was very difficult to cut through large page count booklets and left staff with blisters. Ideally, dozens of answer booklets could be scanned together into one document and then uploaded to Gradescope. Using the student exam numbers and the assignment outline, Gradescope would then be able to split all answer booklets contained into the file into individual submissions. However, due to jamming issues, only two answer books were able to be scanned together. This meant more time spent waiting over the scanner and more time selecting and uploading files to Gradescope.

What did we do?

These were not the biggest courses in the School and were only two of the 32 exams run in the School during the Spring 2022 diet. While the process for hybrid exams is now in place and does not need to be drastically changed for future diets, the Teaching Office staff time required to get the answers ready for marking was unsustainable.

Considering both the student experience, and the markers’ experience remained positive, and the frustrations experienced by the teaching office staff were mechanical related, we took the decision to invest in a dedicated scanner. Able to scan c150 pages / minute, we bought a Kodak i4000 series machine in time for the December diet.

As of the time of writing (August 2023) we have now run two full successful exam diets using the approach outlined above. Not only is this the experience of the School of Informatics, but many Schools across the College of Science and Engineering have reported similar results. Auto matching a script to a user has also dramatically increased in accuracy, after Gradescope responded to our feature request to hide the student name field to markers. This meant we could include two unique identifiers per script, with only one (non identifiable to the marker) being visible to the marker.

The annual Resource Lists rollover is scheduled for w/b 6th June 2022. Below is some information about the rollover process.

Rollover
On 6th June we will start our annual Resource Lists rollover when we will copy lists used in 2021/22 and create a new version for use next academic year (2022/23). Lists from 2020/21 will also be rolled over if the corresponding course is delivered in 2022/23 but didn’t run this year.

Action required
No action is required from you until the rollover is complete. However, if you would like to start working on a Resource List for 2022/23 before 6th June, please contact Library.Learning@ed.ac.uk and they’ll set you up with a new version of your list to edit before the rollover is complete.

Change freeze
Please do not edit your 2020/21 list after 5pm on 3rd June as any changes made after this time may not be rolled over.

What will happen after rollover?
Your new 2022/23 Resource List will be available to review and edit via http://resourcelists.ed.ac.uk and from the new course instance in Learn.

During week beginning 6th June, the current 2021/22 version of your Resource List will become read only. Students and staff will still be able to access previous years’ lists either via the corresponding year’s course in Learn or by searching on the Resource Lists homepage: http://resourcelists.ed.ac.uk.

Next steps
An email will be sent w/b 6th June to let you know rollover has completed as expected and provide more information on updating and setting up new lists for academic year 2022/23.

In the meantime, you’ll find more information on using Resource Lists to request materials for teaching in 2022/23 on the Resource Lists webpage.

The Resource List deadlines are:

• Semester 1: Monday 4 July 2022
• Semester 2: Monday 24 October 2022
• Term 3: Monday 16 Jan 2023

If you have any questions or would like to arrange a training session or 1-2-1 via Teams, please get in touch with Library.Learning@ed.ac.uk.

• Digital Communication, Collaboration and Participation
• Digital Proficiency and Productivity
• Digital Creation, Problem Solving and Innovation
• Digital Learning, Development and Teaching
• Information, Data and Media Literacy
• Digital Identity and Wellbeing

The Informatics Teaching Festival is back for a second consecutive year. 

 Join us for the opportunity to:

• hear interesting presentations around lessons learned in the past year and good practice in online/hybrid teaching from colleagues and inspiring speakers from other schools
• listen to the feedback provided by student representatives regarding their experience with studying in an online/hybrid context
• listen to the feedback provided by teaching support and administrative staff as to their experience with teaching and administration this past year
• learn about new approaches to teaching and educational software
• share your own experience with teaching delivery, student support and course administration during workshops and informal GatherTown meetings
• reflect and come up with ideas together for improving our delivery of online and hybrid teaching, both as a school and in our different courses.

If you’d like to attend any of the following sessions, and are not a member of the School of Informatics, please register your interest here, and a Collaborate link will be emailed to you in advance of the session(s).

Schedule

There are several examples in Informatics where a course is delivered to both an ‘on campus’ cohort and a ‘distance learning’ cohort. These terms can be nebulous, however, you will be aware of the courses being discrete within EUCLID. The on campus course will have a course code beginning INFR with the DL course beginning with INFD.

If you are a course organiser for such a course, please read on.

Each discrete course within EUCLID has a corresponding instance in Learn. If you are delivering the same course to both cohorts it makes sense to create a Shared Course Arrangement. This will feed the enrolments from one cohort (‘the child course’) to another (‘the parent’). You can then hide the child course within Learn to avoid potential confusion.

Further information on how to create this shared course arrangement can be found here:

https://www.ed.ac.uk/information-services/learning-technology/virtual-environments/learn/instructor/building-managing-content/shared-courses 

If you would like to discuss this in relation to your own course, please get in touch.

There are many times when you may need to check submission dates and times for coursework or exam hand-ins. The way you do so will depending on the mechanism used to submit.

The following instructions are for when a student has submitted via the Gradescope link within Learn.

Go to the course in Learn and find and select the link to Gradescope.

Select the assignment name within Gradescope to open. Select Review Grades.

Sort by the Time column header to see which submissions were made after the deadline.

Associated links

Reporting on submission times in Learn

This blog post is aimed at all students sitting an exam or submitting coursework using Gradescope Homework assignment.

Submitting your PDF to Gradescope

• Navigate to the appropriate area in your Learn course (this will be the “Exam” content area for taking an exam, or the “Assessment” area if submitting coursework)
• Open Gradescope by clicking “Submit via Gradescope”
  • Gradescope will open in a new tab
  • You will be taken directly to the corresponding course area in Gradescope
• Open the submission area for the question
• Select the file you wish to upload for your answer – this can be uploaded directly from your device. Upload your file – note that Gradescope does not provide a progress bar, and so it will look like nothing is happening while your file submits. Please be patient and do no click Back or Submit again while you wait.
• You will be shown a preview of your submission. You can rearrange pages if required.
• You will be required to tag which pages from your document correspond to the question part being answered. If your answer to a particular section spans multiple pages please tag each corresponding page. Please also make sure you have labelled each page with a note of which question you are answering. This tagging process takes place after the submission and can be done after the submission deadline without affecting your timestamp for submitting your response.
• Check all pages have been tagged correctly, and confirm your submission by clicking Submit.

Video Demonstration of the PDF Upload & tag process