Many of you will be aware of our FLORA project (if you aren’t, find out more on Melissa’s helpful FLORA blog post). Named after Flora Stevenson, a pioneer of education for girls and women but also a quirky acronym – Feedback, Learning, Online Rubrics and Assessment. FLORA was really our chance to review and reflect, post-COVID, on where we were as an institution with digital exams. Although FLORA has been paused for now, I wanted to reflect on what she gave us.
Anyone who knows me will know I’m not a fan of exams as a means of assessment. I think they can test memory more than application and I remember for my university exams studying profusely for them (especially in my third year when I had 3 x 3 hour exams in 2 days) and then my brain promptly emptied after them. But my feeling is that given we aren’t likely to stop doing exams any time soon, if we are doing them, we need to do them well.
FLORA followed an earlier during-COVID procurement project to buy a digital exams platform in 2020/21. In the end, we didn’t procure anything but probably no-bad thing given how much the exam landscape has changed since then. Who could have guessed GenAI would come along and rock the foundations of assessment in 2022……
What did FLORA do?
FLORA gave us the chance to pull together some interesting data and to reflect on digital exam use at the University. We spoke to students about their thoughts and experiences of exams (digital and on paper), we considered the impact of Gen AI on exams, we looked at how digital exams were done across the institution. We spoke to other institutions about what they were doing about digital exams.
What did we find out?
We conformed lots of unsurprising things. But we did get some surprises.
We confirmed:
- Exams are on the rise in general.
- The demand for digital exams increases but much of the demand is for in-person exams rather than open-book exams (due to the impact of GenAI on assessment). In-person digital exams have particular space requirements, we don’t have a lot of that sort of space on campus.
- Staff often preferred digital exams because it made submissions easier to read.
The surprises:
- We don’t clearly know what counts as an ‘exam’ and not all ‘exams’ are scheduled and run through our Exams Office. This may be because not all exams happen during the standard exam diet. Even those that do aren’t always visible to the Exams Office – we aren’t sure why this happens (and students pointed out the problems it caused – such as timetabling clashes).
- Very few staff seem interested in online invigilation/proctoring. It seems that many have realised how much of a privacy invasion it is (amongst other things).
- Exam sittings with special arrangements are over 10% of sittings. The numbers have more than doubled since 2020/21. And numbers like that are hard to manage. This is especially a problem where arrangements require specialist kit such as language keyboards. Some exams still relied on USB sticks to move submissions between computers and teaching offices. USB sticks fail….
- We had 8 different platforms at Edinburgh being used for digital exams. And even more different processes/workflows for them. And this makes digital exams hard to support centrally and sometimes left support to single individuals or very small teams in Schools.
- Other universities are doing interesting things around digital exams including:
- Developing specific digital exam policies with different approaches to digital exams (ie all exams must be digital, none or specific percentages)
- Renting external locations and sometimes computer hardware to increase their scope to do digital exams during the exam diet.
- Virtual exam teams who meet to deal with any major issue impacting digital (or paper!) exams such as network outages/power cuts etc.
- Some have teams dedicated to supporting digital exams and/or digital assessment.
It has been a really interesting project, and we’ve learned a lot from it, as an institution. The Project Board were extremely helpful and constructive and I was amazed at how invested most of them were in our workshops and meetings.
FLORA’s legacy
FLORA is paused for now whilst we review and prioritise our plans for the future as an institution. However, we are currently writing it up to ensure none of the nuggets we collected during the project are lost. Some of the outcomes are being passed onto FLORA’s sister LOUISA to continue such as the terminology work to clarify definitions around assessment. We’ll publish more soon. In the meantime, keep an eye on these sites:
