Using UX research methods to inform online course design at the Royal (Dick) School of Veterinary Studies

In the Vet School’s Digital Education Unit we assist with the design, support and maintenance of a large portfolio of online postgraduate courses. With this project, we did user research to help us improve the design and usability of these courses in our virtual learning environment, Learn. We used a top tasks survey and interviews to learn more about the tasks our students need to complete and what their experience is like as they complete these tasks.

Top tasks survey

101 students completed the survey, casting a total of 510 votes

In November 2019, we sent out a top tasks survey to all postgraduate students studying online with the Royal (Dick) School of Veterinary Studies.

The main part of the survey contained a single instruction:

Select the 5 items that are most important to you when accessing course materials online.

We then presented a list of 39 tasks in random order.

We asked this question to help us understand what our online students want to do when they log in to Learn.

Top tasks is a research method pioneered by the user experience practitioner Gerry McGovern. You can read more about this method at:

Identifying Customer Top Tasks (Chapter 14 from Transform: A Rebel’s Guide for Digital Transformation)

Respondents said that their top tasks are accessing recorded lectures, and accessing reading lists / study resources

Our results aligned with Gerry McGovern’s observation that a small number of top tasks will typically draw a large proportion of the votes.

The table below shows that over a third of all votes cast went to the top 3 tasks. Over half of all votes cast went to the top 6 tasks on our list of 39.

Question: Select the 5 items that are most important to you when accessing course materials online Votes Votes (% of total) Cumulative vote share
1. Lecture with combined audio and slides 64 12.5% 12.5%
2. Reading lists / study resources 64 12.5% 25.1%
3. Discussion boards 52 10.2% 35.3%
4. Assignment feedback 33 6.5% 41.8%
5. Lecture notes / outlines 30 5.9% 47.6%
6. Lecture slides handout (PDF) 27 5.3% 52.9%
7. Assignment submission / upload 25 4.9% 57.8%
8. Assignment deadlines 24 4.7% 62.5%
9. Course syllabus and learning outcomes 21 4.1% 66.7%
10. Live session access 19 3.7% 70.4%
11. Exemplar coursework 17 3.3% 73.7%
12. Announcements (get updates from course) 16 3.1% 76.9%
13. Assessment marking schemes / rubric 14 2.7% 79.6%
14. Grades (access grades for courses) 13 2.5% 82.2%
15. Lecture audio (MP3) 12 2.4% 84.5%
16. Formative assessment 11 2.2% 86.7%
17. Quizzes or tests 9 1.8% 88.4%
18. Academic study skills development information 8 1.6% 90.0%
19. Additional video content (supplementary videos) 8 1.6% 91.6%
20. Lecture transcripts 7 1.4% 92.9%
21. Anticipated workload 7 1.4% 94.3%
22. Learning outcomes for lectures 6 1.2% 95.5%
23. Finance information 4 0.8% 96.3%
24. Course sign-up 3 0.6% 96.9%
25. Communication policy 2 0.4% 97.3%
26. Course evaluation questionnaires 2 0.4% 97.6%
27. Reflective tool 2 0.4% 98.0%
28. Provide mid-course feedback to course organisers 2 0.4% 98.4%
29. Turnitin originality score 2 0.4% 98.8%
30. Careers information 1 0.2% 99.0%
31. Conferences and workshops information 1 0.2% 99.2%
32. Help and support with using Learn 1 0.2% 99.4%
33. Staff contact details 1 0.2% 99.6%
34. Staff profiles 1 0.2% 99.8%
35. Plagiarism information 1 0.2% 100.0%
36. Extension request 0 0.0% 100.0%
37. Induction activities 0 0.0% 100.0%
38. Review own work declaration 0 0.0% 100.0%
39. Student rep and SSLC info and minutes 0 0.0% 100.0%

The graph below shows the full list of tasks and the number of votes received by each one. The two colours on each bar show the proportion of votes that came from students in their first year of postgraduate study, and the proportion that came from students further along in their studies.

View chart at a higher resolution (opens in a new tab)

Graph showing results of top tasks survey

What these results mean for us

We need to check that students are able to complete their top tasks without any fuss.

Links to lectures, reading lists and discussion boards should be easy to find, and any technical problems with these elements of a course should be fixed as a high priority.

Interviews

We interviewed 10 students to help us understand the context around the top tasks

After the top tasks survey, we did online interviews with students. Interviewees shared their screen where available. We asked about how the tasks fit in to a typical week of study, which tasks they find easy to complete, and which tasks are sometimes more frustrating than they should be.

As part of the interviews, we included short usability tests. In these tests, students were set a task on Learn: for instance, to check the deadline for an upcoming assessment. They attempted the task with minimal assistance while narrating their thoughts. This helped us to evaluate how easy these tasks were to complete, which in turn provided clues as to how we could improve the design of our courses.

The specific tasks used in the usability tests were as follows:

  1. Starting from the course welcome page, show me how you would watch a video lecture from the current week of your course.
  2. Show me how you would find out the deadline and word count for one of your assignments.
  3. What would you do if you needed to get an extension for that assignment?
  4. Show me how you would normally respond to a post on the discussion forum.

What we found out

  • We’re doing a lot of things right with our existing course template. The existing design of courses caters for many of our participants’ key tasks. Accessing lectures, submitting assignments and keeping up to date with discussion boards were generally reported as satisfactory experiences.
  • Students had problems interacting with course materials via mobile devices.
  • Students want to plan their study time for an upcoming week. When a new week’s materials are released, students reported that their first task would be to calculate how they would need to spend their study time for the upcoming week.
  • The order of items on a Learn page is important. When looking at a week’s materials, the order in which activities were listed influenced the order in which students thought they should be completed.
  • Some of our assessment instructions are confusing. In some cases, assessment task instructions were perceived as complex, and there seemed to be potential for more concise instructions with key information made more prominent.
  • My Grades can be a confusing interface within Learn. In particular, My Grades caused some confusion with the way it lists what has been submitted. It was also interesting to see that some of this confusion is a result of My Grades displaying Learn Assignments differently to Turnitin Assignments.

Some interesting student comments

On the Assessment section of their course:

The Assessment folder, probably was for me the worst. It seemed not to contain the information I was looking for in an easy way.

So when you click Assessment [on the left sidebar] what are you typically looking for?

You have an introduction, then you have the anonymity in marking part, which are ok. Then you go down to the formative assessment, and below that you have Assessment 1 and Assessment 2. And when I clicked on Assessment 1 for example, I was looking for a really clear indication of what mattered, what was going to be marked and how, right up on the front.

To me the brief should be front and centre. So I know what it is I have to do and how it will be marked. That should be the very first thing. 

On submitting via Turnitin:

Do you remember much about the process of submitting assignments and how that went?

Yeah, I haven’t had a problem with that. I think you have to agree to something don’t you. And then this just pops up. So no, I’ve always found it easy. I like that you get an email receipt, because a massive bit of me worries that it’s not going to be submitted all right. But I think it’s quite obvious when you have submitted it. And I like that you can go back and view it because after I submit it I do that and see that it’s all right, and that it’s the right document or whatever.

On accessing Learn on different devices:

What devices do you normally use to access your course materials?

It’s pretty much just my laptop. Very occasionally, I might use my phone if I was out on a work trip or something like that — where I didn’t have a laptop. But it’s nearly always my laptop.

And when you have used your mobile, what kind of things have you been trying to do on your phone?

It would usually be if there was something in the discussion board. Like if I was in the middle of group work for example, and I just wanted to check on that. Or maybe if I was just checking if marks has been put up for one of the assessments. But it would be very rare because it’s just a bit tricky — I wouldn’t try to access lecture recordings or things like that on there, because it’s a bit tricky to do.

On being unable to pause lectures in the Articulate Storyline player:

My big issue with all of it is pausing lectures! And I find that slightly simpler on the Chromebook. I have told the tutors, but they just say there’s nothing you can do.

Pausing lectures?

So a half-hour lecture might take me 2 hours to get through. There’s a lot of links in them, and I have to constantly pause it to write down things. And I would like it if it was like a YouTube video, or iPlayer, where you can just press the spacebar and pause it. It would make it so much easier. But what I have to do is, like I’m looking at a lecture, and then I’ve clicked a link in it, and then I want to pause something so I have to come back down here and press pause.

Maybe since we’ve got the screen share up you could show me.

It absolutely drives me mad. [Opens lecture] So for every slide, he’ll start talking, and then I’ll pause it, and then I’ll write something down, and then I’ll play again. But it might be that there’s links to click, so I click away here [browser opens new tab], and read this, and then I come back and I’m just constantly moving my mouse to this bit here [play/pause icon]. If I could just use my spacebar, it would make my life a lot easier.

One student talking about a typical week of study:

Talk me through your week from Friday as far as your studies on this course go.

Ok, So I would — usually I go immediately to the lectures and have a look at what there is. And then I go to the reading list. Oh by the way, if the reading list for that week were in that week, that would also help.

That’s technically tricky, unfortunately.

Yes, because it links to those online bits and pieces I guess. So I would look at the reading list, and I would try to assess how large of a job that is, to read all the essentials at first. And then I look at the nice-to-haves after. I scan than and I scan the reading, and if it looks really heavy, I make sure to sort it so that I can — I lose the ability to concentrate on, especially technical stuff. So we had one week with all the brain and the brain structures and the neurotransmitters in there. It was a nightmare because it’s been over 20 years since I studied that stuff. I didn’t remember anything. So you’re going through both a lecture cycle and a reading cycle where you have to look up every third word of the papers and the lectures. Which makes it really heavy. But you — at least what I do is read as much as I can before my brain shuts down on the reading front and then for lighter entertainment I usually switch to a lecture. So I keep a mix. And then, if I get hooked, if there’s something that’s a personal interest of mine, and there’s 3 lectures all in a row, I’ll listen to them all, one after another after another.

Another student talking about a typical week of study:

For the first half of the term, I generally managed to start the work on a Friday. So I don’t work Mondays and Fridays. So generally on a Friday I’d finish up the week before, and then I might be able to get started. Definitely work over Saturday and Sunday, possibly do 4 hours a day on Saturdays and Sundays? I always start with the lectures. So I’d go to the lectures and hopefully finish the lectures by Monday or Tuesday. Sometimes at the weekend I might do some reading in the evening rather than the lecture material. So normally by the Tuesday I’ve done the lectures for the week. I might have read a paper or two. And then we normally have a live session. […] So normally I’d try and get through — most of the content would be done by Tuesday. And then Friday morning I might finish up any papers and then do the discussion boards.

What we did next

We had several meetings within the Digital Education Unit to draw insights from the interview transcripts. In some cases, the interviewees had identified small technical challenges that required further investigation. In other cases, comments from students gave us ideas for further research, and we discussed other sources we could draw on to evaluate our students’ experience of their courses. But the most important outcome of the research was that it informed the development of a template for our courses.

The template takes the form of a Learn course that we can refer to when building a course. It acts as a style guide for online postgraduate courses at the Vet School, helping us to standardise the navigation and naming choices that are embedded in the courses on Learn.

What I would do differently

I took on the task of interviewing students myself, and later presented the transcripts to my colleagues. We analysed the transcripts collaboratively, and having lots of eyes on the data was useful for putting the findings into context. However, the transcripts were not as vivid as conducting the interviews themselves and hearing students describe their experience with Learn first-hand. Reading the transcripts does not give you the same feeling of what students care about and what they find frustrating. So the next time I do this type of research, I would like to recruit colleagues to help conduct the interviews.

The tasks used in the usability tests were perhaps easier than they should have been. It would have been interesting to put further challenges into the tasks, as this would have told us more about where the design of the courses was successful.

Acknowledgements

Many thanks to Duncan Stephen from the User Experience Service for invaluable advice and assistance with this project, and to the Principal’s Teaching Award Scheme, who provided funding for the project.

Thanks also to my colleagues in the Digital Education Unit for their help in analysing the interview data.

And finally, thanks to the students who responded to the survey, and the ten who kindly agreed to be interviewed for this project.

The Digital Education Unit

Principal’s Teaching Award Scheme

User Experience (UX) Service

Ethical approval

This project received approval from the R(D)SVS Human Ethical Review Committee HERC_368_19.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.