The latest phase of Learn Ultra research investigated the staff and student experience, with a focus on Learn Ultra courses, to guide the next phase in the upgrade. 

A programme of multi-method user experience research sought to investigate ways in which Learn Ultra had enhanced staff and student experiences, and to identify areas where there was room for improvement in the ongoing upgrade. This research was completed between May and October 2022, and was a continuation of the Learn Foundations project approach.

Read more about the upgrade to Learn Ultra

Read more about the latest programme of user research

Read more about the user research with staff

Learn Ultra discovery research with students

The initial phase of the Learn Ultra upgrade involved implementing key navigational functionality and improving the overall look and feel of Learn. Students were engaged in sketching workshops, interviews and usability testing to understand what they needed and expected from the Learn Ultra navigation, and how they naturally interacted with it in the context of completing tasks. Findings from this research were used to steer decisions around the default landing page of Learn Ultra and other structural aspects.

Read more about this research and findings in the related blog posts:

Sketching workshops have provided insight into students’ needs and expectations for Learn Ultra

Understanding how students interact with Learn Ultra and Ultra Base Navigation

Objectives and methods for latest round of student research

The roll-out of the first phase of the Learn Ultra upgrade was completed in the academic year 2022 to 2023. Having established that further research was necessary to inform the next stage of the upgrade, research was planned with students to achieve the following:

• Test the assumption that Learn Ultra offers a better overall user experience
• Assess the usability of Learn Ultra on multiple devices and viewports
• Investigate how effectively Learn Ultra supports students completing high-priority tasks
• Validate design choices for aspects of the interface and development of the Blackboard Learn App
• Identify areas for improvement and adjustment ahead of the roll-out of Learn Ultra courses
• Shape the development of staff training and guidance for Learn Ultra in a way that aligned with students’ needs and expectations

To address these aims, a combination of online methods was used – usability tests, semi-structured interviews and structured questions. Since Learn Ultra courses were not in use at the time of the research (and therefore University students had not experienced using it in a ‘real life’ context with actual course data), a test environment with representative course content was used for the research. A total of 15 students were engaged in the research, one a recent graduate, 12 current students, and two student interns from Napier University (who were used to using the Moodle VLE and had never used Learn). Students were recruited from all three Colleges, and represented a range of year groups and programmes.

Analysis of the data collected was completed by members of the UX team working together with the Learn Ultra project team.

Findings from research with students

The research provided general and specific insights into the student experience of Learn Ultra, identifying positive aspects as well as areas to work on and improve in the ongoing implementation and roll-out.

Insights from usability tests and semi-structured interviews

In the online usability tests, students were asked to complete eight ‘top tasks’ and give each task an ‘Ease of Use’ score (rate it on a scale of one to 10 where 10 was extremely easy). Top tasks are the activities which students deem a priority to complete, taken from the results of the latest top tasks survey completed by students.

Read more about the Learn top tasks voted for by students in a related blog post:

A top tasks survey has shown what staff and students prioritise in hybrid teaching and learning

As they completed the tasks,  students shared their screens to demonstrate the actions taken. This provided context for the semi-structured interviews, to obtain information about students’ associated thought processes, expectations and perceptions.

Students found it easy to use Learn Ultra to complete top tasks

The top tasks they were asked to complete, and the associated average ‘Ease of Use’ scores were as follows:

• Find a lecture recording for a given week – score: 7.9/10
• Find a resource list – 8.0/10
• Find lecture notes for a given week – 9.0/10
• Find assignment deadlines – 9.6/10
• Find lecture slides for a given week – 9.3/10
• Join a live online lecture – 8.2/10
• Find feedback on an assignment – 7.5/10
• Check grades – 9.7/10

Some students were confused by some aspects of Learn Ultra courses

As they completed the tasks in the usability tests, some students missed content contained in the ‘Welcome and Course Information’ learning module section of Learn Ultra courses because they were unclear that this section could be expanded. When they opened other learning modules, they found it difficult to distinguish between a folder (‘parent’) and a content item (‘child’) since parent and child items were adjacently positioned in the interface, and some students interpreted the document icon to mean there was a single file, not an item containing multiple files. This effect was more pronounced on mobile since item labels had been truncated on some devices. When viewing all their courses in grid view, some students said they found it difficult to distinguish between different courses when they had the same banner image.

Students appreciated consistent terminology

One of the tasks in the usability tests required students to find a resource list. To test how well the platform supported students’ exploratory information-seeking, the term ‘resource list’ was not specifically given in the task instructions, instead students were asked to ‘use the platform to find details of items on their reading list’ which required them to interpret the meaning of the instruction and look for appropriate content. In response to this task, several students commented that having fewer terms to describe items was preferable to avoid confusion, noting that ‘Books and Tools’, and ‘Resource List’ were both locations they would visit to find details of items on a reading list.

Most students thought the route to obtain assessment feedback involved was click-heavy

Although not indicated in the ‘Ease of Use’ scores for the task, 13 out of 15 students said they felt there were ‘too many clicks’ to get to their assignment feedback. In the usability tests, they were observed to click the feedback icon which activated an overlay showing the assignment attempts which temporarily covered the area where they could access the feedback.

Responses to the structured questions

These questions sought to probe students’ perceptions and expectations in relation to specific aspects of Learn Ultra features and functionality.

Students were positive about the Blackboard Learn App

One of the areas students were asked about was their use of Learn Ultra on a mobile.  The majority of students said they would use the  Blackboard Learn App if they were told about it – and that they would use it for the following tasks:

• Checking marks and feedback
• Checking the calendar
• Watching lectures
• Browsing lecture content
• Uploading assignments.

Students expected the University logo to link to the main website

Another question sought to confirm where students thought the University of Edinburgh logo (in the top left-hand corner of the Learn Ultra interface) would take them when they clicked (or tapped) it. Eleven out of 15 said they expected it to go to the University website homepage, affirming that this was an appropriate destination for this logo to be linked to.

Most students expected the calendar to include their full timetable and assignment dates

Students were also asked about the calendar, to find out how they felt this should work. Eleven out of 15 students said they expected the calendar to contain their full timetable and assignment due dates, with four saying they expected the assignment due dates but not the full timetable.

Acting on the research findings

The UX team worked together with the Learn Ultra project team to analyse the findings, to identify and prioritise areas to be worked on, and to schedule this work into the ongoing Learn Ultra implementation and roll-out.

Adjustments to Learn Ultra courses

In response to students’ comments about the ‘Welcome and Course Information’ section, amendments were made to include a description to help signpost the contents of this section to students. In addition, students’ difficulties interpreting aspects of the user interface display have been communicated to Blackboard to be fixed as part of their iterative improvement process.

Acknowledging the need for a range of banner images to enable students to easily differentiate between courses, a bank of images has been developed, to ensure staff can easily select course images and header images to use for Learn Ultra courses.

Access the Image Bank on Sharepoint (staff login required) 

User interface adaptations raised with Blackboard

The usability tests revealed several instances where the Learn Ultra user interface was not displaying correctly (for example, overlays masking content and content truncating). These issues have been directly raised with Blackboard to address in their ongoing development of the platform.

Improving the route to assessment feedback

Students’ ‘Ease of Use’ scores suggested improvements could be made to the way they were about to access assignment feedback. Several pieces of work have begun in response to this.

Positioning ‘assessment’ in the user interface

‘Assessment’ is  one of the items included in the course template, which was developed following extensive UX research as part of the Learn Foundations project. Compared to the earlier version of Learn, Learn Ultra has an updated user interface which offers more flexible ways to display items from the course template. To decide on the most user-centred option, students’ comments were considered along with findings from staff research and other feedback from the Early Adopter programme. A series of meetings was held with Learning Technologists, with the goal of agreeing and signing off a final version of the template to be applied to live courses for the 2023 to 2024 academic year.

Additional initiatives on the assessment process

In recognition of its importance for teaching and learning, several initiatives are under way to improve both the assessment journey for students and the assessment functionality available to staff. These include: a specific project led by a Business Analyst from the Learn project team, a review of current guidance on assessment (with a view to enhance this and include specific guidance on multimodal assessment), and engagement with various governance groups to finalise features and processes around Own Work Declaration.

Continuing iteration of staff training, guidance and terminology

The Learn Ultra roll-out is being supported by an ongoing programme of training for staff and guidance and support for students. Insights from the user research with students have fed into the development of fundamental training courses for staff, including the ‘10 things to get started in Learn Ultra’ course, as well as shaping training in more nuanced areas like course development.

Read more about Learn Ultra staff training on the Learning Technology webpage

Read tips and guidance on course development on Sharepoint (staff login required)

Students’ uncertainties about the Learn Ultra calendar tallied with similar points raised by staff, which prompted the development of specific guidance to detail best use of the calendar (in addition to the guidance included in the Good Practice Guidance staff resource).

Read guidance on using the Learn Ultra calendar on Sharepoint (staff login required)

Access the Good Practice Guidance staff resource on Sharepoint (staff login required)

An annual review of terminology was begun several years ago as part of the Learn Foundations project. This review is routinely completed, to assess if the terms included are still accurate, appropriate and widely known to staff and students. As part of this review, students’ comments about terminology in Learn Ultra are being addressed, in particular to look at the use of the ‘Books and Tools’ label.

Read more about the terminology used within Learn Ultra

Evidence to support Blackboard Learn App design and use of University logo

Several research findings provided data to validate specific courses of action, for example, the ongoing development of the Blackboard Learn App, and on a smaller scale, the configuration of the University logo to link to the University homepage.

Next steps: further UX research for the 2023 to 2024 academic year

A programme of user experience research has been scheduled for the start of the 2023 to 2024 academic year when Learn Ultra courses have been rolled out and students have experienced using it with real course content. This research will seek to continue investigating complex areas relating to students’ top tasks,  in order to gain the fullest understanding possible of students’ needs and requirements in the context of the service.

Specific areas to be addressed will include:

• How students interpret and use the course template in the Learn Ultra interface to navigate to content
• Students’ responses to different banner image options
• Students’ understanding and interpretation of iconography in the Learn Ultra interface
• How students find content in the context of different course structures (for example, using learning modules and course folders).

Other aspects will be added as the Learn project team continue to engage with those using and operating the Learn service, for example, staff involved in the Early Adopter programme, student interns and staff taking part in training sessions.