To ensure the ongoing implementation of Learn Ultra remains as user-centred as possible, staff and students were engaged in user experience research between May and October 2022. 

In a continuation of the Learn Foundations project approach, a programme of user experience research – involving various methods – was carried out with staff and students to establish how effectively Learn Ultra supported key needs and expectations.

Read more about the upgrade to Learn Ultra

Read more about the latest programme of user research

Read about the student research

Objectives and methods for research with staff

Eight Learning Technologists and 11 Academic and Teaching Office staff were recruited from various Schools and Deaneries across all three Colleges to take part in the research. Most staff participants were part of the Learn Ultra Early Adopter Group, which had previously been established with the twofold purpose of enabling staff to gain experience and practice of Learn Ultra before the wider roll-out, and providing the project team the opportunity to gain feedback and insight from staff about Learn Ultra which would help shape its implementation.

Research with staff sought to achieve the following aims:

• Test the assumption that Learn Ultra offers a better overall user experience
• Assess how effectively Learn Ultra supports completion of staff members’ priority tasks
• Identify areas for improvement ahead of the roll-out of Learn Ultra courses
• Engage with staff from the Early Adopter programme to consolidate good practices and feedback from their experiences of using Learn Ultra.

Research methods included usability tests and semi-structured interviews, conducted online. Usability tests were conducted where possible on staff members’ own Learn Ultra environments with authentic course data – if this wasn’t possible, a test environment was used. Semi-structured interviews were held online and involved engaging staff in dialogue about Learn Ultra as they shared their screens to capture their comments, views and explanations in context.

The data collected were analysed in sessions attended by members of the UX team and the Learn Ultra project team.

Findings from research with staff

Overall, the research revealed positive staff perceptions of Learn Ultra. Some areas were identified for improvement, and the research also highlighted areas where further guidance and support on the use of Learn Ultra features and functionality would be beneficial.

Insights from the usability tests

Top tasks surveys previously ran with staff revealed the activities they felt were highest priority when teaching online. Using results from the most recent survey completed in 2021, the tasks which staff deemed the most important were compiled – to form the basis of the usability tests. Learning Technologists and Academic and Teaching Office staff were asked to complete these tasks using Learn Ultra, to investigate how effectively it supported completion of these priority activities.

Read more about the most recent top tasks survey in the related blog post:

A top tasks survey has shown what staff and students prioritise in hybrid teaching and learning

Learning Technologists’ tasks and ‘Ease of Use’ scores

Learning Technologists were asked to complete several tasks from the ‘top tasks’ list and say how easy they found them. Although these activities did not necessarily fall into their usual remit, they were familiar with them and it was important to test how successfully Learn Ultra supported these sorts of task from a broad range of staff perspectives. The tasks, and the associated scores reflecting how easy the Learning Technologists felt each was to complete were as follows:

• Add details about a particular course – score 9.6/10
• Add a reading list – score 7.8/10
• Schedule a live lecture – score 6.8/10
• Add lecture notes – score 8.8/10
• Add lecture slides – score 8.8/10
• Add lecture recordings – score 6.4/10
• Let students know that new lecture notes are available – 8.6/10.

Given these tasks were not usually completed by Learning Technologists, it was pleasing to see that they were able to complete all the tasks, and that they found most tasks quite easy. Tasks they were unsure about included: using the platforms to use for live lectures, and scheduling live sessions with the Calendar. They were also unfamiliar with where to access Media Hopper Replay to add lecture recordings, which accounted for the relatively lower ‘Ease of Use’ scores for these tasks.

Tasks completed by Academic and Teaching Office staff, and associated ease scores

Using their own Learn environments with real course data, the tasks Academic and Teaching Office staff were asked to complete and the associated ‘Ease of Use’ scores they assigned were as follows:

• Add a course curriculum – score – 8.3/10
• Schedule a virtual session – score 6.6/10
• Add a video that you have recorded – score 7.0/10
• Add lecture notes and slides – score 8.6/10.

Like the Learning Technologists, Academic and Teaching Office staff were successfully able to use Learn Ultra to complete all of the tasks. Some staff were unsure of the technology to use for virtual sessions, and did not immediately understand the scheduling options in Collaborate. They were also unclear of the process to embed videos, having initially struggled to find them in the ‘Content Market’ part of the interface. This meant that they found these tasks more difficult to complete compared with others.

Insights from the semi-structured interviews

After staff completed the tasks in the usability tests they were engaged in questions about different aspects of Learn Ultra. This helped tease out their associated views, requirements and queries.

Most staff liked the shallower folder structure in Learn Ultra

When questioned about the flatter structure of Learn Ultra (compared to the original version of Learn), staff from the Academic and Teaching Office said they preferred the shallower layout and felt it could make content easier to find for students. The Learning Technologists also preferred it, however, some anticipated it might make it difficult to move existing course content for some courses (for example, content that had been arranged in deep, nested folders) to Learn Ultra.

Some felt aspects of content styling and formatting in Learn Ultra courses were restrictive

From an accessibility perspective, the interface of Learn Ultra represents an upgrade compared to the original version of Learn, and therefore, some of the options for styling and formatting content have been adjusted to ensure a more accessible user experience. The move to a shallower folder structure was one of these changes – in addition, options to change text colour, insert custom HTML and manipulate tables for the purposes of layouts and structuring text are no longer available. While the shallower structure was generally liked, some staff felt other adjustments represented a loss in flexibility and visual appeal. Some staff also commented that some of the icons used in the interface (for example the ‘document’ icon, the icon used for lists in the text editor, and the icon for video content) were not intuitive on first glance.

Some staff wanted to tweak parts of the evolved course template

Extensive UX research carried out in the Learn Foundations project previously informed the design of a course ‘template’ which took the form of nine ‘core’ items plus an optional ‘plus two’ items. In the original version of Learn, these items could only be displayed as a left-hand menu, however, in the Learn Ultra interface there is greater flexibility, with the option to place items in a top navigational bar as well as on the left-hand side, representing an evolved version of the template. When staff were asked about the course content section in the semi-structured interviews, 43% of Academic and Teaching Office staff said they would use the evolved template without making changes. Others said they would like to be able to make changes to accommodate the needs of specific types of courses – for example to remove the ‘Assignment’ item for non-accredited courses, and to remove ‘Lecture Recordings’ item for courses where this was not applicable, and to change the ‘Welcome and Course Information’ label to signpost  the contents of this section to students more clearly.

Some staff identified changes to the Assessments section

Several points were raised about the assessments functionality in Learn Ultra. Staff noted that the question bank worked differently and that several question types (for example, surveys, hotspots and ‘fill in the blank’ questions) were no longer available. They felt that the range of grades presented in the marking schema was unclear and noted the absence of the ‘Declaration of Own Work’ checkbox.

Mixed responses about the Learn Ultra discussion boards

When prompted, a few staff commented on the discussion boards. Noting the differences between the original version of Learn and Learn Ultra, some said they preferred the Learn Ultra discussion boards because of their layout, the improved ability to keep track of new posts and the flexibility to align them with weekly course content. Others said that they may like to change them, for example by including a link to Turnitin, and a way for students to contribute anonymously.

Staff clarified their requirements for groups functionality

Some instructors said they wanted to be able to set up group spaces for discussions, journals and Collaborate sessions. They expressed uncertainty around how to achieve this in Learn Ultra, as well as to release course material to different groups at different times, and to add individual group classes to the calendar.

Several staff were uncertain about how to use the Learn Ultra activity stream

The activity stream is a new feature in Learn Ultra, there is not an equivalent in the original version of Learn. It shows a timeline of activities coming up, including events and alerts. When questioned, a few staff members said they were unclear of how it would work – in particular they wanted to know how the system determined which alerts and activities appeared in the stream.

Some staff were unsure of the best way to use the calendar

Staff realised that for events and classes to show in the Learn Ultra calendar, they would need to manually input them. They were uncertain of whether this would be good practice, taking into account the existing scheduling tools for their respective courses, and recognising the risks of confusing students by changing things.

Acting on the research findings from staff

Staff from the Learn Ultra project team worked with the UX team in the analysis sessions, therefore they were able to identify and prioritise areas to be worked on, and to schedule this work into relevant workstreams in the ongoing Learn Ultra implementation.

Confirming template for the 2023 to 2024 academic year

Acknowledging comments from staff about how they would use items of the course template, together with insights and feedback from the Early Adopter programme and previous UX research, a series of meetings with Learning Technologists representing the different Schools was begun to discuss options with a view to agreeing and signing off a final version of the template to be implemented in the 2023 to 2024 academic year.

Adjusting to content styling changes while ensuring Learn Ultra remained visually appealing

Staff comments about the options to style content expanded on feedback already gathered as part of the Early Adopter programme. Changes to content styling aspects (including iconography, colour, tables, HTML editing) had previously been noted and the accessibility-related reasoning behind these changes had been identified. Options for colouring text had been altered to avoid reliance on colour as a signifier which would exclude people unable to identify colour contrasts. The ability to use tables and HTML functionality to structure content had been restricted to ensure content did not render inaccessibly in responsive view.

To enable staff to make content in the Learn Ultra interface as visually appealing and user-centred as possible in an accessible and inclusive way, several measures have been implemented. An Image Bank containing specially-prepared pictures to be used in the interface has been developed, and guidance on use of media items has also been updated in the Good Practice Guidance. Additional pieces of guidance (based on instructional design principles) have been developed to support staff in the arrangement of course content, with detail on how to structure content in the flatter hierarchical structure using folders and learning modules and how to move courses from the original version of Learn to Learn Ultra.

Access the Image Bank on Sharepoint (staff login required) 

Access the Good Practice Guidance on Sharepoint (staff login required) 

Access the guidance on course structure on Sharepoint (staff login required) 

Access guidance on moving courses from the original version of Learn to Learn Ultra on Sharepoint (staff login required) 

Workstreams initiated to focus on assessments in Learn Ultra

The UX research surfaced responses from both staff and students about assessments in Learn Ultra, and feedback about assessments had also been contributed through the Early Adopter programme. To ensure a holistic consideration of assessments, a specific project has been initiated by a Business Analyst working within the Learn project team. Furthermore, a review of the guidance on assessments has also begun to update information in specific areas (including multimodal assessment). Engagement is also underway with governance groups across the University to update and establish processes for Own Work Declarations.

Updating guidance and training for use of discussion boards, groups, activity stream and the calendar

Since the UX research was completed, staff in the Early Adopter programme have continued to experiment with discussion boards, groups and the calendar, and their insights, together with the research findings have shaped different types of best practice advice, guidance and training for effective use of all three features of Learn Ultra. The Good Practice Guidance (noted above) includes advice on the best ways to use each feature to achieve effective communication with students, and extra guidance documentation has been developed to explain use of the calendar and activity stream. The ‘10 things to get started in Learn Ultra’ training course covers essential information – such as how to set up groups and discussion boards, and there are additional training sessions which are more specific and in-depth.

Access guidance about use of the calendar and activity stream on Sharepoint (staff login required)

Access ’10 things to get started in Learn Ultra’ training course

Feedback from research influencing Blackboard’s planned feature iterations

Iterations of the discussion boards, groups and the activity stream is dependent on Blackboard’s development schedules and roadmap. Feedback from staff about these features has been communicated to Blackboard and is shaping the ways they continue to develop them.

Next steps: Further UX work for the 2023 to 2024 academic year

It was acknowledged that some features and functionality of Learn Ultra required further investigation before decisions could be made about the most user-centred means of configuration and implementation. At the start of the first semester of the 2023 to 2024 academic year, Learn Ultra courses will be live, and staff and students will have experienced using it with authentic course material in the real studying context. At this point, a follow-up programme of UX research will be timely. Specific areas identified for the focus of this work include the following:

• Staff requirements and experience of using Learn Ultra discussion boards and groups
• Students’ experience of Learn Ultra discussion boards and groups
• How students interpret and use template items to navigate to content
• Student responses to different banner image options
• Students’ understanding and interpretation of iconography in the Learn Ultra interface
• How students find content in the context of different course structures (for example navigating between learning modules and course folders).

As the Learn project team continue to engage with staff through training sessions and the Early Adopter programme, other aspects to be investigated will be added to the UX research programme.