Students took part in a card sorting activity as part of Learn Foundations research. This helped us understand where they looked to find information in Learn to support their learning in the hybrid context. 

We carried out a card sort study to investigate how students expected to find content arranged within Learn. This was part of Phase Three of Learn Foundations, a project which strives to improve the user experience of Learn. The card sort was a repeat of a similar study carried out in Phase One of the project, and was designed to see if students’ expectations for navigating Learn had changed with the shift to hybrid teaching and learning. It was part of a wider programme of research carried out in Phase Three which also included a top tasks survey, usability tests and semi-structured interviews.

For an overview of the Phase Three Learn Foundations research you can read the separate blog post:

Understanding what staff and students need from Learn for hybrid teaching and learning

Initial card sort helped shape Learn Foundations template

A card sort is a type of study that helps to uncover how users expect to see information organised and structured, for example on a website. People are presented with a list of pieces of information (each one on a separate ‘card’) and are asked to sort these into categories, giving a label to each category they come up with. The resulting set of named categories helps to define the best way to arrange the information in line with what the user expects. In the web context, this can help build a user-centred information architecture.

In Phase One of Learn Foundations, a card sort was carried out with students, presenting them with typical items they would find within Learn and asking them to sort them into groups. The four strongest groupings which emerged from this study were: Course content/materials, Course information/administration, Assessment and Help and support. The card sort also revealed what language students used to label categories, which, together with the groupings, directly helped inform the development of the Learn Foundations template.  You can read about this card sort and the results in a separate blog post by Duncan Stephen:

Card sorting has informed a new information architecture for Learn courses

Template widely adopted in the shift to hybrid

The Learn Foundations template was developed with the aim of supporting staff and students’ needs consistently across the Schools and Deaneries of the University. It was produced after extensive research with staff and students to discover how they used and expected to use Learn, and it contains the following nine ‘core’ menu items to support task-based navigation of the courses interface:

• Welcome
• Course Information
• Announcements
• Course Materials
• Library Resources
• Assessment
• Course Contacts
• Have Your Say
• Help and Support

In addition to these core items, the template includes an optional ‘plus two’ menu items, designed to be adapted by Schools to customise the interface to meet specific contextual course needs. A glossary of terminology was also developed, designed to accompany the template to promote a consistent approach.

In Phase Two of the project, 10 Schools and Deaneries signed up to adopt the template. As the Covid-19 pandemic took hold, 20 Schools adopted the template to help them make the transition to hybrid teaching and learning.

Repeating the card sort in the hybrid context

With the shift to the hybrid model of teaching, it was appropriate to repeat the card sort to investigate whether students learning in this context expected to see information arranged differently from before.

By comparing the associations between items and the groupings emerging from the Phase Three card sort with those from the Phase One sort, we could understand whether students’ expectations for navigating Learn had changed. We could therefore draw conclusions about whether the template was still valid in the hybrid context or whether it needed to be altered to meet students’ needs.

As in Phase One, the card sort was only carried out with students. The students taking part were presented with a list of items they would expect to find in Learn and asked them to group the items in a way that made sense to them. The list of 51 items comprised 47 items included in the Phase One card sort, with an extra four items added to the list – one (‘Office hours’) based on feedback from student interviews carried out in Phase One, and the other three to reflect the hybrid context, as follows:

• Lectures (Live lectures)
• Virtual Classroom/Seminar (Live Seminar)
• Help Using Learning Technology Tools

The original 47 items contained one more than the list of items included in the other piece of quantitative research, the top tasks survey – this item was ‘Course handbook’ which was not considered a ‘task’ and therefore included in the card sort but not the top tasks survey.

Response from students – what we learned

205 students completed the survey, which although fewer than in Phase One (775 students), was a pleasing response given that when the study was run, fatigue levels in students and staff were high due to the ongoing effects of lockdown in the pandemic.

The data students provided in the card sort helped us to understand various aspects about the way they  want and expect to be able to use Learn. Firstly it helped illustrate which items they perceive as similar, and which they expect to find grouped together. Secondly, because the students proposed names for the groupings (or categories), the study provided insight into the type of language and terminology students were using to refer to tasks and items they encountered within Learn.

Students expected to find things in the same place

Four strong groupings of items emerged from the card sort exercise, which mirrored those from the Phase One card sort, indicating that students’ expectations of where to find items within Learn had not changed in the hybrid context. The groupings from Phase Three are illustrated in the image below:

The image shows the similarity matrix returned from the software (‘Optimal Workshop’) used to run the card sort. This matrix lists all the cards in the study (on the diagonal axis) and compares each of the 51 cards to every other card to illustrate the percentage of participants who grouped them together. The darker blue areas of the matrix highlight which items people commonly clustered together.

As was found in Phase One of the research, Course materials and content emerged as the strongest category, with the items in this group being clustered by 54-96% of the students who took part in the study.

It was striking to see that the students had grouped similar cards in each of these four main categories as students in the Phase One study had done. Despite teaching and learning being delivered differently, students were expecting to navigate Learn in the same way they had done in the non-hybrid context – they had not expected to see items grouped into ‘physical’ and ‘online’ categories as might have been expected. This was a strong indication that the current structure of menu items in the template was still valid, and was still meeting students’ needs in the hybrid context.

Students placed items into four main groups

Students clustered approximately 9-13 cards into each of the four main groups, accounting for approximately 80% of the total cards they were asked to sort. The remaining approximate 20% of cards were placed in various categories with less congruence over groupings. Items students expected to find in each of the four main groups were as follows:

Course materials and content:

• Learning outcomes for lectures
• Lecture notes/outlines/handouts
• Lecture recordings
• Lecture slides
• Lectures (or live lectures)
• Reading lists/study resources
• Resources for preparing for seminars/tutorials/labs/practicals
• Video conference
• Video  content (not lecture recordings)
• Virtual classroom/seminar (or live seminar)

Course information:

• Announcements
• Class locations
• Class times
• Course curriculum/outline/syllabus
• Course handbook
• Course-related events
• Field trip information
• Programme handbook
• Workload and expectations for course

Help and support:

• Careers information
• Course improvement surveys
• Help and support with my course
• Help and support with using Learn
• Help using learning technology tools
• Office hours
• Staff contact details
• Staff profiles
• Staff Student Liaison Committee minutes
• Student rep information
• Student support information
• Studio projects information

Assessment:

• Assessment marking schemes/rubric
• Assignment deadlines
• Assignment feedback/feed forward
• Assignment submission/upload
• Exam dates
• Examplar coursework
• Formative assessment
• Grades
• Group assignment submission
• Past exam papers
• Peer assessment
• Quizzes or tests
• Review own work declaration

Some items were difficult for students to place

Several cards had been sorted into various different categories, or had remained ‘unsorted’ suggesting there was little consensus from the students on where they would expect to find these items in Learn. Examples of some of these cards were as follows:

• Discussion forum – placed in categories including: ‘Discussion’, ‘Course Page’, ‘Help’, ‘Interaction’, ‘Course info’, ‘Learning tools’, ‘Work’, ‘Day to day learning’, ‘Participation’, ‘Lectures and Course Materials’
• Staff Student Liaison Committee Minutes – placed in categories like: ‘Support’, ‘Feedback’, ‘Announcements’, ‘Other’, ‘Admin’, ‘Wider University information’, ‘Getting Started/Staying Connected’, ‘Contacts and Information’
• Blog creation, update or comment – placed in categories such as: ‘Blog’, ‘Resources’, ‘Wider University Information’, ‘Everyday studying’, ‘Extra’,  ‘Learning aids’, ‘Chalkboard’, ‘Group and discussion tools’ ‘Your online presence’, ‘Further support’
• Learning journal (or other reflective tool) – placed in categories like: ‘Peer stuff’, ‘Assessment’, ‘Learning tools’, ‘Studying’, ‘Extra resources’, ‘Participation’, ‘Learn and revise’, ‘Organising’, ‘Reflection and beyond’, ‘Evaluation’
• Group sign up – placed in categories including: ‘Group work’, ‘Course admin’, ‘Information about course’, ‘Preparing for class’, ‘Help and support’, ‘Course materials’, ‘Community building’, ‘In the loop’,  ‘Sign ups’.

Categorisation of some items varied at a School level

For a minority of the cards, the grouping differed according to the School the student was part of, reflecting how different items within Learn have different purposes associated with them in the context of different Schools.

Examining the way the card Studio projects information was placed illustrated this. Students in the School of Mathematics felt this belonged in ‘Information’ or ‘Learning’, but students from the Edinburgh College of Art placed it in ‘Group learning’ and ‘Announcements’, Business School students felt it belonged in ‘Coursework’ or ‘Assessment’, and students from the School of Law suggested they would expect to find it in ‘Course information’.

This illustrated that while a template could be used by all Schools to group and organise items within Learn, there was still a need for individual Schools to customise Learn to meet their students’ needs. To ensure students located the more nuanced items (like ‘Studio projects information’), it was clear that Schools may need to either create nested menu items in one of the four big groups, or adapt the additional two menu items provided in the template to suit their needs as appropriate.

Students used the same terms to label categories

The terms students used to name the categories indicated whether the terminology used within Learn aligned with what students would expect. The most commonly suggested category labels from the Phase Three card sort were very similar to those from the Phase One sort. These included:

• Assessment – used to sort 1424 cards in Phase One and 573 cards in Phase Three
• Course Information -used to sort 1348 cards in Phase One and 419 cards in Phase Three
• Course content – used to sort 644 cards in Phase One and 216 cards in Phase Three

The category names emerging from Phase One helped inform the terms used for the Learn menu items. The final labels for the information architecture were chosen following a co-design session with students. In this session, students chose to use ‘Course materials’ as a menu item instead of ‘Course content’ despite ‘Course content’ being a commonly-used term in the card sort.

Read about the co-design session in the tree testing Phase One research:

Using tree tests to refine an information architecture

Comparing the terms students suggested to the glossary of terminology provided for use within Learn, there was alignment in that ‘Assessment’, ‘Course Information’ ‘Course Materials’ and ‘Assignment’ are already included in the glossary.

Slight differences in terminology between year groups

The card sort data was filtered to examine the category names used by first years (the first to experience hybrid teaching and learning) and non-first years (students who had been part of the University for one year or more and therefore had experienced teaching and learning other than hybrid), to see if the terms used by these student groups differed significantly.

On the whole, terms used by first years and used by non-first years were pretty similar. Several words were used slightly more by first years than non-first years and vice versa, but the differences were not sufficiently marked to prompt the need for a change to terminology.

Terms used more by first years included: ‘Activities’, ‘Online’, ‘Recordings’, ‘Live’.

Terms used more by non-first years included: ‘Coursework’, ‘Extra’, ‘Announcements’.

Consistency in card sort results showed templates still valid

Students taking part in the Phase Three card sort proposed the same four main groupings for the items as those taking part in the Phase One sort had done. This was quite surprising as it showed that despite the changes to learning and teaching with the hybrid model, students were still expecting to navigate Learn in the same way. Instead of expecting a split between physical and online items, they were expecting to find these together – for example items like ‘Virtual classrooms’ alongside more traditional learning items like ‘Reading lists’ and ‘Lecture notes’ within the ‘Course materials and content’ grouping.

As was found in the Phase One study, the four main groupings typically contained a broad range of approximately 9-13 items each, but the research offered no insight as to how these items should be organised within each grouping, it is likely that students would expect this to be done in a way most appropriate to the course or School, which may not be the same across all Schools. Students from different Schools had categorised cards differently suggesting that there was still a need for a degree of flexibility in organising content within menu items in Learn.

These findings provided a clear directive that in order to align with students’ expectations and provide the best possible experience, the existing Learn template with the nine core items and two additional, customisable items should be retained. It suggested that changes to the Learn template should not be made.

Terminology also remained the same in the hybrid context

The words students used to describe and label the groupings were also very similar when the data from the Phase One and the Phase Three studies were compared. Further comparative analysis revealed a very slight difference in the words used by first years (who had not experienced pre-Covid teaching, only teaching in the hybrid model) compared to non-first years, but there was no evidence of any new or emerging terms in used to describe items connected to hybrid learning which were missing from the terminology.

More blogs about Learn Foundations user research

Phase Three Learn Foundations has included both qualitative and quantitative research. You can read about the full programme of research in a separate blog post:

Understanding what staff and students need from Learn for hybrid teaching and learning

Details and results of the top tasks survey  are included in this post:

A top tasks survey has shown what staff and students prioritise in hybrid teaching and learning

Information about the research and findings from the Phase One Learn Foundations user experience programme are included in Duncan Stephen’s post:

A service design approach to Learn Foundations