Continuing our UX research on AI features, we worked with the EdHelp team to design and run user tests to learn how students respond to their ELM-enhanced online enquiry service, AskEdHelp. This post documents our methods, findings and recommendations. 

The inclusion of AI in online experiences is becoming the norm, and at the University, the availability of ELM – the University’s very own AI innovation platform – has facilitated and accelerated increased adoption of AI in a growing number of our digital products, systems and services.

Read more about ELM, the University of Edinburgh’s AI innovation platform

About ELM page on the University of Edinburgh website

A notable application of ELM is within EdHelp, the University’s student enquiry service – which provides a first point of contact for students wishing to access a range of services at the University. ELM is used to power a chatbot called AskEdHelp which handles student queries in the areas of Library, Information Technology, Student Finance and Student Administration. The chatbot was designed by EDINA and it works by using Large Language Models and Retrieval Augmented Generation in ELM to interpret conversational queries from students and generate suitable responses, drawing on data from defined knowledge bases.

AskEdHelp was launched in 2024 and having operated it successfully for a while, the EdHelp Service team wanted to learn more about how students interacted with it.

Read more about the development of AskEdHelp

EDINA – An insight into our new service, the AskEdHelp chatbot service

Having conducted research on several AI-powered features before, the UX team was ideally positioned to lead this work. We have previously researched and blogged about:

• The UX of an early iteration of a Drupal AI assistant
• What students thought of an AI-enhanced searchbot
• How staff interacted with an AI content design assistant

Furthermore, anticipating an increasing number of applications of ELM to digital experiences at the University, we were keen to work on AskEdHelp to enable us to expand on our existing expertise and strengthen our abilities for designing and carrying out tests of AI UX in the future.

Working with the EdHelp team, we established research goals

Since it had been developed by EDINA, AskEdHelp had been incorporated on the EdHelp section of the University website by the EdHelp team, making it available for all students to use. The EdHelp team had been able to access logs showing queries and responses and had continued to work with EDINA to make tweaks to the chatbot operation to improve the quality of responses, however, they wanted to understand whether the presentation of the AskEdHelp chatbot on the EdHelp webpages could be improved to align with students’ expectations and interaction habits, and also to learn more about how students naturally looked for information. In particular, the EdHelp wanted to understand:

• Students’ preferred methods for finding out information about University topics
• How students interact with the EdHelp site to find answers to their queries
• Students’ perceptions of the AskEdHelp chatbot and its purpose
• If there are any usability issues with the EdHelp site

Adopting these objectives as the research goals, the next step was to shape a test for students to participate in.

We designed a test to learn how students perceived and used AskEdHelp

The test scenario was simple – involving participants being asked to assume the role of students looking for help on various topics. The tasks were decided based on the known capabilities of AskEdHelp – in other words, relating to the Library, Information Technology, Student Finance and Student Administration.

The tests began with some initial questions about:

• Participants’ previous use of EdHelp
• Their broader information-seeking habits

The tasks within the test were as follows:

1. Please have a look at the content on this page [EdHelp page]. Please tell me what do you think this site is about, what do you think you can do here, and how you would use this page?
2. Considering the ‘Contact EdHelp’ button on this page, please tell me what you would expect to see if you pressed this button?
3. Please now press the button and tell me how what happens next compares to your expectations
4. You want to find out how you can use the Library once you have left the University. How would you use the EdHelp site to do this?
5. Now you want to ask a follow-up question based on the information you’ve found out so far. How would you do this?

Follow-up questions sought to understand how students had made sense of AskEdHelp in its context on the EdHelp page, and to know how their experience of using it compared to their usual information-seeking methods. These questions included:

• Now you have used AskEdHelp, could you tell me what you think the ‘thumbs-up’ and ‘thumbs-down’ buttons are for?
• Could you talk me through the different options to get help from this page?
• Has this experience made you think differently about how you would search for information about the University in the future?

Students’ responses to the tests helped inform recommendations

Tests were completed with five student participants, and collectively, the way these students interacted with AskEdHelp and the EdHelp website, the answers they provided to the questions and the detail they provided as they spoke through their actions provided rich insights to guide improvement of the overall AskEdHelp experience in line with students’ expectations and perceptions. The main observations and recommendations are summarised below.

Students engaged with only some of the content on the EdHelp page

At the time of the tests, the EdHelp page contained textual, image and video content, arranged in boxes or cards. When asked to review the content on this page and speak about its purpose, students were drawn to the ‘Get Help’ box and the ‘Contact EdHelp’ call-to-action button, however, none watched the video.

Having identified multiple routes to get help, students were unclear which to choose

On the EdHelp page, students recognised ‘Get Help’, ‘Contact EdHelp’ and ‘Contact us’ as the help options available to them (pictured below), however, they did not know the differences between the various routes, and therefore were uncertain which one they should pick for any given circumstance.

Screenshots showing the the different routes to get help from the EdHelp page

The ‘Contact EdHelp’ button suggested it would open a form when pressed

When students were asked what they expected to happen when pressing the ‘Contact EdHelp’ call-to-action button they felt they would be prompted to enter their name and other details into a form to capture details of the query, which would be submitted to be handled by an traditional service desk enquiry service.

Students were uncertain how to choose from the AskHelp options available

Having pressed the ‘Contact EdHelp’ button, students saw a list of options to select from, with a short sentence of guidance text advising them what to do (pictured below). The selection of an option was required to help triage queries to either be handled by ELM (in a way appropriate to the category selected, drawing from the appropriate knowledge base), to be handled by the EdHelp team, or to be directed to another relevant source dependent on the topic (for example the Student Immigration Service for visa queries). Not all students read the guidance text, and others were unclear why they should make a selection and sought to skip past the options presented to get to a place where they could enter their query. One student considered selecting the final option ‘Submit your enquiry to the EdHelp team’ which would have taken them to the service desk enquiry submission form.

Screenshot showing options upon pressing the ‘Contact EdHelp’ button

Screenshots showing pathway presented when users choose ‘Submit your enquiry to the EdHelp team

All students were able to interact with AskEdHelp and get a relevant response to their question

As they completed task 4, all the students navigated to the AskEdHelp free-text box and, in their own words, typed a question about using the Library once they had graduated. Each query generated a response that the individual students felt was useful. They judged the quality of the response they received by the inclusion of relevant links – as they felt this demonstrated that the response had come from a verifiable source.

Students were unclear about differences between the chat options

When prompted to share what they would do to ask a follow-up question, all of the students opted to continue to type into the free-text box. Reviewing the other options available, they were uncertain what pressing the ‘Live Chat’ option would provide (one participant thought it would take them to a chat with a person – whereas in fact, it took them to an enquiry form to complete), and said they would continue to type queries into the free-text box rather than choosing the ‘New Chat’ button.

Screenshot showing chat options in AskEdHelp once a chat has been initiated

Screenshot showing the enquiry form that appears when the ‘Live Chat’button is pressed in EdHelp

The evaluation buttons were familiar to students

All the students felt they would use the thumbs-up and thumbs-down buttons to give feedback on the AskEdHelp chatbot. One noted they would only use the thumbs-down button if they received an unsatisfactory response, and another said they would use the feedback button to provide more detail for improvement.

Convenience and speed impacted students’ choice of search strategy

When asked if their experience of using AskEdHelp would impact the way they searched for University information in the future, students highlighted speed as a primary factor affecting how they sought information – a web browser search was their preferred search method because it was easily accessible and quick, whereas navigating to AskEdHelp involved multiple steps and screens to click through. Comparing the outputs obtained from web browser searches compared to an AI chatbot like AskEdHelp, several commented that they preferred the succinct responses, rather than being presented with a long list of links to look through.

Recommendations centred on clearer, more consistent signposting of the help options

Taken together, the observations from the tests with students suggested that several tweaks to the presentation and positioning of the help options provided by EdHelp could provide a smoother, more intuitive user experience.

Removing unused content on the EdHelp webpage could help highlight the help options

In the current page set-up, the EdHelp webpage contains a video and textual content that users tend to skim past in their search for help. Removing this surplus-to-requirements content could help ensure users can engage more quickly with the help available – testing with representative student users could confirm if this had the desired effect.

Clearer signposting could help students choose the right route for their query

EdHelp offers students a choice of ways to obtain help, but it is not immediately clear when they should pick each option. To guide students in the right way, the routes could be presented in an instructional way, for example:

• Contact EdHelp – our ELM-powered chatbot, AskEdHelp. Choose this route for AI-generated responses to queries about the Library, Information Technology, Student Finance and Student Administration
• Live Chat/EdChat [available within the AskEdHelp chatbot] – our service to chat with staff, available during office hours [to be specified]. Choose this route for queries to be directly answered by staff members
• Get Help – our knowledge base where you can find information on various topics. Choose this route to self-serve your query
• Contact us – our traditional enquiry-handling service, to have your query submitted to the service desk

Testing of changed descriptions could help establish if this made the options clearer to students.

Consistent labelling of the categories handled by AskEdHelp could help users appreciate its capabilities

Currently the categories that AskEdHelp can deal with are described on the EdHelp page as ‘… the areas of Library, IT, Student Finance and Student Administration’. On the Welcome to EdHelp page, the options are presented slightly differently: ‘The Library’, ‘Information Technology’, ‘Tuition fees/loans/scholarships’ and ‘Your student record/your student documents’. A small but impactful improvement could be to ensure the categories are worded in the same way. User testing could then help verify what students understood by the different categories – to check they were intuitive and phrased to align with words students would naturally use.

We’re continuing to work with the EdHelp team

Referring back to the research goals, this initial round of research with students established that students could successfully interact with AskEdHelp, the ELM-powered chatbot, to obtain meaningful responses to a defined query. Insights into how students interacted with and made sense of the EdHelp options guided recommendations for several usability improvements around clearer placement of the options available, to position AskEdHelp alongside the other help options, and to help establish EdHelp as a primary source for students looking for information about the University. Our next steps are to carry on supporting the EdHelp team to make changes and continue testing with students to guide continuous improvement of their service offering.