FindAMasters AI-assisted summary pilot
At the end of 2024, one of our third-party listing sites, FindAMasters, approached us about taking part in a pilot looking into seeing if an AI-assisted process could complete its ‘brief’ (creating third-party programme listings) using information from our postgraduate degree finder. We thought this was a useful opportunity to put some AI to a practical test. In the end, it reinforced the need for caution when using AI, and that for many tasks AI still may not save you time as you have to review and amend its outputs.
Currently, updating the University of Edinburgh postgraduate programmes on FindAMasters is a manual process. It’s a time-consuming task when we have over 500 postgraduate programmes. Naturally, we were interested to see if an AI-assisted process could undertake this task accurately, as it would save us potentially weeks of work.
We agreed to take part in the pilot and were asked to provide a sample of postgraduate programmes to use during the pilot. We selected examples like ‘Biodiversity, Wildlife and Ecosystem Health (Online Learning)’ and ‘Management MSc’.
The programmes were selected because they have their own unique quirks. For example, Biodiversity, Wildlife and Ecosystem Health (Online Learning) has different certification and delivery methods, including part-time intermittent study options. While Management MSc has variation in the fees and costs section, with an application fee and deposit being required as part of the application process. Others also featured unique elements that would help us test the accuracy of an AI tool.
If this pilot was successful, we would move on to a 50% test of postgraduate programmes.
Version 1 – Disconnect between the degree finder source and the AI-generated content
FindAMasters provided us with an initial ‘brief’ for both postgraduate programmes using their default prompt.
From taking an initial comparison of the AI-generated ‘briefs’ and the postgraduate programmes, it was clear there was a disconnect between the source and the output.
Some of the issues from the AI-generated content included:
- Incorrect information – This included some incorrect application deadlines and a reference to visas for a programme that requires you to be employed in Scotland.
- Issues with ‘invention’ – The AI process generated inaccurate/misleading information presented as fact, normally due to this information not fully being available within the degree finder. This was particularly noticeable within ‘how you will learn’ methods, ‘a typical day’ blurbs, and career outcomes (invented job titles).
- Issues with fees – It could not provide fees information, probably because this information can be found on the fees website rather than the degree finder.
- Struggled with certification and delivery methods – It appeared to struggle with Part-time/mixed certification programmes like the Biodiversity, Wildlife and Ecosystem Health programme, where the exit routes are not explained sufficiently, and other sections are treating it a bit like a full-time programme.
The issue with ‘invention’ was a particular concern, as we need to ensure that all information across our third-party listing sites is accurate. The prompt used specifically stipulates that the LLM should not guess or invent when putting together these briefs. Here’s a section from the prompt used:
You **WILL NOT** guess, infer, assume, or invent any aspects, elements, topics, or content of the degree programme summaries, and only use the content provided within `programme_description` as the knowledge base for each programme!
From this initial test, we were not satisfied with what the AI-assisted process had produced. Therefore, we couldn’t advance to the 50% test without knowing more about the process used, including the temperature of the current AI prompt, and we needed to see a significant improvement in the levels of inaccuracy and invention within the content.
Version 2 – Improvements, but still issues with invention
FindAMasters got back in touch to answer some of our questions regarding the prompt and provided us with a second version with some modifications to the prompt.
Firstly, regarding the temperature set-up of the AI prompt, it was initially set to 1 (the midpoint). The reasoning was that setting the temperature too low makes the AI more likely to make mistakes as it ‘thinks’ too literally. For the next round of testing, it would be set at 0.6.
Our contact from FindAMasters provided a second batch of overviews using an updated prompt, and overall, there was an improvement to the programme descriptions compared with the first version.
In general, it felt like the AI-assisted process did a good enough job of replicating what’s on the degree finder page when the information is provided. On the other hand, when the information isn’t provided, it can produce inaccurate/invented content, which is a major concern.
The career outcomes were much improved from the first round of testing, with no obvious invention. Interestingly, this part of the prompt didn’t appear to have been changed from the first round of testing, and it gave the LLM a license to disregard the rules around potential career paths.
“**YOU WILL** make one exception to this rule for the ‘Career Outcomes’ section! Here, you are allowed to extrapolate, forecast, and intuit potential career paths that students could expect to take following the completion of programme, based on your own knowledge.”
With the invention in the ‘Career outcomes’ section being a problem with version 1 and not with version 2, it feels like the output is slightly random when this part of the prompt hasn’t changed.
In version 2, we still found issues with the ‘How you will learn’ and ‘A typical day’ fields. Because this information is not currently provided on some degree finder pages, the AI model simply makes a guess at what it thinks will be suitable for these fields, which leads to inaccuracy within these sections of content.
We’re not using AI-generated content anytime soon
I think it’s safe to say we’re not changing to AI-generated items any time soon, as we got pretty mixed results with the content:
Potential issues of using AI to populate our third-party listing sites
- Invention: Invention is a common issue with AI-generated content. Despite the commands from the prompt, there were still instances of ‘invention’ when the information wasn’t This is really concerning when it is essential, we provide accurate information.
- CMA (Competition and Markets Authority) concerns: We have a degree of responsibility for any University of Edinburgh content surfaced on third-party sites such as FindAMasters. If AI was rolled out more widely, we’d have more content to monitor for accuracy, and would need processes and resources to do this – is this better than taking control ourselves?
- AI pilot was focusing on the wrong solution: I think time would perhaps be better spent on a direct crawl of our sites, which is what some of our other third-party partners do for us, lifting the information verbatim as we worded it and publishing that. Happily, it looks like this will be an option for us going forward.
Potential benefits of using AI to populate our third-party listing sites
- Spend our time doing other things: We hire part-time staff who copy information from the degree finder to third-party listing sites such as FindAMasters. It’s a time-consuming task when we have over 500 postgraduate programmes. If an AI-assisted process could potentially undertake this task accurately, we could have our part-time staff working on more important things than data entry.
We discussed with stakeholders and FindAMasters, but the outstanding issues meant while we enjoyed participating in the pilot as an interesting experiment, it’s not something we want to take forward at this time.
I think this pilot highlighted the importance of not rushing into AI blindly, with there clearly being more cons than pros at this stage. I think the direction of travel is pretty clear, and we will need to utilise AI in the future but having an understanding of when and when not to use this technology is important.
I think there is definitely a need to find a reliable automation that could reduce the overhead of updating. We’ve experimented with using AI-powered tools on Excel, such as ‘Text Insights’ which we’ve found to be more efficient at providing summaries than ELM (Edinburgh Language Model), although we need to do more testing to ensure it would be capable of undertaking such a big task.
Read more about our team’s experiences with AI
If you were interested in this blog, read about the experiences of the rest of the Prospective Student Web Team:
Great blog Louis. With my CMA hat on, this line fills me with terror: the AI model simply makes a guess at what it thinks will be suitable for these fields.