I used ELM (Edinburgh (access to) Language Models) to test content to see if it followed the University of Edinburgh’s style guide. It returned mixed results and I’ve highlighted my key findings below.

What is ELM? 

ELM is the University of Edinburgh’s AI innovation platform. It helps staff and students safely use advanced AI technologies called Generative Artificial Intelligence (GAI) and Large Language Models (LLMS). 

You can find out more about ELM below: 

ELM – (Edinburgh (access to) Language Models) 

Testing process 

To do our tests, we first downloaded the University of Edinburgh’s A to Z style guide, which can be found here: 

University of Edinburgh Style Guide A to Z 

We uploaded the style guide to ELM and then prompted ELM to review content using the style guide. We used pages across the University’s website to test how useful ELM was in referring to the style guide and its ability to issue corrections. 

We looked to get a selection of pages that contained a lot of content and utilised different formats in order to try and cover as many of the topics on the A to Z style guide as possible. 

Some of the pages we tested are listed below: 

Masters by Research – School of Chemistry 

Undergraduate Archaeology – School of History, Classics and Archaeology 

Programme Structure (MSc) – School of Economics 

Edinburgh Life – Accommodation  

Protecting your bike against theft – Transport  

Below are our main takeaway points from the testing process. 

ELM cannot access web pages, so content has to be pasted in as plain text 

One very important thing to note with ELM is that it cannot access external URLs or content outside of what’s directly in the chat interface. If you wish to have content from a web page checked, you will have to paste the content into the text box on ELM. 

When pasting page content into the chat interface, it removes all formatting from the page which is not ideal as formatting is one area you will want ELM to check against the style guide. 

Links in the style guide make it difficult for ELM to truly follow it 

Since ELM (at the time of writing) cannot access external URLs, it already encounters some issues with the University of Edinburgh’s A to Z style guide.  

Guidance on the following topics link to other pages on the University’s style guide website, with no guidance given in the actual A to Z style guide: 

• Italics 
• Punctuation, including quotation marks and the use of symbols like % and & 
• Dates and numbers 
• Contact details 

There are also topics on the style guide that have a small amount of information, but have more information contained on the web pages. 

As a result of these links, ELM cannot give confident answers in relation to the style guide. 

For example, when checking content across websites, ELM often returned some advice on italics like this: 

“Check the use of elements like book titles or important terms. The guide dictates when and how to use italics and quotation marks.” 

ELM is not positive in its evaluation to italicise the title of a publication as it has no information to work of, so while it does not outright ask us to use italics, it recommends checking. 

Upon checking, the guidance on the style guide website is to “never use italics online”. 

For ELM to return a more confident answer, the guidance that is included solely on the web pages will need to be included in the A to Z style guide. 

The longer the piece of text you ask ELM to check, the more likely it is to return incorrect or irrelevant advice 

We asked ELM to proof the content on the sites we checked. We used the following prompt when asking ELM to check content against the style guide: 

“Using the style guide attached, can you check the following content follows the style guide?” 

When this prompt was used, it returned some irrelevant advice, especially when there was a lot of content on a page. On the Programme Structure (MSc) in Economics page, ELM returned advice on the use of ampersands, despite the content having no ampersands. 

It also returned incorrect advice on two different pieces of content, suggesting that University of Edinburgh should be written in lowercase when used in a sentence, which contradicts the style guide.  

While ELM was returning some correct advice, it was listed alongside the incorrect and irrelevant responses. 

Using a shorter snippet from pieces of content to be checked returned more helpful information 

Instead of asking ELM to check all the content, we asked it to only check everything under individual headings. We used the same prompt when we asked it to check all the content across the different websites. 

ELM’s feedback in this instance was more useful. It highlighted areas that had followed the style guide correctly, noting how acronyms and hyphenation had been used, and made suggestions on improving formatting.  

However, formatting had been altered once I pasted the text into ELM, which removed the link text and bullet points that accompanied the text. ELM’s suggestions were around adding link text and bullet points, so it was suggesting changes that had been followed prior to text being pasted into ELM. 

We asked ELM questions that had previously been used when the style guide was tested by users 

We asked ELM to complete a number of tasks that had previously been used to test the style guide.  

You can read more about these usability tests on Emma Horrell’s blog post: 

Usability testing the Editorial Style Guide site 

Our aim was to see if ELM could complete the tasks, and if it could, if it was accurate.  

The following prompt was used for the test: 

“I’d like you to imagine you are a person working at the University of Edinburgh. I’d like you to imagine you have been given a piece of copy or text to put on one of the web pages you maintain. Someone draws your attention to a few things in the text that they advise you to check using the Editorial Style Guide. Do not use any other information other than that in the style guide. These are the tasks:” 

ELM provided sound advice, but it wasn’t following the style guide 

One of the tasks involved finding out if you could use a raw URL in a piece of text. Information on links is on a separate web page, so the expected result was that it would not be able to find an answer. 

ELM provided good advice on writing good link text and how using a raw URL should be avoided unless it is necessary. But this wasn’t taken directly from the style guide. 

When prompted about where it found this information on the style guide A to Z document, it noted that this information wasn’t fully available on the attached document, but that the raw URL should be changed to follow general content management and web accessibility practices.  

Furthermore, when asked if the word ‘optimized’ could be used, it highlighted that since University of Edinburgh is a British institution, it likely prefers the British English spelling. It again provided sound advice based of context, but did not back up its answer from the style guide. 

Would an improved style guide help ELM return better and more confident results? 

Throughout the testing process, ELM provided mixed results when asked to check content against the style guide.  

There are many factors to consider in trying to help improve ELM’s responses, including: 

• Would an A to Z style guide that contains all the advice that should be followed work better than the current one which links out to different web pages for certain topics, and would this provide better results? 
• How can we make clearer prompts to ELM to ask it to check content against the style guide conventions and rules?  
• Is there a limit on text I can paste into ELM before it starts to become more inaccurate? 
• Are there better ways to upload web pages into ELM? 

This is just a sample at what’s to be looked at when using the ELM to help follow the style guide. AI is constantly evolving, but it important to know its limits and to effectively and efficiently know how to use it to obtain the best results. 

Digital Content Style Guide Intern joining the UX team 

To help us look at what can be done to improve the style guide, our team is recruiting a style guide intern to assist with improving the style guide. 

Our Style Guide Intern will research ways to improve our editorial style guide. They will: 

• Help review the format of the style guide 
• Help make the style guide more user-centred 
• Investigate new and innovative ways to apply the rules laid out in the guide. 

If you are a current University of Edinburgh undergraduate student and you’re interested in applying for the role of Digital Content Style Guide Intern, you can find more information here: 

Digital Content Style Guide Intern job description