Early adventures in UX AI research: Pilot testing Drupal AI Assistants
AI Assistants are an exciting new Drupal development, intended to support and empower content creators and site editors to achieve their tasks. Having designed a UX research approach to test Drupal AI Assistants with University staff, I conducted a pilot test to try out the set-up: planned scenario, tasks and identified UX success indicators.
Drupal is the open-source content management system used by the University for EdWeb2 and other important systems. Drupal AI Assistants were unveiled at DrupalCon Barcelona in September 2024 as part of Drupal CMS (Drupal’s newest product). To ensure their development progressed in a user-centred way, the AI Assistants needed to be tested with people representative of the Drupal CMS target audience, to understand how people naturally expected to use and interact with them. With this knowledge, areas for improvement could be identified and prioritised.
I felt that these AI Assistants could be helpful to University staff publishing content within EdWeb2 so I was keen to test this assumption by doing some testing and user research. I worked with Jamie Abrahams, track lead for Drupal AI, from AI specialist firm Freely Give, to design tests and carry them out.
Read about how I worked with Jamie to design the tests in a separate blog post:
How do you learn about the UX of AI? – designing tests for Drupal AI Assistants
In this blog post, I outline the detail of the pilot test: what we tested, the associated success indicators and what we learned to inform iterations on subsequent rounds of testing.
In separate a blog post, I’ve written about the subsequent rounds of testing and iteration:
Iterated testing for rapid, rich learning: Researching the UX of Drupal AI Assistants
And in an additional post, I summarise the overall findings, observations and my reflections and describe what’s next:
Making AI useful and usable – learnings from UX research of Drupal AI Assistants
I used a basic scenario and tasks for the pilot
Initial scenario: You run a community group and you’ve taken the first steps to set up a website to keep people informed about what you do.
Task 1: You can publish textual news content on your site but you now need to be able to add images to the news items you publish. How would you use the AI Assistant to make sure you can add images to future news articles you publish?
[Expected result: image field added to news content type]
Task 2: In addition to the news articles, you want to be able to publish longer-form pieces providing information about collaborative community projects you have worked on, to include things like logos and links to the partners you have worked with. How would you use the AI Assistants to help you do this?
[Expected result: new content type created for long-form articles containing relevant fields]
Follow-up task question: How successful would you say the AI Assistant was in helping you achieve this task?
I had identified UX success indicators to enable quick, analysis-informed iteration
In order to try to continually improve the way the Assistants functioned, we worked on the basis that we would make iterations to the testing environment and set-up in between tests, in response to what we learned.
We identified that we might like to make front-end changes to the Assistants (controlling how they displayed information) and also to tweak the back-end (controlling the Assistant ‘persona’ and the underlying guiding instructions describing how the Assistant should work).
To plan for rapid iteration, I needed to be prepared to quickly analyse the data I obtained from the tests on the go. Therefore, in the test design and planning, I had identified several indicators to look out for which would demonstrate a successful UX for the participant. These were as follows:
- Participant has a positive response to being presented with the AI Assistant
- Participant can communicate with the AI Assistant using language they would naturally use
- AI Assistant responds to participants’ prompts with relevant answers (no hallucinations)
- Participant can understand, respond to or act upon the response from the AI Assistant
- Participant completes task with the help of the AI Assistant
- Participant reflects on a positive experience and evaluates accordingly.
What the participant did in the pilot and how the AI Assistant responded
Once they had accessed the test environment using an anonymised log-in, the pilot participant found the AI Assistant (labelled ‘Drupal Agent Chatbot’) and started on the first task.
Task 1: Add in the functionality to add images to news items
The stages in the interaction went as follows:
- Pilot Participant: typed prompt asking for ‘information about adding images to news items’
- Assistant: responded with a numbered list of four instructions (the first stage advising the user to check to see if the existing news content type already had an image field, and later stages advising of the actions to take to add this field)
- Pilot Participant: asked how to check the existing news content type
- Assistant: provided two sets of numbered instructions, the first to check the content type, the second to add the image field if it wasn’t there already, and a closing offer to create the image field on the news content type for the participant.
- Pilot Participant: went to follow instructions, navigating away from the Assistant and clicking into the interface. This action caused the Assistant to close and instructions to clear meaning the participant couldn’t follow them.
- Pilot Participant: asked Assistant to create the image field for them
- Assistant: returned a ‘something went wrong’ message
Task 2: Add in the functionality to publish long-form articles
The stages in the interaction went as follows:
- Pilot Participant: typed prompt asking how to add in a long-form article
- Assistant: responded to say it was necessary to create a specific content type (suggesting ‘Blog’ or ‘Long Article’ content types). It provided three sets of numbered instructions:
- The first set to guide the participant through a process to check if a suitable content type existed already
- The second to guide them through the process to create a content type or modify an existing content type. (assuming one didn’t exist)
- The third set explained how to use the content type to publish content (once created).
At the end of the instructions, if offered help to create or configure a content type.
- Pilot Participant: asked Assistant to help them make the long-form article content type
- Assistant: clarified name and description for the content type, also asked if there were any specific fields to be included
- Pilot Participant: asked Assistant to create a content type called ‘Long Article’ for long-form blogs and articles, and asked it to include quote fields and body text fields
- Assistant: replied to say it was ‘looking up the answer’, then advised that the content type ‘Long Article’ had been created with the fields specified. It offered to help with additional fields and customisation and provided three links:
- One link view and manage the fields in the new content type
- A second link to check the field storage configuration for a field within the content type.
- A third link labelled ‘Details’ which revealed technical specific details of the configuration added into the system
- Pilot Participant: clicked the first two links – each resulted in a ‘page not found’ error, suggesting the task hadn’t been done. When they investigated the interface, however, they found the new content type listed alongside the other content types in the ‘Structure’ section of the interface.
What went well in the pilot and what needed to be changed ahead of the next tests
Since it was the first attempt at testing there was much to be learned from the pilot. I reflected on the results of the test to prioritise iterations to make before the next round of testing.
Scenario, tasks and success indicators all worked well
The participant was able to understand the test scenario and the tasks with ease. Reflecting on the test afterwards, the success indicators served me well to do a quick analysis of what the experience was like for the participant and to pull out things to be changed for the next rounds of testing.
Positive aspects of the user experience
The UX provided by the AI Assistant was successful for several reasons.
Participant liked the Assistant and was able to interact with it effectively
The participant responded positively to the AI Assistant and communicated with it naturally. They said they could see the value of it to help them learn how to do things in Drupal.
The Assistant presented information clearly
The participant commented that they liked the way the Assistant presented the information (as numbered instructions). They were compelled to act upon it.
One out of two tasks was successfully completed
The second task was completed with the help of the AI Assistant.
Areas to improve and iterate for the next rounds of tests
Some aspects were not so positive and we sought to improve these before testing again.
Ensuring the instructions didn’t disappear
To enable them to follow the instructions provided by the Assistant, the participant needed to navigate away from the Assistant and into the interface, but doing so caused the chat history to be wiped. This meant they couldn’t act on the instructions. This could be fixed quickly in the front-end.

The left screen shows the Assistant’s instructions, however, when the participant clicked in the interface to follow the instructions, they were wiped (as shown in the right-hand screen)
Making sure the Assistant could complete tasks
At the end of the first task, the participant asked the Assistant to carry out the work which had prompted an error message. This needed to be fixed as the expected behaviour of the Assistant was to perform tasks, not just provide instructions.
Ensuring the links to check Assistant’s work worked
After completing the second task, the Assistant provided links for the participant to check its work. These links didn’t work so the participant had to check what the Assistant had done by looking at the interface themselves. This broken link issue was also quickly fixable in the front-end.
Avoiding the need for users to check the system settings before acting
In its instructions for both tasks, the Assistant advised the participant to check what was in the interface already. The participant was confused by this, since they felt the Assistant should already know what was in the site. This was fixable in the back-end description of the Assistant.
Avoiding the presentation of too many instructions and options to choose from
In its responses, the Assistant provided several sets of instructions as well as the offer to do the task and in one case technical information. This gave the participant a lot of information to process and react to. The participant felt the technical detail was not useful for them as they didn’t understand it and since the instructions were presented first, only by the second task did the participant realised that the Assistant could complete tasks for them. It was possible to remove the technical detail from the Assistant responses with a quick fix, however, simplifying the number of options was not an instant fix as it relied on sustained training of the Assistant to provide clearer information.
The post-pilot tweaks prepared us for the next rounds of tests
This short pilot test with a single participant provided a very rich source of data. It demonstrated that the scenario and associated tasks were appropriate, and that my UX success indicators were useful for the analysis of the data. With a couple of front-end tweaks made, we were well-positioned to maximise our learning from the next iterations of testing.
Read about the next rounds of testing in my follow-up post:
Iterated testing for rapid, rich learning: Researching the UX of Drupal AI Assistants