In this post, Professor Tim Drysdale discusses how a broader view of assessment, coupled with support for an institutional culture that links educational theory to interventions (and evaluating the outcomes), can create fertile ground for the productive use of genAI in Higher Education. Tim is the Chair of Technology Enhanced Science Education and Director of Strategic Digital Education in the School of Engineering. This blog post is part of the March-June Learning & Teaching Enhancement theme: Assessment and feedback revisited↗️.
Higher education assessment practice was already in need of innovation before generative artificial intelligence (genAI) became mainstream. We cannot ban genAI or prevent its usage. Despite concerns around “labour, data, & inequality”, and the lack of a performant fair-trade alternative, avoiding student usage of genAI would fail to prepare them for the world into which they graduate. Nonetheless, genAI has “exacerbated” assessment issues because of its disruptive new capabilities, such as rapidly generating essays informed by customised information supplied by the user.
A year’s reading in a single prompt
We cannot circumvent the usage of genAI in assessment tasks merely by setting clever questions for our students. Obscure, recent, and unpublished data can all be drawn on by a large language model, simply by providing that data as part of the prompt, even if it is a large amount. Current commercial genAI models can swallow over 700,000 words in a single question – some 49 hours of reading for a typical adult, and much or all of the reading that might be expected of a 10-credit course. The current cost of requesting a 20,000-word essay based on 700,000 words of custom books, transcripts, notes and other text is about the same as a lunch deal in a student bar, and the University might well be picking up the tab anyway.
Not all that glitters is pedagogical gold
There are already educational technology products emerging to offer these sorts of writing capabilities to students, along with features such as automatically summarising assigned texts and generating multi-choice questions for self-testing. The resulting educational approaches are limited in scope and already at odds with what we know about good education, but there is little if any critical evaluation on offer from suppliers. The disconnect between the educational literature and properties of commercial edtech will likely worsen in the rush to supply products to a sector that is itself, not yet sure, what it needs or how it will evolve. However, the educational literature is clear in which directions we should (and shouldn’t) go, if supporting learning is our main goal. Let’s start with where we shouldn’t go.
Proctoring and exams are not the solution
Observing students so as to dissuade them from using genAI is tempting to many as the lowest-cost, quickest solution to the immediate problem of reinforcing the validity of current practices. Unfortunately, remote proctoring for coursework is highly problematic and must be circumvented to avoid the harm it causes. Nor can we retreat to a reliance on in-person invigilation of hand-written and digital final exams. As well as assessing only a narrow slice of a broader education, an over-reliance on final exams would compound the Russell Group’s challenges with feedback because it entrenches the assessment of learning, after the learning has been done. Thus, a broader view of assessment is called for, which leads us to where we should go.
Assessment for learning
To many, assessment means “grading”; an evaluative act undertaken after learning has occurred to score or rank the learning outcomes. Yet those outcomes are unpredictable and inconsistent even when the courses are, “planned with great care, delivered effectively, [and] engages students.” With students reaching different understandings “within minutes,” let alone being in the same place at the end of a course, a one-size-fits-all learning and summative assessment leaves more students under-served as our cohorts grow in size.
On the other hand, assessment used regularly or even continuously during a course can be biased away from grading and toward helping students navigate toward the learning outcomes that have been set for them. This is known as ‘assessment for learning’. While it will undoubtedly help in some cases, we don’t need AI-enabled tools to address this shift so we can get started right away. The main call to action is to concentrate on linking educational theory to interventions, and evaluating the outcomes.
Below, I offer some cases that directly or indirectly use (or don’t use) AI to illustrate this point.
1. Assessing iteration with formative feedback
Many learning processes are based on iteration, but are typically assessed only at the end-point; perhaps also a mid-point if resource allows. For digitally-mediated activities, such as interactive simulations, virtual activities and remote practical work, there is a rich data stream that can be tapped to provide continuous feedback to students. We recently showed students are twice as likely to complete a remote laboratory task if they can see a simple, live, graphical representation of their actions relative to the teacher’s suggested approach. With no AI training needed, this tool can be deployed in small or large classes as soon as the teacher has done the exercise themselves, and it is fully transparent in its working. Going further, text-based emotion detection may maintain privacy while allowing virtual assistants to offer hints or activities that take a student back to flow, and away from confusion or boredom.
2. Personalised / adaptive learning
While staff should set the intended learning outcomes, students bring their own starting place. Personalised tutoring systems can help students find their own route more successfully. However, there is likely to be tension between the minimum viable product of a virtual tutor that simply provides the correct answer, versus one that engages in drawing the student out through an inquiry learning process, especially as genAI is not expected to understand and reason with our mental models. It may also require us to map our curriculum in more granular detail, perhaps closer to the micro-curricula model, which will allow for a diversity of sources for each element. This will help students with scheduling study and deadlines, and assist staff with adaptively planning group learning activities to better suit the cohort on the course. Even with issues around reasoning capability, if framed correctly, it is likely that virtual tutors can add value with text-to-speech capabilities, such as:
• Acting as facilitators to individual and group discussions;
• Acting as avatars to ask anonymous questions on behalf of students;
• Provide live (private) feedback to lecturers and teaching staff when they seem to have skipped a logical point, become unclear or inconsistent;
• Analyse video of apparatus that students are constructing in laboratories to give feedback on their experimental methods (without surveilling the students themselves).
3. Experiential learning
Reflective assessment is essential to experiential learning – without reflection, how do we know if the experience resulted in any learning? Reflective learning is considered somewhat expensive to implement in terms of staff time, yet I’m given to understand there are already AI-guided reflective processes used at the end of some synthetic scenario learning approaches. AI agents are not currently considered to have reflective capabilities. However, it is the reflection process in the learner that is what matters, and the AI agent may assist in marking.
Perhaps the value will be in the use of an adaptive genAI-driven dialogue, which can help students learn to reflect, and draw out enough reflective material from them. This material can then be summarised for review by a teacher. This reflective process can happen multiple times throughout a semester, with staff time being allocated to those students who can benefit most from intervention. We don’t want to lose the personal touch, but shifting staff time from triaging students to helping them would likely improve outcomes.
4. Authentic scenarios
It’s generally accepted that skills need to be learned whilst embedded in a context, yet placements and internships have limited availability. GenAI can be used to create virtual yet authentic professional practice scenarios, such as mimicing dialogue with hospital patients and colleagues in other professional situations. A synthetic web can be used to train communications professionals about how to handle crisis situations. Similar benefits are expected to accrue in other learning tasks where genAI can be used to enrich the context at comparatively low cost, such as automatically generating individually contextualised projects, topical scenarios or tutorial discussion briefs, and helping students flesh out rich examples of their ideas applied to practice.
Learning to change, continuously
In closing, an increase in digital technology is inevitable, due to a mismatch between growth in student numbers and resources. The sector probably cannot afford to change as fast as we would like, and there is still much to learn about how we need to change, so continual change is in our future. Nonetheless, we can significantly increase our chances of thriving in the future by building our institutional capacity to continuously pilot, scale, and adopt teaching, operational and research approaches that are aligned with, build on, and advance modern educational literature.
Tim Drysdale
Prof Timothy Drysdale is the Chair of Technology Enhanced Science Education and Director of Strategic Digital Education in the School of Engineering. He is also seconded part-time to the University’s Curriculum Transformation Project focusing on digital innovation for experiential learning. His main research activity is in Engineering Education (Key Research Area 10 for the School), where he leads the Remote Laboratories group. He and his team have developed an entirely new infrastructure and approach for operating online remote laboratories on traditional campuses (practable.io), winning international awards from the Global Online Laboratories Consortium (Remote Experiment Award 2024) and the Association for Learning Technology / Jisc Award for Digital Transformation in 2023.
