Louis Thurston’s work (circa 1927) developing the ‘Law of Comparative Judgment’ demonstrated that people are very good at making accurate relative-value judgments between two choices but less reliable when assigning absolute values within a scale of measurement. Adaptive Comparative Judgment (ACJ) is the 21st century version and is now fully facilitated online, making the process logistically feasible at scale. Put simply, Adaptive Comparative Judgement asks markers to compare assessments in pairs to establish a scale of relative quality. ACJ demonstrates much greater reliability in large-volume assessment when using a distributed team of assessors and also enables high numbers of students to efficiently receive valuable feedback from several assessors, whether used for staff assessing students or student peer assessment.
The University has been exploring the potential in the use of computer software and online tools as part of our technology-enhanced learning strategy to help our students develop an understanding of how to use assessment as a powerful aid to learning. Over the last couple of years, we’ve piloted the use of ACJ with students undertaking the Edinburgh Award to both formatively and summatively peer-assess each other anonymously, while also providing key learning points in their feedback to each other.
Students’ submissions are uploaded to an online system used for the peer assessment. Reviewers, in this case students themselves, are then presented with two submissions on which to make a comparative judgement – essentially deciding which submission better fulfils the set criteria. Once the student reviewer has made their judgement and left feedback, they are then presented with a new pair of submissions on which to make their next comparative judgement. This process happens repeatedly and, behind the scenes, the system uses an algorithm to rank all submissions based on the outcome of these comparisons.
“As part of the Edinburgh Award I took earlier this year I had the opportunity to use ACJ. I was really impressed. I used ACJ twice during the Edinburgh Award, first for us to see and comment on other students’ work and for others to provide me with feedback on ways that I could improve my work, and then again at the end of the Edinburgh Award, where ACJ allowed me and my fellow students to peer assess our final work.
ACJ is really impressive at supporting the learning process as it gives you, as a student, insight and perspective on your work that you wouldn’t normally get if it was just your tutor providing you with the feedback. You get to see the different ways that other students have tackled the same problems, different perspectives within the same context, not just from within your group, but right across the cohort, which is really helpful. The feedback from your peers is also richer, more personalised, more contextualised, more targeted than your tutor is able to give you given the huge time constraints that they are under. It really helps you develop your skills, both in your work and your general analytic skills, which will prove really useful after university in the workplace.
ACJ is like a crowd-sourced/social media style feedback and assessment tool, which is really innovative and very powerful. I can honestly say that it has been one of the greatest learning experiences for me during my academic studies at the University.”
Briana Pegado (final year student 2013/14 and EUSA President 2014/15)
The success of these early trials encouraged further pilots on several large mainstream courses across the University to evaluate the impact this may have on students in developing a deeper understanding of how to effectively use assessment and feedback in their learning.
The innovative use of ACJ within the Edinburgh Award, received an award in the ‘Best use of Social and Collaborative Technologies for Learning’ category of the eLearning Awards 2014. Within the Edinburgh Award, we continue to look for new ways to enhance our provision and provide students with the best possible learning and development experience. All suggestions welcome!
Reference: Thurstone, L.L. (1927). A law of comparative judgement. Psychological Review 34. 273-286.
Find out more about ACJ at the University of Edinburgh:
Employability Consultancy slides on ACJ and the Edinburgh Award
Using ACJ in the School of Physics and Astronomy
Dr Gavin McCabe leads the University’s Employability Consultancy. With a background in statistical epidemiology, international development, careers advice and higher education projects, he is now responsible for supporting institutional strategy and initiatives relating to students’ employability, development and graduate attributes, as well as fostering associated local-level activities and enhancements. Current areas of responsibility include: the Edinburgh Award, SLICCs (Student-Led, Individually-Created Courses), Making Transitions Personal and the Development Hub.
