In this post, Dr Vassilis Galanos continues his exploration of metrics, arguing that the passive acceptance of a metrics-oriented culture is what feeds, establishes, and normalises hype and high adoption rates of Generative Artificial Intelligence (GenAI) machinery. This post is part 3 of 3, and belongs to the Hot Topic theme: Critical insights into contemporary issues in Higher Education.

In the previous two posts (Higher Education State Critical and Rigorously Established Fear), I argued that, in Marxian terms, the surplus value generated from intellectual labour in academia is enhancing the institution’s reputation and funding. This shift is slowly reversing the map (numerical indices) with the territory (learning and teaching experience), using the latter to navigate the former, instead of the opposite. In this post, I argue that the passive acceptance of this metrics-oriented culture is what feeds, establishes, and normalises hype and high adoption rates of Generative Artificial Intelligence (GenAI) machinery, such as OpenAI’s ChatGPT, Microsoft’s Copilot, or Anthropic’s Claude.

This is a fast-forward historical recap of what preceded the emergence of GenAI. Student grades, attendance, and staff’s citations fuel the academic-industrial complex by fostering connections between businesses ready to absorb the highly-graded students while partnering with high-ranking research initiatives. This relationship turns human intellect into marketable products, since the relationship is mutually parasitical: as much as the industry wishes to present an outwardly-facing, scientifically and moral high-ground with the approval of the academy,  while the latter wishes to benefit by collaborations with the industry that increase revenue, prestige, and can be presented as societal impact.

This process is reminiscent of the commodification trends in industrial capitalism, where labour was quantified and valued based on its contribution to profit. Social media metrics further entrench this commodification, transforming intellectual achievements into social capital. Shoshana Zuboff calls this “behavioural surplus” – a term that might as well fit within the academic landscape. Students and staff self-regulate, ever aware of the metrics that loom over them, dictating their academic behaviour. This is the darker side of the notion of the quantified self the vision in which people would continuously track in order to optimise their performance (and well-being) that fell into decay once the profit-driven motives of self-tracking industries were sufficiently experienced, from health apps to selfie share, in most cases training customised advertisement algorithms and facial recognition software, often used for military and policing purposes.

This self-tracking culture fits perfectly into academia’s metrics obsession. Students monitor their grade point average (GPA) like investors tracking stocks, and researchers obsess over their h-index scores like they’re anxiously awaiting show reviews. While waiting for these longer-term affirmations, theye both gain temporary satisfaction through social media interactions in secret hope one of their outputs (research, business, or otherwise) will become viral – indeed, the algorithm for virality in social media and academia might be very similar.

From what has already been mentioned, GenAI visions uplift this obsession to an extreme level. Initially, it presents itself as capable to produce the boring aspects of a text that everyone wishes to avoid, for example, the opening and concluding sentences, the formal proofreading and grammatical/syntactical corrections, the angle ideation, and short explanations for relatively common-sense knowledge. Supposedly, thus, they save time which, in theory, can be used for leisure (of course, in Academia, the concept of “leisure” is very controversial and means different things to different people – should we prohibit the consumption of an academic text while on holidays? For one, we do not prohibit the academic study of leisure, especially if it attracts big grants).

Upon closer inspection, time is not saved at all, especially for those in precarious temporary contracts, or with student loans, or who need a promotion, or on scholarship deadlines. What Generative AI’s time-efficient output may do, is increase the amount of produced content, but without an unchanged time-table (the contracted hours, or the time prior to entering the job market). Students and teachers become mere data nodes, constantly producing text to feed the technical and the social machinery. This aligns with the historical trajectory of technological advancements that have progressively extended the volume and precision of bureaucratic production and control while intensifying the intellectual labour within the same time interval.

And the final Marx quote for the day:

“The shortening of the hours of labour creates, to begin with, the subjective conditions for the condensation of labour, by enabling the workman [workperson] to exert more strength in a given time. So soon as that shortening becomes compulsory, machinery becomes in the hands of capital the objective means, systematically employed for squeezing out more labour in a given time” (Marx 2013: 285).

Endless self-tracking and performance optimisation, powered by GenAI and sustained by social media metrics culture, thus turns the academic journey into a frantic dash for better numbers. While Generative AI claims to offer personalised feedback and guidance, it amplifies the anxiety around self-improvement and harmonising output within “acceptable” frames. Students and staff focus more on meeting target numbers than engaging deeply with their work (or others’), mirroring the constant self-optimisation driven by social media feedback loops.

If, as academics, we also think of ourselves as activists, not merely observing but influencing the politics of what we study (for some, this is inevitable anyway for we cannot suppress our influence – the question is whether we admit it and what we do with it), we should consider how the infrastructures that oppress social groups we wish to defend are entrenched through the criteria we develop and use to measure success and failure.

Words like “success,” “failure,” “impact,” “assessment,” “measurement,” “mark,” “rank,” or “grade” carry legacies of phallogocentrism (the internet is still complete with videos of males measuring their manhood in toilets), imperialism and colonialism, military hierarchy and operationalism, and nonhuman and human enslavement (marked on the flesh by branding iron until today). In this metrics-driven landscape (where “data-driven” is but a euphemism), academia risks becoming a parody of itself. Here, surveillance, commodification, and self-quantification dominate, supported by a broader culture of social media views and reactions that enables as reductionist a thinking as the nine emotions featured in the recent film, Inside-Out 2.

Generative AI, the latest instalment in the history of automated education, intensifies these trends, aiming to squeeze more surplus profit out of education and research, which in turn exacerbates an aesthetic of the safe and acceptable writing we have already established in academic circles. This, in turn, normalises a degenerative culture of unimaginative repetition. Hence, I prefer to call it ‘Degenerative AI’.

My rant is over. I am leaving you with the following song about numbers from the 1969 season of Sesame Street, composed by Denny Zeitlin and featuring vocals by Grace Slick of Jefferson Airplane: https://www.youtube.com/watch?v=G5stWhPNyec

References for Part 1, 2 and 3

Andreski, S. (1973). Social sciences as sorcery. New York: St. Martin’s Press, May.

Archer, M. (2024). Unsustainable: Measurement, Reporting, and the Limits of Corporate Sustainability. NYU Press.

Cixous, H. (1974). Prénoms de personne. Paris: Seuil.

Cixous, H. (1994). The Hélène Cixous Reader. (Susan Sellers, Ed.). Routledge.

Derrida, J. (1979). Spurs: Nietzsche’s styles. University of Chicago Press.

Marx, K. (2013). Capital: A critical analysis of capitalist production (S. Moore, E. Aveling, & E. Untermann, Trans.). Wordsworth.

Zuboff, S. (2022). Surveillance capitalism or democracy? The death match of institutional orders and the politics of knowledge in our information civilization. Organization Theory, 3(3).

Vasileios Galanos

Dr Vassilis Galanos, SFHEA is a visitor at the Edinburgh College of Art and works as Lecturer in Digital Work at the University of Stirling. Vassilis investigates historico-sociological underpinnings of AI and internet technologies, and how expertise and expectations are negotiated in these domains. Recent collaborations involved the history of AI at Edinburgh, interrogations of generative AI in journalism (BRAID UK), artist-data scientist interactions (The New Real), and community-led regeneration interfacing with data-driven innovation (Data Civics). Vassilis has co-founded the AI Ethics & Society research group and the History and Philosophy of Computing’s (HaPoC) Working Group on Data Sharing, also acting as Associate Editor of Technology Analysis and Strategic Management.

In this post, Dr Matjaz Vidmar offers Part 2 of his exploration about the future of the take-home essay as a form of assessment in the era of generative large-language models. Matjaz is Lecturer in Engineering Management and Deputy Director of Learning and Teaching overseeing the interdisciplinary courses at the School of Engineering. This post is Part 2 of 2, and belongs to the Hot Topic theme: Critical insights into contemporary issues in Higher Education.

As explained in Part 1 of this article, generative writing tools need not be feared as the end of take-home writing assignments, if they are grounded on students’ critical reflection in connecting theory and practice. On the contrary, with the increase of group-work and experiential learning models, the take-home assessment has become more common in fields beyond the social science and humanities domains, such as engineering. For example, in the final assessment set in a number of courses I designed and now deliver across management, systems engineering and futures design domains, 60-70% of the final mark is obtained from a final, take-home essay.

However, for this to work at a time when chat-bots can read and write much faster than humans, I have developed an explicit assessment brief to discuss practical experience with respect to the core course concepts and literature. This assessment structure pre-dates the advent of generative writing tools, and is based on the pedagogy of reflexive critical thinking as a major marker of experiential learning and knowledge making. Most importantly, the students are also required to offer some original insight that the practical experience inspired. This leads them beyond the well-rehearsed generic principles from course literature and requires them to examine unique points of view (a dimension where generative algorithms struggle).

In my experience, whilst it is possible for students to articulate a well-structured generative writing prompt that identifies all three key components of the assessment answer (underlying theory, practical experience, and new insight), in doing so, they have, by-and-large, already demonstrated the critical understanding and skills outcomes from the course. Thus, the writing of the text in itself is not as pedagogically relevant (especially in an era where such tools are widely used in any case). Furthermore, by keeping the assignment length reasonably short (<1500 words), the limited scope and required conciseness requires writing skills that ensure the clarity of message, again demonstrating critical thinking and academic practice.

These considerations and observations have also been demonstrated in practice. During marking take-home essay assignments in all of my courses with this mode of assessment, we could only detect a small number of submissions where more extensive use of generative writing tools was likely (<5%). Furthermore, these submissions did suffer from the tale-tail signs of algorithmic syntax, both in terms of the artificial linguistic forms and predicted poor integration of different sections, even where the content was edited to be more or less on point.

Overall, we can note that there was no detectable advantage in terms of quality of writing output. If comparing the mark distribution for the final take-home essay assignment in the course Technology and Innovation Management (5 / MSc), for which we have the required comparable longitudinal data[1], there is no discernible change in student relative performance between academic years 2021-22 (pre-ChatGPT; n=74/63) and 2023-24 (post Chat-GPT; n=77/65). If anything, within the normal boundaries of cohort differentiation, the results from the more recent academic year(s) are slightly worse, despite the teaching team’s clear policy that use of generative writing tools is allowed as means to improve grammar, syntax and text flow.

Overall, this model is demonstrating that, with more focus in course delivery on experiential learning and then framing the take-home essay as an examination of critical reflection, the current generation of generative writing tools do not pose any serious threat to the robustness and integrity of take-home essay as a form of assessment. In addition, if the assessment objectives target explicit linkage of theoretical concepts to a managed, in-class experience, then the intellectual work required to construct a writing prompt for a generative writing tool already meets the core learning outcomes examined by such an assessment.

Having said that, it is nonetheless important that we educate learners about ethical and epistemological issues surrounding large language models within the context of in-class exercises and take-home writing support. On the epistemic side, it is critical to stress that language models are based on statistical patterns of language use, and as such cannot serve as un-checked sources of knowledge.For many widely accepted theories, the two can be strongly aligned. But with newer or more peripheral bodies of literature, the margins of error in accurately representing scholarly insights increase significantly.

On the matter of ethics, two critical issues need to be communicated:

1.  The models have been constructed with inherently biased and often exploitative data practices.
2. Inputting own original ideas into publicly available tools can lead to them being used for future model development / training and thus lack of credit to the originators.

Colleagues have already set the scene for making the most of generative writing tools, proposing both more adaptive teaching practices as well as assessment innovation. Given that it is here to stay, we should focus on making the most of this new technology to enhance the experiential learning process, and reject the temptation to revert to outdated assessment practices, as that would also inevitably make our teaching less relevant to students.

[1] Apart from lowering the word-count (which was planned ahead of time and unrelated to arrival of generative writing tools) there was no change to the assessment brief. Three other courses which I organise, with comparable assessment models (Building Near Futures, Systems Engineering: Thinking and Practice and Social Dimensions of Astrobiology and Space Exploration) also show no discernible advantage to students, though they are new, so there is no pre-generative-writing-tools data. Colleagues teaching Technology Entrepreneurship 5 / MSc, which I set up in 2020-2022 academic year with similar final assessment also report there is no change in results and little use of generative writing tools.

Matjaz Vidmar

Dr Matjaz Vidmar is Lecturer in Engineering Management at the University of Edinburgh and Deputy Director of Learning and Teaching overseeing the interdisciplinary courses at the School of Engineering. He is researching the collaborations within Open Engineering by bridging technical and social dimensions of innovation processes and (eco)systems as well as futures, strategies and design. In particular, he co-leads The New Real programme, a collaboration between the Edinburgh Futures Institute and Alan Turing Institute, experimenting with new AI experiences, practices, infrastructures, business models and R&D methodologies, including the flagship Open Prototyping. He is also the Deputy Director of the Institute for the Study of Science, Technology and Innovation and is involved in many international initiatives to develop the future of these fields, including several start-up companies and an extensive public engagement programme on interplay of STEM, arts, and futures literacy. More at www.blogs.ed.ac.uk/vidmar.

In this post, Dr Matjaz Vidmar explores the future of the take-home essay as a form of assessment in the era of generative large-language models. Matjaz is Lecturer in Engineering Management and Deputy Director of Learning and Teaching overseeing the interdisciplinary courses at the School of Engineering. This post is Part 1 of 2, and belongs to the Hot Topic theme: Critical insights into contemporary issues in Higher Education.

Since the global launch of chat-based access to generative large-language models (most notably, ChatGPT), a sense of technological and moral panic set in across the education sector. In particular, it seems that despite the well-documented existence of “essay mills”, where assignments could be written by hired “professional writers”, we nonetheless relied extensively on take-home written work as form of formative and summative assessment. With the now ubiquitous and freely-available writing tools in the hands of students, a number of colleagues became worried that marks on their courses were unsafe and considered (re)introduction of in-person timed exams as a way to stop what was perceived a wide-spread possibility of “cheating”.

However, is it that perhaps the issue with take-home written assessment is not about a sudden crisis arising from new technological development, but rather with some of our teaching and assessment practices?

In many ways, the sector has been struggling for years with the parallel challenges of the need to grow student numbers to keep up with maintaining expensive real-estate and facilities, whilst at the same time questioning what is the role of (higher) education in the modern information-driven, digital society. As more up-to-date knowledge is increasingly available online, the traditional (formal) forms of higher education have been resorting to focusing on honing in graduate skills and attributes in critical thinking, effective professional practice, and cross-cultural citizenship. Thus, it has become less clear how to identify the most effective and appropriate teaching and learning models.

One promising direction emerged when grappling with more complex and advanced concepts: an emphasis on experiential learning and group / project work. This also follows the leading pedagogical frameworks about practice-based learning and greater integration of individual ways of knowing and learning. The emphasis here is placed on critical personal reflection by connecting practical experience of challenges, and the strategies adopted to address them, with the academic understanding of the issues at stake.

The schematic shown below brings together some of the critical contributions mentioned above into a more holistic approach to learning and teaching, framing experiential learning as the grounding of the development of students through experimentation and experience. However, crucial to the attainment and assessment of deep personal understanding of the learning process is a meaningful reflective observation, connecting the (individual) practice with (collective) sense / knowledge-making.