Louise Ker, Strategic Research Executive, explains the concept of ‘responsible research metrics’, what the University is doing to implement this, and what this means for research and researchers.
The research landscape is exploding with opportunities to harness ever larger and more diverse datasets to deliver powerful progress and innovation. The inner workings of higher education are no different. We don’t need to look too far outside our own backyard to find both opportunities and ethical challenges in using data. Universities themselves now hold significant and ever-expanding datasets on both researcher and student activity.
What do we mean by ‘Responsible use of research metrics’?
Essentially this is about the use of numbers and data analysis in research assessment, particularly at the individual level.
Familiar examples of ‘research metrics’ could include research grant income, citations, or journal impact factor (JIF).
If you are not familiar with JIF, this is a measure of the mean citations to recent articles in a given journal, for example the journal Nature has a very high JIF, as does Science, Cell etc.
‘Responsible Use of Research Metrics’ is about setting out how we will use numerical measures and analysis appropriately and fairly, alongside more qualitative information, in research assessment to best support high quality research.
Why ‘Responsible research metrics’?
Imagine you are a senior Professor, faced with hundreds of applications for a postdoc, or are sitting on a REF panel with hundreds of submissions to assess. How do you assess fairly all the individuals and outputs within the extremely limited time that you have? Do you rank outputs by Journal Impact Factor (or whether they are published in Nature/discipline equivalent high impact journal) as a first cut?
Or what about if you are a researcher, being asked to achieve a particular grant income target, as a minimum performance goal by your institution?
The former is a pragmatic consideration, the latter example has already made the news for all the wrong and tragic reasons. In both cases, using just a single measure cannot reasonably provide a good proxy measure for research quality. Universities, Funders and Researchers need practical and robust methods of research assessment, which recognise, support and reward excellence in research.
The San Francisco Declaration on Research Assessment (DORA)
The focus in recent years in particular on Journal Impact Factor use throughout academia and its well-known short-comings has led to various academic groups campaigning for and developing alternative approaches to research assessment which aim to more fairly assess research activity.
The San Francisco Declaration on Research Assessment (DORA) initiative is the brainchild of the American Society for Cell Biology in San Francisco back in 2012. DORA’s vision is to “advance practical and robust approaches to research assessment globally and across all scholarly disciplines”.
DORA provides a roadmap for global reform in research assessment that universities, funding bodies and individual researchers can sign up to.
What is the view of research funders and other stakeholders?
All 7 of the UKRI Research Councils are signatories to DORA. Funders such as the Wellcome Trust are now imposing conditions of grant requiring evidence of responsible use of metrics. REF2021 has a requirement that submitting universities be able to demonstrate the responsible use of metrics in research evaluation.
What is the University of Edinburgh doing?
The University of Edinburgh wants to support staff to continue to lead the highest international standards for research assessment, and enable the world-class research that drives innovation and benefit for society.
The University has signed the San Francisco Declaration on Research Assessment (DORA).
In parallel to this, the Research Support Office has been leading on developing an initial University statement on the ‘”Responsible Use of Research Metrics”. Central to this is basic good statistical practice: not relying on a single measure (where there’s a signal, there’s noise), recognising known and new limitations of available measurements (data quality and validity), and clearly communicating analysis undertaken (reproducibility).
What does this mean for me as a researcher?
Achieving effective and positive use of research metrics as part of supporting our research activities requires a partnership approach between individuals, Schools, Colleges and the wider University. If you are a researcher at the University you have a critical role to play, including for example:
- Ensuring that all relevant outputs, such as software, articles, IP, policy impact, PGR students, etc are appropriately and fairly recognised through accurate records on Edinburgh Research Explorer.
- Ensuring appropriate attribution of credit for outputs based on scholarly contribution.
- Notifying line managers of any instances of inappropriate use of metrics in line with this policy.
Opportunities for the future
Fundamentally, ‘Responsible Use of Research Metrics’ is about the fair treatment of individuals during research assessment, and ensuring that research assessment recognises what we most value in research.
During the development of the Statement, we ran two focus groups for representative academic staff from all three colleges. We addressed some of the practical challenges, such as within recruitment and panel assessment, and the ideas across the group covered approaches such as encouraging the use of ‘biosketches’ where applicants or panel submissions are encouraged to summarise key achievements/outputs in a short paragraph, or encouraging usage of a ‘basket of research metrics’ as the norm when considering purely quantitative data.
The publication of the Statement represents the first step, and this work will continue, with the University having recently set-up a working group on Research Metrics to oversee this.
Funders, and other relevant bodies are also working hard in this developing space. ‘What works in research’ is a fascinating area of research all on its own – see our post on research analytics. The University of Edinburgh, as one of the highest research power Universities in the UK is uniquely positioned to take forward new initiatives in this space and I am looking forward to seeing how we can best develop and support this work to the benefit of our researchers and the global research community.
Further Reading
- The San Francisco Declaration on Research Assessment (DORA)
- The Leiden Manifesto for Research Metrics
- The UK Forum for Responsible Research Metrics
- Wellcome Trust new Open Access Policy and position on responsible metrics
Louise Ker, is Strategic Research Executive in the Strategic Research Development team in the Research Support Office.
More from this author
