Category: benchmarking

Developing a Framework for Innovation Intermediation

matjazvidmar/ / benchmarking, evaluation, impact, Innovation, methods, outcomes, policy, Post, process, research, science

My exciting journey with the Innovation Caucus started one rainy morning in Spring 2017, when by chance I spotted an advertisement for internship applicants doing the rounds over email. This was followed by an email from my supervisor, asking all of his PhD students if we have seen the call and whether we were interested. Not being someone who declines any opportunity, my reply was immediate – yes!

Having found out about the Innovation Caucus and its work some months previously, when putting together a notice for the departmental newsletter about our engagement with policy, I was really excited by the opportunity to further translate my research interest into useful knowledge for policy-making. Having applied and made it through to the interview, I was ecstatic! Speaking to Tim and his team was interesting and inspiring, and once I was offered the internship, it took even less time than before to say “yes” and accept it.

As I am really passionate about my PhD research topic (social aspects of technology development and innovation) and my subject matter (Space Industry – yes, the stuff “up there”) I took quite some convincing to take on new challenges within the Innovation Caucus brief. In part, this was because I really wanted to create a new space of shared knowledge and sense-making, i.e. to challenge the theoretical concepts with empirical findings and policy realities – and I could only envisage doing so within the topics about which I was already somewhat knowledgeable.

However, in discussion with Innovate UK and the Economic and Social Research Council (ESRC), I did eventually reshape my interest into developing a broader framework and typology of innovation intermediation in any geographically bound sectoral system of innovation. This was of value to Innovate UK, since it supported the ongoing development of their portfolio of Catapults and Knowledge Transform Networks, as well as other projects and policies.

This experience was a great lesson for me, not only in working with and delivering for a policy-making system, but also in expanding my own research interests into domains I did initially find uncomfortable. Presenting the headline findings of this work at one of the most prestigious innovation conferences in the world, DRUID (2018), helped me appreciate the power of broader generalisation of academic knowledge, in order to achieve more substantial societal impact.

The lessons learned and experiences from this project also enabled me to engage better with new concepts, unfamiliar settings and unknown stakeholders in my subsequent work. For instance, these skills have proved critical in working on a consultancy project for the OECD and as a Research Assistant in academia.

I have to express my big thanks to Tim and his team for their support and mentorship and to all involved with the Innovation Caucus, particularly Innovate UK and the ESRC teams involved with my internship. It was your determination and generosity that turned this project from a 3-month desk-job into a transformational professional journey.

lxvchcxo_400x400

This post has been published in October 2018 at Innovation Caucus blog: Developing a framework for innovation intermediation.

Find out more about Innovation Caucus.

Cost Benefit Analysis: “What Have the Romans Ever Done for Us?” – ER3

matjazvidmar/ / benchmarking, cost benefit analysis, evaluation, impact, outcomes, Post, research

Cost benefit analysis is an attractive evaluation method, as it can provide concrete, often quantified, data about interventions, usually in a form which is easily communicated to the clients, policy makers, funders and the general (lay) public. In its core and at its best, cost benefit analysis is a very direct and straightforward evaluation process, whereby inputs and outcomes are weighted against each other and logical conclusions about the efficacy of a programme can be reached.

However, all three of these elements – inputs or costs, outcomes or benefits, and efficacy or the relationship between the two – are highly contestable. To begin with, defining your parameter space and acknowledging constrains and assumptions is the key element of this approach to evaluation. These decisions, even if very well argued for, are ultimately just decisions; a global cost benefit analysis, if such a thing was ever possible, would need to encompass much of the factors and effects left on the other side of the dividing line for the evaluation to be a true representation of the net impact of the programme.

Secondly, even though the aim is to have a quantified data as possible – best if every input and impact are turned in some sort of monetary measure – both costs as well as benefits are often indirect or intangible. In Cellini and Klee’s most stark example (2010, p. 500): what is “the value of wilderness or an increased sense of community”? Furthermore, even if a measure can be put to notions such as wellbeing, another – perhaps most challenging of all – decision has to be made, namely what ratio between costs and benefits defines effectiveness of even efficiency?

However, in my limited experience, cost to benefit analysis is effective if the intervention being evaluated is narrow and well defined in terms of the available resources, the scope and the intended outcomes, or better still, when all of the above have an intrinsic monetary value attached. The intended outcomes I look for in my research are related to innovation and consequently increased economic activity, contributions to GDP, business growth, job creation, etc., hence quantification of these parameters is not very difficult as they often come as monetary values to begin with.

The most challenging for me is to benchmark the efficacy of this cost to benefit ratio and, to be honest, even though it would be to a degree possible to put a judgment on how significant the benefits have to be to deem a programme a success, I prefer to correlate these ratios to background trends such as global economic activity, comparisons to global GDP growth, global business and job creation, and add qualitative data where possible, as I believe the later provides a broader judgment on how the intervention is impacting those in and close to it.

This advanced cost benefit analysis can then feature prominently in a new paradigm of impact evaluation – the Correlated Quantified Impacts (QCI) – the topic of the next post.