Category: cost benefit analysis

Quantified Correlated Impacts (QCI) – ER4

matjazvidmar/ / cost benefit analysis, evaluation, evidence, impact, logic model, methods, outcomes, Post, research, value

To start building a (more) coherent picture of impact evaluation in science and technology programmes, we need to look for a constellation of many different methods to provide a meaningful insight into the need for, and success of, an intervention. Consequently, evaluation research is/should be organisational modus operandi, rather than a set of separate top-level exercises.

I propose a new paradigm in impact evaluation of investment and development in science, technology and innovation, namely Quantified Correlated Impacts (QCI). This approach is based on both quantitative as well as qualitative data collection, as bibliometric and econometric figures are correlated with ethnographic methods – interviews, focus groups and surveys – to determine the perceived causal contribution of the different factors, with particular focus on those pertaining from the intervention.

At it’s core, QCI are underpinned by a logic model; which is connecting the intervention with the evidence justifying the planned outputs; and leads form the inputs through action towards short-, mid- and long- term outcomes.

LogicModelBlog4EvalRes
Logic model for the proposed networking strategy in UK/Scottish Space Sector

As part my research project, for example, I am involved in an intervention driving economic growth in the UK/Scotland through stimulating the collaboration across the UK/Scottish Space industry by increasing sectoral networking. This is a particularly important part of my research in business incubation in the Scottish space sector and the related (sectoral) systemic properties, such as institutional framework, networks of actors, and knowledge creation and dissemination (following Malerba’s Sectoral Systems of Innovation approach (Malerba, 2005)). Also, there is a wealth of evidence about the importance of networking for the success (and growth) of small businesses (Brüderl and Preisendörfer, 1998; Ostgaard and Birley, 1996).

The suggested action to generate these positive effects is to support the growth of small to medium sized businesses by integrating them in a wider network across the sector and wider. This will be facilitated by the creation of, and enrolment of actors into, an on-line database/forum/platform to provide easy access to contacts. Having established that, there are also provisions to host networking events (thematic or generalist), to solidify the ties and introduce more actors into the network, particularly from the non-core businesses.

In terms of evaluation, key facilities need to be established prior to the beginning of the evaluation of outputs (database and its uptake, and the networking events). The database growth can be analysed quantitatively (i.e. number of enrolled individuals, organisations, etc), while the networking event qualitatively (i.e. interviews, feedback, ethnography).

The key next step is to tie the intervention with the outcomes/impacts through an advanced cost benefit analysis. In the example given, this can be done by analysing the investment made with respect to the growth and revenue of the companies most interconnected within the newly established network, comparing to the more peripheral ones, or ones outside the network.

The last part is the crucial correlation, which provides tangible benchmarking for the overall success of a programme (within the cost benefit analysis). This is done by comparing the noticed trends in key parameters (in our case job creation, revenue growth, etc.) with corresponding regional, sectoral, national or global trends. The key objective is to trace any significant difference which can then be (in part! – see below) attributed to the intervention.

Crucial information, however, comes from the collected qualitative data which maps the action to its value for the participants, i.e. what was the contribution of a specific intervention to the overall change. For instance, in the example above we investigate the effect/importance of the networking on business success. This data can only be obtained by interviewing the participants in networking events, and running surveys and focus groups with representatives of the companies/individuals on the database. The key questions to ask will be: What made the difference?; How?; and How significant was it? We can then comment on the part the intervention played in the difference found between the participants performance and correlated trends.

Overall, this approach enables the evaluator to marry the desirable clarity of cost benefit analysis, where standards of success/failure can be contested, with a more balanced set of criteria and tangible links. The key features are quantified data (engagement figures, costs, returns, growth, etc) about the intervention, which is qualitatively (interviews, focus groups, etc.) examined as a contribution towards the difference in participants’ performance with respect to correlated background trends (sector growth, national job creation, GDP, etc.) – revealing the impact of the programme.

As said, this new, Quantified Correlated Impacts (QCI), framework is currently under development and I am sincerely opening its tenets to comments and suggestions. (And, please, do have a look at the other posts in the series, too: ER1, ER2, ER3.)

Many thanks in advance!

Cost Benefit Analysis: “What Have the Romans Ever Done for Us?” – ER3

matjazvidmar/ / benchmarking, cost benefit analysis, evaluation, impact, outcomes, Post, research

Cost benefit analysis is an attractive evaluation method, as it can provide concrete, often quantified, data about interventions, usually in a form which is easily communicated to the clients, policy makers, funders and the general (lay) public. In its core and at its best, cost benefit analysis is a very direct and straightforward evaluation process, whereby inputs and outcomes are weighted against each other and logical conclusions about the efficacy of a programme can be reached.

However, all three of these elements – inputs or costs, outcomes or benefits, and efficacy or the relationship between the two – are highly contestable. To begin with, defining your parameter space and acknowledging constrains and assumptions is the key element of this approach to evaluation. These decisions, even if very well argued for, are ultimately just decisions; a global cost benefit analysis, if such a thing was ever possible, would need to encompass much of the factors and effects left on the other side of the dividing line for the evaluation to be a true representation of the net impact of the programme.

Secondly, even though the aim is to have a quantified data as possible – best if every input and impact are turned in some sort of monetary measure – both costs as well as benefits are often indirect or intangible. In Cellini and Klee’s most stark example (2010, p. 500): what is “the value of wilderness or an increased sense of community”? Furthermore, even if a measure can be put to notions such as wellbeing, another – perhaps most challenging of all – decision has to be made, namely what ratio between costs and benefits defines effectiveness of even efficiency?

However, in my limited experience, cost to benefit analysis is effective if the intervention being evaluated is narrow and well defined in terms of the available resources, the scope and the intended outcomes, or better still, when all of the above have an intrinsic monetary value attached. The intended outcomes I look for in my research are related to innovation and consequently increased economic activity, contributions to GDP, business growth, job creation, etc., hence quantification of these parameters is not very difficult as they often come as monetary values to begin with.

The most challenging for me is to benchmark the efficacy of this cost to benefit ratio and, to be honest, even though it would be to a degree possible to put a judgment on how significant the benefits have to be to deem a programme a success, I prefer to correlate these ratios to background trends such as global economic activity, comparisons to global GDP growth, global business and job creation, and add qualitative data where possible, as I believe the later provides a broader judgment on how the intervention is impacting those in and close to it.

This advanced cost benefit analysis can then feature prominently in a new paradigm of impact evaluation – the Correlated Quantified Impacts (QCI) – the topic of the next post.