Dr Francesca Calo recently joined the OU as a lecturer and is working closely with the Centre for Voluntary Sector Leadership. Francesca’s research focuses on the topic of evaluation – an important issue for all third sector organisations.
Read Francesca’s introduction to her recent blog on this topic below and follow the link to the OU’s Department of Public Leadership and Social Enterprise (CVSL’s academic ‘home’) to read the blog in full.
The last 12 months have been challenging for all of us, but most have been challenging for organisations and people that deal with vulnerabilities every day. During COVID-19, the role of public policies and the importance of public sector organisations, voluntary sector organisations, social enterprises in our lives have become even more clear and prominent. We heavily rely on public policies, public services, voluntary sector organisations and social enterprises services to address health and societal problems, exacerbated by the pandemic. These actors will be hopefully at the forefront of reconstruction of our societies after COVID-19. The evaluation of their impact will become even more relevant after COVID-19 when resources hopefully will be invested in players with social purposes.
In 2009, Alex Nicholls titled his article, in which he discusses the emergent reporting practices used by social enterprises, with the question “We do good things, don’t we?”. After more than ten years and a context that has changed quite a lot in the last ten months, it is the same question that resonates in my mind when I think about social enterprises and more broadly non-profit organisations, specifically in relation to their potential role in the world post Covid-19.
Before Covid-19, social enterprises have been tasked with competing for, and delivering, health and social care contracts on behalf of the state (Alcock et al., 2012; Hall et al., 2012) on the perception that they provide higher levels of innovation, cost-effectiveness and responsiveness (Bovaird, 2014). Despite this rhetoric, the evidence that provision by social enterprises is ‘better’ than available alternatives is notoriously weak (Calò et al., 2018). While the evaluation of interventions in health and social care has arguably become increasingly more sophisticated, this has not been the case where social enterprise is concerned. However, the evaluation of the impact that social enterprises have, will become even more relevant after Covid-19 when resources should be invested in players that can do “good things”.
In a study that I carried out with colleagues at Glasgow Caledonian University, we assess the potential of three methodological approaches common in health evaluation – a systematic review, realist evaluation and quasi-experimental investigation – and we apply them to the complex realm of social enterprises. Systematic reviews which collate and assemble all the up-to-date empirical evidence to answer a specific research question (Shemilt et al., 2010), have been considered a robust form of research because, if undertaken correctly, they are believed to increase the generalisability of results and assess the consistency of evidence (Mulrow, 1994). Realist evaluation focuses on how a specific intervention works, for whom, and in what circumstances (Pawson and Tilley, 1996), and is designed to identify the combination of generative mechanisms and contextual characteristics in achieving outcomes (Fletcher et al., 2016). The quasi-experimental investigation aims to assess the effectiveness of specific health interventions (Craig et al., 2008) and determines what works in terms of measurable outcomes (Fletcher et al., 2016; Moore et al., 2015)
We faced different challenges when applying these approaches to the social enterprise realm. Two were common to all the methods employed.
Reflecting upon the limitations of the three methods used in this study, two main lessons can be drawn that could be useful for policy-makers, researchers and practitioners. The first relates to choosing the most appropriate comparator, the second relates to the possibility of generalising results linked to the appropriateness of employing methods commonly used in public health in the context of social enterprise.
If in the future, after Covid-19, policymakers aim to understand the added value of social enterprise organisations, an integrative research approach based upon the context combining different research methods and design should be implemented to improve generalisability. Selection of comparator groups (if a comparator group is important to address the research question) should be based on an in-depth analysis of the context in which the social enterprise is embedded, and the comparator group chosen should be based upon the policy aims the evaluation wishes to address. Evaluate that “we are doing good things” doesn’t come without a cost. Each of the methods adopted in our paper was time-consuming and resource-intensive and require if the researcher to possess advanced skills. Public officials should then recognise the complexity and resource-intensive nature of such evaluations, and resource it accordingly. Only in this way, the question “we do good things, don’t we?” will be addressed.
Image of Glasgow Caledonian University Library by 老爺的圖書館.
17th February 2021