Decoding methods for effective social impact measurement
With the increasing global and local focus on ESG, corporates are under immerse pressure to be transparent around the sustainability of their operations, impacts on the environment, how they address the social issues in their context as well the governance thereof.
This article draws on Tshikululu’s experience and expertise built over the past 25 years around useful data collection mechanisms to facilitate meaningful measurement of social impact and reporting of sustainable value creation both for accountability and decision making. It also contextualises the key concepts of triangulation in data collection, attribution and contribution that every social investor needs to understand as the foundational building blocks for credible social impact reporting.
Effective social impact reporting relies on robust Social Impact Measurement (SIM) that uses trialed and tested, as well as innovative methods and processes to demonstrate social change and where possible attribution of the changes to implemented activities[1] lie. Tshikululu utilises various methods ranging from mixed methods approaches, monitoring of outcomes as part of the ‘Monitoring as Evaluation’ approach, to the Development Assistance Committee (DAC) evaluation criteria of The OECD (Organisation for Economic Co-operation and Development) and logic models or log frames.
Sound data collection: Triangulation – examining data from different points of view
Once the major task of setting up relevant ESG and sustainability metrics is concluded, a critical consideration is ensuring data quality throughout the monitoring cycle. A key component of effective data collection is the use of mixed methods, multiple data sources and different data collection strategies to develop a comprehensive understanding of a phenomena under investigation[2], [3], [4] i.e., Triangulation. Examples of data collection strategies that are used for triangulation purposes include: desktop research, project document review, in-depth interviews, focus group discussions and surveys as well as various participatory research methods that include qualitative approaches that involve the direct collaboration of researchers and stakeholders.
Community mapping and needs assessments are a particularly useful approach not only to identify thestrengths, challenges and opportunities of a community but also to inform ESG strategies and reporting on social impact and sustainability metrics. Having conducted several community needs assessments across the country on behalf of our various clients, Tshikululu cannot understate the benefits of this approach.
Social impact: Understanding attribution vs contribution
With the increasing pressure for companies to ascribe demonstratable impact to their ESG related projects, being versed in the concepts of attribution and contribution in impact measurement is non-negotiable.
Contribution analysis can help companies understand a project’s contribution to the observed results through a process of interpretation of causation, rather than determining causation through comparison to a counterfactual control group. In essence it relies primarily on project monitoring data comprising inputs, activities, outputs and outcomes and anecdotal impact to test the theory of change against what has been observed through routine monitoring efforts. It also considers the influence of other external factors.
As an example, a project focused on improving the livelihoods of young people through the creation of waste recycling opportunities within a company’s value chain, would have to test the inputs, activities and outputs of the project and consider all the applicable external factors in order to determine the project’s contribution to the improvement in the livelihoods of those young people.
Attribution assessment on the other hand, differs to contribution analysis in that it utilises empirical research methods requiring a systematically selected sample (i.e., size estimates based on a 95% confidence level and 5% margin of error and appropriate data analysis procedures to measure the impact through causality. It also entails a counterfactual in the form of a comparator such as a baseline (pre-intervention) assessment or control group to ascertain what the outcomes would have been in the absence of the project.
Using the waste recycling example, a baseline or a control group of similar young people that are not in the project would need to be set up to make causal inferences, that participation in the project leads to definite improvements in the livelihoods of all young people, controlling for external factors as well as similar projects that young people may have also been exposed to. This type of analysis will allow a company to claim with certainty the impact of their project but is the more resource intensive option assessing impact.
In working with social investors over 24 years, Tshikululu has grappled with triangulation, contribution and attribution as important concepts and methods for efficient social impact measurement and reporting. In order to bring rigor to the monitoring, measurement, and reporting around the ‘S’ in the ESG, it is important for concepts of Triangulation, Contribution and Attribution to be at the forefront of ESG project design in order to achieve social impact.
[1] https://www.sopact.com/impact-measurement
[2] Mertens, D.M. & Wilson, A.T. (2019). Programme evaluation theory and practice: A comprehensive guide. New York: The Guilford Press
[3] Patton, M. (1999): Enhancing the Quality and Credibility of Qualitative Analysis.
[4] Noble, H. & Heale, R. (2019). Triangulation in research, with examples. Evidence Based Nursing, 22 (3), 67-68