Impact Measurement for Systems Change

With the emergence of systems thinking in social impact, the need for measurement and evaluation tools informed by systems thinking is increasingly becoming a necessity. An increase in publications such as Principles for Effective Use of Systems Thinking in Evaluation by the Systems Thinking in Evaluation, Topical Interest Group (SETIG 2018) and the Systems Practice Workbook by the Omidyar Group is further advancing this thought.

The Skoll Centre for Social Entrepreneurship, in collaboration with Forum for the Future, also organised a session at the Skoll World Forum 2018 on measuring systems change, engaging academics, practitioners, and researchers.

Our research team has identified key challenges to overcome to measure the impact of a system focused intervention:

• Time and distributed impacts: Systems change interventions typically require a concerted effort, involving many actors, over a long period of time, during which both challenges and opportunities shift. While the timeline of progress in the social impact sector is slow,  the funding available is short term and outcomes focused.  Researchers and practitioners agree that these arelong-term efforts with no agreed finish line or point at which you can say you are done.

• Attribution: Because of these complex features of  systems change undertakings, it is often difficult to collect consistent, over time data at the ‘systems level’.  This complicates the aspiration to attribute causality to the interventions and to cumulate efforts. There are typically several different steps between the intervention and eventual outcome, along with a large gap in timing of the intervention and the resulting outcome.

• Consensus: We do know from scores of efforts, that a key element is common ground and agreement about some basics of measurement and interpretation.  Building this kind of shared view across a sector is itself a challenge.  But it is crucial to have this broad base of agreement on how to measure systems change. If we do not have consensus on an approach in the social impact sector, it is very difficult to mobilise funding or encourage concerted action.

“There are still disputes about what is social enterprise and what is systems change. Another dispute is how you measure it. You can lose a lot of people from the debate if you decide to measure following one approach.” - Systems Thinking educator working with a Foundation, USA

Key Principles

Our contributors shared some principles that they follow when designing a strategy for measuring systems change.

Integrity of approach: Since we cannot measure change at the system level and progress is slow, we start by looking at the approach of the organisation.

“As an example, if we take a lab that is working on a cure for cancer, suppose they have been working for decades with no cure in sight. This does not imply that they have failed in their mission. However, how do we measure the impact of their work? This requires us to do ‘system sensing’ beyond the scope of the intervention or product. Instead, you measure the quality of the lab and their adherence to scientific protocol.” - Systems Thinking educator working with a Foundation, USA

Similarly, you use a somewhat rigorous way of assessing impact to look at change over time. This change can be non-linear. However, you must spot the ability to adapt and maintain focus on a goal but be open to changing tactics and strategy.

• Guiding star: It is crucial that at the beginning of an intervention, a team has plotted the dynamic of the current system that they wish to change in order to measure the effect of their intervention. After this dynamic has been mapped, they need to envision an end-state that they want to achieve in order to create a guiding star for their intervention.
• Innovation versus transformation: It is important to differentiate between systems innovation, which implies working within a system to create incremental change, and systems transformation, which is working on transforming the dynamic of the system itself. For our interviewees, systems innovations are more provisional and contextual, they don’t necessarily change the status quo. These contrast with the much rarer system transformations, the ones that profoundly transform the system by addressing root causes of problems or by changing how agents relate to one another.
• Spill overs: Funders must determine the spill over effects from engaging in a specific systems change venture. There are other organisations operating in a system that funders can disempower by funding one specific enterprise. What if we back one organisation that inadvertently leads to the failure of nine others? In such cases, we can typically only use estimates.

Metrics to keep in mind

• Custom Metrics: One of our contributors is running a multi-sector coalition developing custom metrics early in the intervention based on a sound theory of change. For them, they measure how many new actors they are bringing into the solution, placed in a system where they can affect change. They also measure how they are impacting funding strains in their problem area.
• Impact levels: You can also differentiate between the levels of impact you are exploring, whether it is short- or near-term outcomes, or impact on more complex system dynamics. As an example, when examining impact, we can look at correlations with better outcomes in the system itself, ripple effects on other outcomes and the non-linear spread of impact. You can also look for evidence that the system itself as gotten healthier in some way. In fact, a key question for funders is, what are the indicators aligning to make deeper progress in making systems healthier?
• Failures: We also must look at people who have failed to achieve their desired outcomes and walked away. This is a critical piece of information on what tactics might not achieve systems change and helps to eliminate self-reporting bias.
• Using a secondary data source: A secondary data source is always helpful to get feedback from outside the organization. If we have self-reported data, we must enquire about independent sources of data on the health of the system. This can also help us find a baseline or control along with self-reported data.

From our contributors, those who are funders expressed a keen interest in the ability to spot early signals from ventures that are geared to create systems change. Identifying proxy-indicators for what makes an organisation more effective at influencing systems and shifting the status quo can help them optimise their funding strategy.

As we can see, impact measurement for systems change is a work in progress. The principles stated above are currently used by our contributors to define their individual approaches to measuring their activities.

Author: Nikhil Dugal is a systems change consultant with the Skoll Centre for Social Entrepreneurship. He is a Skoll Scholar, having completed his MBA from the Saïd Business School in 2018.