Posts

Impact Measurement for Systems Change

With the emergence of systems thinking in social impact, the need for measurement and evaluation tools informed by systems thinking is increasingly becoming a necessity. An increase in publications such as Principles for Effective Use of Systems Thinking in Evaluation by the Systems Thinking in Evaluation, Topical Interest Group (SETIG 2018) and the Systems Practice Workbook by the Omidyar Group is further advancing this thought.

The Skoll Centre for Social Entrepreneurship, in collaboration with Forum for the Future, also organised a session at the Skoll World Forum 2018 on measuring systems change, engaging academics, practitioners, and researchers.

Our research team has identified key challenges to overcome to measure the impact of a system focused intervention:

  • Time and distributed impacts: Systems change interventions typically require a concerted effort, involving many actors, over a long period of time, during which both challenges and opportunities shift. While the timeline of progress in the social impact sector is slow,  the funding available is short term and outcomes focused.  Researchers and practitioners agree that these arelong-term efforts with no agreed finish line or point at which you can say you are done.
  • Attribution: Because of these complex features of  systems change undertakings, it is often difficult to collect consistent, over time data at the ‘systems level’.  This complicates the aspiration to attribute causality to the interventions and to cumulate efforts. There are typically several different steps between the intervention and eventual outcome, along with a large gap in timing of the intervention and the resulting outcome.
  • Consensus: We do know from scores of efforts, that a key element is common ground and agreement about some basics of measurement and interpretation.  Building this kind of shared view across a sector is itself a challenge.  But it is crucial to have this broad base of agreement on how to measure systems change. If we do not have consensus on an approach in the social impact sector, it is very difficult to mobilise funding or encourage concerted action.

There are still disputes about what is social enterprise and what is systems change. Another dispute is how you measure it. You can lose a lot of people from the debate if you decide to measure following one approach.”

Systems Thinking educator working with a Foundation, USA

Key Principles

Our contributors shared some principles that they follow when designing a strategy for measuring systems change.

  • Integrity of approach: Since we cannot measure change at the system level and progress is slow, we start by looking at the approach of the organisation. 

As an example, if we take a lab that is working on a cure for cancer, suppose they have been working for decades with no cure in sight. This does not imply that they have failed in their mission. However, how do we measure the impact of their work? This requires us to do system sensingbeyond the scope of the intervention or product. Instead, you measure the quality of the lab and their adherence to scientific protocol.”

Systems Thinking educator working with a Foundation, USA

Similarly, you use a somewhat rigorous way of assessing impact to look at change over time. This change can be non-linear. However, you must spot the ability to adapt and maintain focus on a goal but be open to changing tactics and strategy.

  • Guiding star: It is crucial that at the beginning of an intervention, a team has plotted the dynamic of the current system that they wish to change in order to measure the effect of their intervention. After this dynamic has been mapped, they need to envision an end-state that they want to achieve in order to create a guiding star for their intervention.
  • Innovation versus transformation: It is important to differentiate between systems innovation, which implies working within a system to create incremental change, and systems transformation, which is working on transforming the dynamic of the system itself. For our interviewees, systems innovations are more provisional and contextual, they don’t necessarily change the status quo. These contrast with the much rarer system transformations, the ones that profoundly transform the system by addressing root causes of problems or by changing how agents relate to one another.
  • Spill overs: Funders must determine the spill over effects from engaging in a specific systems change venture. There are other organisations operating in a system that funders can disempower by funding one specific enterprise.  What if we back one organisation that inadvertently leads to the failure of nine others? In such cases, we can typically only use estimates. 

Metrics to keep in mind

  • Custom Metrics: One of our contributors is running a multi-sector coalition developing custom metrics early in the intervention based on a sound theory of change. For them, they measure how many new actors they are bringing into the solution, placed in a system where they can affect change. They also measure how they are impacting funding strains in their problem area.
  • Impact levels: You can also differentiate between the levels of impact you are exploring, whether it is short- or near-term outcomes, or impact on more complex system dynamics. As an example, when examining impact, we can look at correlations with better outcomes in the system itself, ripple effects on other outcomes and the non-linear spread of impact. You can also look for evidence that the system itself as gotten healthier in some way. In fact, a key question for funders is, what are the indicators aligning to make deeper progress in making systems healthier?
  • Failures: We also must look at people who have failed to achieve their desired outcomes and walked away. This is a critical piece of information on what tactics might not achieve systems change and helps to eliminate self-reporting bias.
  • Using a secondary data source: A secondary data source is always helpful to get feedback from outside the organization. If we have self-reported data, we must enquire about independent sources of data on the health of the system. This can also help us find a baseline or control along with self-reported data.

From our contributors, those who are funders expressed a keen interest in the ability to spot early signals from ventures that are geared to create systems change. Identifying proxy-indicators for what makes an organisation more effective at influencing systems and shifting the status quo can help them optimise their funding strategy.

As we can see, impact measurement for systems change is a work in progress. The principles stated above are currently used by our contributors to define their individual approaches to measuring their activities. 


Author: Nikhil Dugal is a systems change consultant with the Skoll Centre for Social Entrepreneurship. He is a Skoll Scholar, having completed his MBA from the Saïd Business School in 2018.

Strategies for Implementing Systems Change

Initiatives using change strategies in the tradition of Collective Impact (CI) work on the levers of systems change and collaborative action, while funders such as Co-Impact are seeking to find a new way to fund systems change activities.

At the SCO, we spoke directly to actors engaging in systems change to capture a brief snapshot of the strategies that are salient to them in their day-to-day practice. The following list represents a distilled version of the topics that were raised:

Collaborative Change:

Often, the problem to be addressed is so large and complex that it is outside the scope of a single organisation to solve it.

As a single organisation, you pick a geography and a service, and you grow bigger and bigger and attract more resources. In order to create a systems focused organisation, you go big at the beginning.”

– Social entrepreneur running a multi-sector coalition, USA

One of our contributors shared that their journey started with a focus on growing one organisation, but they were leveraging their learning to tackle a problem faced by a billion people. This led them to start an organisation informed by collaborative initiatives and alliances looking to solve a problem in its entirety. It was critical for them to be a neutral broke/r, so they spun out of a social venture into a multi-sector coalition solving the same problem but with a new set of tools. Now they are bringing the government and private sector into the solution to solve for scale.

While a major focus in this space is on toolkits, building relationships in this space to work in collaboration is equally important.

“Changing systems is a lot about realignment of relationships within a system. Connecting new people to people they were not working with before.”

– Systems Innovation Expert, UK

Field Building:

Another key activity that a systems change venture undertakes is building the sector and focus on the problem itself.

One of our contributors started by asking where they can add value in the field. They made the case that their target problem was a global development issue and put a spotlight on it with a front-page article in a major news publication. They continue to advance the work by discussing how to solve the problem, building a roadmap to reach 1.5 billion people. This is a role that no single organisation could fill.

Understanding Power:

It is critical to be aware of the power you hold in the system. A contributor running a financial inclusion venture stated that they spent significant time as a small player building an ecosystem in the geographies in which they worked. Over time, their organisation and the sector grew, making it more appealing for major incumbents in parallel sectors to enter the space. Due to their power in the system because of easy success and visibility, the venture came to be part of  an exclusive group of key policy professionals working to grow the sector and establish best practices at the policy level in multiple countries across an entire continent.

“It about power, which is often not talked about. Are you aware of your power, the system and how you can influence this change?”

– Systems Innovation Expert, UK

For a note on power to cause harm, see point below on Impact Measurement under ‘Funding Systems Change Activities’.

Taking a backseat:

There is a delicate balance between being a servant and a thought leader. Both skills are required to undertake systems change. This requires a lot of listening and developing a positive reputation for being more of a listener than a talker.

“From studying organisations that manage to shift systems, we know that they identify what needs to happen in the system and then let go of being the ones to do it.”

– Director of a Social Entrepreneurship Incubator, UK

Iterative learning:

Experimentation and iterative learning are also key. This work requires engagement in constant learning from various approaches used. This also prompts us to recognise the limited control any one actor has when trying to change a system. Unanticipated changes can also change systems and we have to be cognisant of that.

Funding Systems Change Activities:

With the increasing importance of ‘field catalysts’ that galvanise systems-change efforts of disparate stakeholders to achieve population-level change, how do we fund these systems change efforts? Our contributors noted several interconnected elements needed for funders to effectively support systems change efforts. 

  • Impact Measurement: “Donors have to get better at understanding they are affecting a system just by putting money into a problem. You’d want to know the positive and negative impact of getting a grant or investment. How can we know the potential negative impact?  How do we account for negative impact?” –  Systems Thinking Educator working for a Nonprofit, USA
  • Understanding Systems: “A lot of prominent programmes in the sector are talking about deploying large sums of money for single organisations. This is a positive direction, but we have to observe how the capital that is deployed fits into the definition of systems change?” – Director of a Social Entrepreneurship Incubator, UK
  • Funder Selection: “Selecting your investors and making sure that you have the right fit for the kind of systems work you want to do is very important. Are they willing to spend money on the systems changes that need to happen in that product or service market?” – Social entrepreneur running a multi-sector coalition, USA
  • Capacity Building: “How do we prepare foundations for the type of work they want to do? At the end of the day, if programme officers define who gets funding, how can they think about this differently? We see a need for a thought partner to donors on how this affects their grant making.” – Social entrepreneur running a multi-sector coalition, USA

Designing a system change intervention requires consideration of a range of strategies that are system-focused and enacted from the beginning of an intervention. The values of field-building, collaborative change and ‘taking a backseat’ are prominently applied by practitioners. Practitioners also note the value of understanding their role and power in the system in which they work, and how important it is to pick the right funder for implementing their systems change activities.

Author: Nikhil Dugal is a systems change consultant with the Skoll Centre for Social Entrepreneurship. He is a Skoll Scholar, having completed his MBA from the Saïd Business School in 2018.

Do Social Entrepreneurship and System Change follow the same principles?

This is the latest blog update from the Skoll Centre’s main research initiative, the Systems Change Observatory.

Many organisations, platforms and coalitions have reported examples of systems change in action. These include the Finance Innovation Lab, Future of fish and EYElliance. However, these organisations do not fit into the model of for-profit social entrepreneurship. Their approach is cross-disciplinary, with a focus on collaborative action and building coalitions of actors from each system they intend to change.

“While implementing a solution inordinately changes systems, the approach to the programme design and its guiding principles are a key differentiator.”

– Systems thinking educator working with a Foundation, USA

In our last post, we discussed the distinctions and overlaps between shifting systems and scaling a solution. This includes focusing on root cause analysis and shifting a system permanently in the long-term. From these discussions another thread emerged – differences in the principles which underline social entrepreneurship and systems change. Although the terms can overlap, there are a few distinctions to note. These include:

Scaling impact vs. problem solving

As a social entrepreneur, there is a tension between growing an organisation’s impact and working to solve a problem.  Our contributors noted a difference in working at the system level and at the enterprise level. One of our contributors emphasised that as a ‘systems entrepreneur’, they spun out of a social venture to not just grow one organisation’s impact but solve the population-level problem.  When engaging in systems change, work must be focused on analysing and mapping the system with the intention to create systemic change.

“Should you grow impact or solve a problem in its entirety or both? [We are] bridging the gap between social entrepreneurship and systems change.”

– Social entrepreneur running a multi-sector coalition, USA

Role in the Ecosystem

Social entrepreneurship fulfills a different role in the ecosystem as well. The focus is typically on a specific venture or organization, while systems entrepreneurship is more likely to focus on making connections between different elements of a system to address the target problem.

“…you sit in a different place in the ecosystem. One of the benefits of that is we have very different conversation with institutions like WHO, USAID, Game changing conversations”

– Social entrepreneur working in the technology sector, India

Conflict of Interest

Our contributors noted that there is a tension between creating returns for investors and taking systemic approaches that require collaborative action. This implies that a lot of systemic interventions fall outside the scope of for-profit social enterprises that are in-turn responsible to their shareholders. Spending time away from the business also has a disconnect with funders and investors.

“In the marketplace you are trying to win, building the ecosystem is a tension as a for-profit company.”

– Social entrepreneur working in the technology sector, Zambia

Capacity

Another driver of systems change is capacity. An organisation has to be structured to achieve systems change and cannot achieve this if it decides to create systems change partway through fragmented approaches and interventions due to the factors listed above.

Commonalities

Our contributors also shared some common principles between system change and social entrepreneurship. These include:

  • Experimentation: While vision and mission setting are extremely important to know what change an organisation wants to achieve, the process is also constantly evolving as approaches and market segments are tested and flexed. As you engage in both activities, you often encounter roadblocks that you did not expect, and you create new ways to navigate around them.
  • Building on your Expertise: While trying to shift the system in multiple ways, entrepreneurs identify approaches that are well outside of their capacity. Instead, they develop a theory of change and target the problem indirectly with an approach that fits their expertise.
  • Complexity: The complexity of the system at play can be different for different businesses and target problems. Some cases require policy change and collaboration with the public sector, while some are more focused on changing how information is shared in the system. For each of these cases, you have to create the right tools fit for the purpose.

As we can see, there are several distinctions and overlaps in the principles defining social entrepreneurship and systems change. For-profit social entrepreneurship traditionally focuses on an organisation and its impact, while a venture that intends to create systems change follows a more collaborative approach to change and holds a different power dynamic with actors in the system.


Author(s): Nikhil Dugal is a systems change consultant with the Skoll Centre for Social Entrepreneurship. He is a Skoll Scholar, having completed his MBA at Saïd Business School in 2018.