Introduction
There are often unique challenges when it comes to measuring the outcomes and impact of advocacy campaigns, which traditional methods don’t always account for. This article summarises core tips to help you plan your evaluation in a way that supports rather than constrains the long-term work of building power and changing complex systems.
Advocates, organisers and campaigners usually have limited resources to make big changes in the world. When we’re up against wicked problems or powerful oppositions, we want to know that what we’re doing is making a difference. That’s why we need to embed cultures of evaluation into our work and our organisations.
When integrated with our work effectively, evaluation tells us what we are doing, how well we are doing it and what difference we are making.
Evaluation can help us to be accountable to our communities, plan more strategically, and build a culture of learning in our organisations.
However, many traditional evaluation tools and methods have been developed in the context of direct service delivery and project-oriented work. There are often unique challenges when it comes to measuring the outcomes and impact of advocacy campaigns, which traditional methods don’t always account for. This has sometimes made campaigners wary of wasting time on gathering ‘vanity’ metrics instead of ‘doing the work’. But there are ways of doing evaluation well, even in contexts of complex social change.
Luckily, evaluators, campaigners and organisers have been working together to address these challenges. There are more and more resources available that provide advice and methods for doing evaluation in advocacy and organising.
Below we’ve distilled the core themes from this literature to help you plan your evaluation in a way that supports rather than constrains the long-term work of building power and changing complex systems.
Lesson 1: Evaluation Methods need to Account for how Social Change Happens
Advocacy often involves working towards long-term goals that can’t be accounted for in short funding cycles. Timeframes can be unpredictable in these contexts (Coffman, n.d.; Coe & Schlangen, n.d.). It can be hard to prove you’re making progress when the ultimate goal is so far in the future.
Policy, legal and electoral change can take years, and so it’s important to track progress towards these goals, for example through building power and support (Engler, Lasoff & Saavedra 2019). When planning your evaluation you can break goals down into smaller interim measures to assess your progress while keeping long-term change in view.
Tip: Break your long term goals down into smaller interim measures to assess your progress along the way to long-term change.
However, even if we break our goals down into shorter-term interim measures, external factors and political contexts can make measurement more complicated for advocates and organisers. In his movement action plan, for example, Bill Moyers demonstrates how social movements often ‘take off’ and enjoy a surge in support after key trigger events. Inevitably, once the energy from these moments has worn off, a movement is likely to experience a drop off or contraction in support.
If we’re only measuring raw numbers of participants or supporters, without taking these political contexts into account, we might view these natural ebbs and flows as indications of failure. Engler, Lasoff and Saavedra (2019, p. 55) recommend accounting for the lifecycles of mass protest movements when building our impact measures by not just measuring the changes in raw numbers of participation, but looking at the changes in baseline: ‘is this higher (in terms of activity, fundraising, organisation) than prior to the last period of intensified activity’?
Once you’ve established a baseline, it becomes easier to see whether your long-term supporter numbers are increasing, while still accounting for short-term fluctuations in numbers. The risk of not taking these broader contexts into account when choosing our metrics and indicators is that our evaluations won’t be able to give us an accurate view of our impact.
Tip: How might external events and contexts influence the outcomes of your campaign? Think about how your chosen indicators can account for this context when conducting an evaluation.
The long timeframes for social change and the complexity of external political environments can also make it difficult for advocates and organisers to prove that their actions led directly to particular outcomes when conducting evaluations. Moreover, there are usually multiple groups working across an issue, with no one actor solely responsible for change.
No matter how rigorous our methods, it’s often only possible to demonstrate that a campaign contributed to change, rather than being the sole cause of that change. This is what evaluators call ‘contribution over attribution’ (Coffman, n.d.; Innovation Network 2009). Being transparent about these limitations is important and does not decrease the value of an evaluation. Instead, it demonstrates an understanding of the multiple pathways it takes to achieve social change.
Tip: In contexts of complex social change, it’s often only possible to demonstrate contribution rather than attribution: that our campaign contributed to change but was not the sole cause of change.
Lesson 2: Evaluation Frameworks need to Understand Power in its Various Forms
Advocacy and community organising work don’t exist in a vacuum. You are always operating in a broader ecosystem of power relations. We therefore shouldn’t try to be apolitical in our evaluations. Instead, it’s important to consider power in its multiple forms.
Katie Fox (2019) describes how the various forms, levels, spaces and expressions of power can influence evaluation. For example, understanding the three expressions of power ‘power to (power as capacity)’, ‘power with (power as relationships)’ and ‘power within (power as individual agency)’ can provide us with key frameworks for assessing power-building efforts.
What metrics might you use to understand capacity, relationships and individual agency in your evaluations? Understanding power over can also help us to keep the broader structural forces that influence our campaigns in view (Fox 2019).
Tip: If you want to get smarter about power in your evaluations, have a look through our resources on power here.
At the heart of this analysis is understanding the social view of power. Power is not a one-way force, but exists across people, institutions and structures. Because power is complex and dynamic, social change often doesn’t follow a pre-determined path.
Our evaluations therefore need to be as adaptive as our social change strategies. Fox recommends a developmental approach to evaluation. Developmental evaluation is an adaptive approach that provides feedback in real time as change is unfolding. Building an understanding of power into our evaluation frameworks means that evaluation can help us to understand how change happens under what conditions.
Tip: Social change isn’t always linear, we often need to adapt our strategies mid-campaign. There are different evaluation frameworks, such as developmental evaluation, that allow you to respond adaptively while still generating valuable evidence that you’re on the right path.
Lesson 3: Evaluation as Power-building and Capacity-building
…making winning the singular focus of advocacy efforts—which many of us have done for years—can miss the opportunity to produce more fundamental and lasting change. Policy or systems change wins (or losses) should be viewed as interim steps toward the end goal of building people power.
– Julia Coffman ‘Supporting Advocacy Ecosystems to Build Power’
Some evaluators recommend rethinking how we account for power in our goals altogether. In traditional models, we often view building power and relationships as an interim step towards the ultimate goal of changing policy or legislation. However, Julia Coffman and Katie Fox have both suggested flipping this logic: seeing a policy win as the interim step towards the ultimate goal of building power (Coffman 2023; Fox 2019).
This is because ‘advocacy strategies that build people power are more likely to sustain, and to further build on, wins after they occur’ (Coffman 2023). If you don’t prioritise building long term power, it’s just as easy for a policy win to be overturned in a few years. In this way of thinking, a policy win is not the end of the road, but instead an indicator of a community’s growing power and influence. This means we need to make sure our evaluation questions and measures help us to understand how well we are building power, relationships and leadership, not just achieving changes in policy, legislation or electoral outcomes.
Tip: In order to have a clear evaluation you need a clear theory of change. What assumptions do you have about how change happens? Where do power and relationship-building sit in this?
Lesson 4: Avoid Indicator-led Thinking and Data Extractionism in Evaluations
It can be easy to get caught up in choosing indicators and diligently collecting data for our evaluations. But Coe and Schlangen (n.d.) caution that indicator-led thinking can ignore the messy and unpredictable way that change happens. They argue that prioritising indicators can lead to an overemphasis on measuring activities rather than outcomes –
- how many actions have we run?
- how many people attended?’
- RATHER THAN: what change have we contributed to?
Moreover, Bennett (2021) argues that nothing should be measured without action being taken from that measurement: ‘no input without output’. How is your evaluation (and the data you are gathering) actively guiding decision-making? If you are simply gathering data for the sake of it, this can fall into the territory of ‘vanity metrics’.
Tip: Before you start gathering data make sure you understand why you have chosen a particular outcome or impact measure. How are you going to use that data for learning or to change practices?
It is also important to remember that data are never neutral. The data we gather, how we interpret it and how we put it to work has distinct sociopolitical ramifications. Data can help to perpetuate structures of oppression such as racism, misogyny, homophobia and transphobia. This is because we use data to guide our decision-making. Bad data means bad decision-making. Where technological developments have accelerated processes of data collection, the move towards the monitoring and categorising of citizens has been linked to longer traditions of governance and control.
Isin and Ruppert (2015) have linked contemporary practices of data collection with the history of ‘counting, categorising, and ordering’ that was at the heart of colonial logics of power and knowledge. These practices disproportionally affect historically marginalised and racialised communities. Some researchers have outlined the ‘extractive logic’ of data collection and how these practices can sustain the ‘matrix of domination’ (Vera et al. 2019; Collins 1990; Costanza-Chock 2020).
…no data practice can fully escape the pull of non-innocent relations – the embeddedness in racial capitalism and colonialism – that shape data. – Vera et al. 2019
Evaluators are not immune from these logics.
When designing our evaluations we need to think critically about what data we are choosing to gather about whom. Evaluators have a responsibility to centre equity and antiracism both in the design of evaluations and the communication of results.
Lesson 5: Use Participatory and Justice-based Methods
One way to reject data extractionism in our evaluation models is by centring participatory practices. Our methods of evaluation should support rather than constrain our social justice aims (Klugman 2010; Borgman-Arboleda, Clark n.d.). This means including a range of stakeholders at all stages of evaluation, from planning through to dissemination of findings, and giving change makers the space to redesign evaluations to better suit their needs and principles.
Rather than trying to fit social justice groups into a typical evaluation model, it can be much more powerful for groups to become involved in re-defining how evaluation works, in partnership with evaluators who share an understanding of social justice. – Borgman-Arboleda & Clark, n.d. “Considering Evaluation”, p. 5
In fact, the Equitable Evaluation Initiative (2023) recommends going a step beyond ‘participatory’ principles towards ‘reciprocity’ in our evaluations. They offer three questions to help move your evaluation towards reciprocity:
- How might you step into conversation that acknowledges interdependence?
- How might you co-create accountability and allow for mutual benefit?
- How might relationship, trust, and shared understanding be fostered?
Equitable Evaluation Initiative (2023)
Tip: Involve different stakeholders at different stages of the evaluation, from planning through to communication of results. Elevating different voices in the process ensures that your evaluation is designed to meet the needs of people, not just metrics.
Final Thoughts…
Evaluation is about embedding a learning mindset into our campaigns and social movements. Social change is complex, non-linear and takes time.
As we have shown throughout this article, this can create unique challenges for evaluation, particularly for those using advocacy and community organising approaches to change. But evaluation is not ancillary to social change, it is social change.
Social change makers identify problems in the way things are currently done and seek to fix those problems. This necessarily requires an evaluative mindset. Building a better world requires us to question not just the injustices we see outside ourselves, but also our own approaches and outcomes.
Being reflective in our approaches to change allows us to think critically about our impact not just on policies and institutions, but also on our communities. Avoiding such reflection risks perpetuating the very harm we hope to change. At its most basic, evaluation is essentially a practice of humility, reciprocity and reflection, all of which are necessary to social change.
Tip: Introducing evaluation into your group or organisation can start small, for example by adding a few reflective questions into your regular meetings.
Explore Further
- Campaign and Social Movement Evaluation: Start Here
- Three Tools for Advocacy Evaluation
- Three Guides for Campaign Evaluation
- How do you measure the impact of activism and advocacy work?
- Campaign Monitoring and Evaluation: Measuring what matters, by Maxwell J Smith
- Measuring Impact with Three Social Movement Strategists
- Disrupt Burrup Hub: Campaign Evaluation