Diagram with steps 1, 2, 3. Under 1 arrow "Reach", points right towards "2 Response" which points right towards "3 Uptake".

Measuring Effectiveness of Narrative Change Campaigns

By

Introduction

Explore the three levels of evaluation for measuring the effectiveness of narrative change campaigns – Reach, Response and Uptake. Then once you have an idea of what you want to measure and how to measure it, develop a specific data collection plan and decide how the data will be recorded and shared.

When putting together your evaluation design/plan, you need to start by working back from your campaign objectives. If the campaign has a clear short-term, policy outcome as the goal this process is relatively simple. However, for most campaigners working on narrative change, their objectives are focused on shifting the narrative or “changing the weather” as it is sometimes referred to.

This article is an excerpt from a Guide titled – #KommMit Toolbox: Narrative Change practice to advance social cohesion by the International Center for Policy Advocacy ICPA. This article includes a short case study about the Poppy Hijab Campaign by British Future in the UK.

Levels of Evaluation for Measuring Effectiveness of Narrative Change Campaigns

In putting together an evaluation plan for a narrative change campaign, we propose focusing on three levels as realistic measures that inform your understanding of the effectiveness of the campaign:

1. Reach

Reach –  How many people in the target segment(s) did you reach? How many people in general paid attention? 

In this first evaluation level, you are simply looking at how many people were exposed to the campaign messages, whether they were from your target segment(s) and also how many times they were exposed to your message.

For example, knowing basic estimations and segmentation of readership for your various target publications, viewership for a target TV programme, or attendance figures for a public event you’ve organised or the campaign is part of would help you to estimate the exposure. This was the approach taken in some of the campaigns we have showcased in these guidelines.
 
The third dimension of reach looks at how many times you reached the same audience. According to marketing and political campaign practice, the so-called “Rule of 7” principle significantly increases the chances that the audience will notice and also eventually buy your product or message.

This goes back to the basics of our frames definition and the so-called salience of frames.

Put more simply, people tend to notice and eventually believe the things that are repeated a lot. This is harder to measure accurately, but given that it is a fair assumption that people tend to get their news from the same sources, looking at how many times the campaign got covered in each source would be a good start in this regard.

Possible Data Collection Methods

  • Analytics from website and social media traffic – hits, likes/follows and plays
  • Estimations around readership/viewership for various media that cover the campaign 
  • Attendance figures at public events and meetings

2. Response

Response – Did you get the coverage and responses you wanted? 
 
Moving to the second evaluation level, the focus moves past exposure to how people responded to the campaign.

The target for many campaigns is to have key opinion leaders and ultimately, representatives of the public from the target segments responding positively to the campaign.

Finding out if key opinion leaders have responded well often involves a media and social media analysis and you can get a quick feel for the public response by looking at comments and likes on websites and social media. However, this is also risky as these comments sections are home for trolls of all kinds, who tend to be very active in migration debates.

If your campaign entails offline activities, such as an event of some sort or conducting face-to-face campaigning, you will quickly get an idea of how well the message is playing out and what type of responses the campaign messages induce and trigger. 

Building an evidence base for this evaluation level will combine collecting more structured data from a media analysis process, but also anecdotal evidence from meetings, events and coverage. So, it is worthwhile to have campaign team member(s) record such data on an ongoing basis in a campaign evaluation log that is a shared and easily accessible document. 

Possible Data Collection Methods

  • Analysis of articles and opinion pieces written about the campaign from public and opinion leaders 
  • Analysis of comments, likes and responses on social and traditional media 
  • Analysis of responses from opinion leaders in conferences and meetings
  • Running focus group with the middle 

3. Uptake

Uptake – Are the right people adopting and using the campaign messages or proposing the policy position you put forward? 
 
This is where the ultimate goal of your campaign resides – meaning that you are looking at whether the public discourse has shifted in the way that you wanted it to.

Simple metrics are focused on whether key opinion leaders have started to use your language or talk about the problem and/or solution in the way you describe it.

More challenging metrics involve looking at the language, values and policy options that target segments of the public you are talking about in the discussion – often done using focus groups or polling. 
 
One of the key challenges in this debate is how much credit you can take for any shift in opinion in the public discussion. Obviously, people’s opinions shift based on multiple inputs and in a free and open democratic culture, this is a good thing.

So, the advocacy principle we propose you follow is to look at contribution rather than attribution, i.e. to what extent did your campaign add to a narrative shift, rather than trying to take full credit.

In order to track a campaign’s influence in the evolution of a public discussion, many people use tracer phrases or hashtags that easily flag that some contribution has been made.

For example, in the campaign case by British Future we researched, ‘poppy hijab’ as a phrase or even the artefact itself can serve this tracer function. The timeline for measuring uptake also poses a challenge, as if often takes time beyond the natural timeline of the campaign to see a shift in the public discourse. So, considering some ex-post measurements in the evaluation design is also a good idea.

Possible Data Collection Methods

These methods would be conducted in an ex-post way to see if the audiences have taken on some of your messages, and are talking more positively (or even less negatively) about migration:

  • Analysis of dominant discourse in the mainstream political debates around migration and options being proposed
  • Analysis of migration discussions with the public and opinion leaders from the middle
  • Focus group with the middle segments

Case Study

CASE 1 – Poppy Hijab Campaign – British Future – UK

To challenge growing anti-Muslim sentiment, British Future developed a campaign called the ‘Poppy Hijab’, which commemorates the 400,000 Muslim soldiers who died fighting for Britain in World War 1. Launched around the centenary of World War 1 in 2014, they made a hijab featuring the commemorative poppy and launched it in the commonly migrant sceptical and right-wing newspaper, The Daily Telegraph, under the slogan “Proudly British – Proudly Muslim”.
 
The minimum communication target for the campaign was focused mostly on the response level, i.e. positive coverage of the campaign in target newspapers (The Daily Telegraph, The Daily Mail, The Star) and TV channels (Sky TV) valued by the middle in the UK, who are usually much more sceptical of migration and migrants with Muslim backgrounds in particular. And indeed, they did achieve that with multiple positive articles and good interviews and coverage in many middle-oriented media outlets. So, the reach level could be measured from the standard readerships and viewer numbers. 
 
The uptake level of the campaign is tougher to measure. However, one key result was when the Prime Minister at the time (Cameron) made reference to the poppy hijab in his “Successful, multicultural Britain” speech:
“A country where you can be Welsh and Hindu and British, a country where you can be Northern Irish and Jewish and British, where you can wear a kilt and a turban, where you can wear a hijab covered with poppies…”.
 
Another indicator of uptake of campaign messages that was reported as significant by campaigners was the fact that the association of military veterans in the UK chose to put the poppy hijab scarf on sale in its shop together with other war commemoration artefacts/symbols. This signifies the longer legacy of the campaign and continued uptake of the main campaign messages on an annual basis.

Data Collection

Once you have an idea what you want to measure and how to measure it, it’s time to move onto the specific data collection plan and how the data will be recorded and shared.

Assigning the ‘Data’ Job

Make the data collection/monitoring part of someone’s job on the campaign team. This data collection work is key in implementing the campaign evaluation plan and needs to be part of someone’s job on the campaign team.

Rather too often, the evaluation part of an advocacy campaign is put on the back burner as it is seen more as a project function rather than one directly feeding into the campaign. However, as was illustrated in the previous element, having the monitoring data current and feeding ongoing decision-making is key to having a full picture of the campaign responses and to inform the discussion of potential responses. Of course, it will also provide the basis for the broader and longer-term evaluation process.

Recording the Data

Ensure that data is recorded in evaluation logs or spreadsheets on an ongoing basis.

Getting all the data together over the intended timeline of the campaign will involve team members repeatedly doing media and social media monitoring and recording instances of reference to the campaign or tracer words (such as campaign hashtags) in some type of log.

Also, responses to the campaign will come back more anecdotally from team members who are involved in offline campaign activities and who are engaged with media and policymakers, and these stories also need to be logged. This logging process will also be the basis to do the campaign monitoring described in the last element.

Processing the Data

Develop reports based on the log to stimulate evaluative reflections and discussions.

Going above the raw data in the log, the final stage in this process is the production of a report that reflects on whether targets were met or not and the reasons targets were hit or missed in this instance.

This report will provide a valuable basis for a broader discussion of the effectiveness of the campaign, the potential next steps and what can be learned for further campaigns.

Planning Checklist

Consider the following questions:

  • How will you measure the reach, response and uptake for your campaign?
  • What specific data sources and methods are you going to use to collect data for each level?
  • Which approaches best suit your purpose, budget and data availability?
  • Who will be assigned in the role of the monitor or evaluation data recorder for the campaign?
  • What kind of log are you going to use to record the data you collect?
  • How often should media and social media analysis be done? 
  • How will team members share the data generated in the field and through offline activities?
  • In what kind of report will you share the data and reflections based on the data in the evaluation log? Who will you assign to develop it?
  • How are you going to discuss and process the learnings from the campaign evaluation report?

Access Resource

Explore Further

Share Resource

Share on LinkedIn

Topics:

Collection:

Tags:

Format:

Language:


image Attribution-NonCommercial-ShareAlike CC BY-NC-SA

Contact a Commons librarian if you would like to connect with the author

Topics