Message Testing Methods for Narrative Change

Introduction

Is your NGO engaged in narrative change campaigns? Here are 7 methods to test your messages from the International Center for Policy Advocacy ICPA.

While the narrative change approach holds a lot of promise, the reality of building a campaign approach with an attitude change goal that works for a particular context/audience presents significant challenges and risks. This is why empirically-tested narrative strategies are at the heart of ICPA’s work to support & deliver change in often polarised debates like migration & civic space.

This resource synopsises the testing methods we have experimented with over a seven-year period and have proven to be effective. It is not an exhaustive taxonomy of the full range of methods available. It rather provides first level practical answers to questions & concerns we regularly hear from NGOs engaged in narrative change campaigns:

• Is testing really feasible/affordable for non-profit orgs?
• How do you do such testing & which methods work best?
• Is testing really worth the effort?

The practical orientation in this guide assists campaigning organisations to make initial decisions on a suitable testing approach that fits their needs, capacity and budget.

It is worth noting while the focus of this guide is on individual methods, there is an emerging consensus for the need for more longer term/longditudinal monitoring and testing on key public attitudes.

What is Message Testing?

In simple terms, message testing is a process of getting feedback from your intended target audience on a campaign strategy, copy or content to see if they get the expected responses.

This valuable feedback allows campaigners to adapt content before launching and during campaigning work. In project design terms, message testing provides essential data mainly for the formative evaluation stage of campaigning and community engagement work and can significantly inform longer-term evaluation & learning.

Why Test?

When the goal of a campaign is changing the attitude of a more sceptical target audience, there is simply more risk you can get it wrong or worse still, even make the situation worse. So, there is a need to invest in understanding what is working (and not) through testing. You can then adjust as you go to build a proven campaign approach that has the best chance of achieving your goals.

The first goal is to figure out what is NOT working. When we target audiences we are not so familiar with, we sometimes build messages and stories based on assumptions that turn out not to resonate with the audience.

Further, campaigners should not risk their content/messages triggering the opposite responses to those intended and actually hardening attitudes (triggering the so-called ‘backfire effect’).

Testing works especially well in identifying those elements that are not working or even backfiring, so you can make informed decisions on what to remove or adjust based on this feedback. On the more positive side, taking an iterative, test & learn approach will help you adjust and finalise the messages and material that are working and scale up from there to maximise impact.

This approach will also build a rigorous evidence foundation to prove your campaign strategy before rollout and such evidence can be a strong basis to mobilise support for your initiative and the narrative change approach for the longer term.

What and When to Test?

A wide range of material is commonly tested including pitches, top line narratives/messages, stories & protagonists, slogans/hashtags, visuals/memes, video material and website content.

We have found it useful to plan testing at two stages in campaign development:

1. Concept level 
   Once you have identified a target audience/segment of the public and developed your messaging/pitching approach, and even some draft content, it is useful to see if your strategy is working before spending large amounts of money on expensive production costs.
2. Content level 
   This is the more traditional understanding of when to test, i.e. once images, video and campaign content has been developed and you are trying to identify which content and messaging works better.

But if you can’t be so strategic for whatever reasons, it’s important to remember that “any testing is better than no testing”.

Overview of 7 Common Message Testing Methods

In each of the tables we provide

• an overview of each testing method
• the data you get from each test
• when’s the best time to use each particular method
• the capacity needed to effectively employ the method
• high & low-cost options for each method
  

The tables are an attempt to answer the first level practical questions that NGO partners commonly ask us. We have also provided some basic cost estimates for each, so you have some ballpark idea of the range of costs in Europe in 2022/2023. What is not provided is detailed step by step advice on implementation.

To see the Testing Methods Tables access the website and/or PDF.

• Testing Method 1 – Focus groups to get rich emotive group responses
• Testing Method 2 – Opinion polling on messages to robustly test messages & triangulate results
• Testing Method 3 – Randomised Controlled Trials (RCT) to measure attitude change with confidence
• Testing Method 4 – A/B testing on Social media to pick options that work better with your audience
• Testing Method 5 – A/B testing in email lists to pick options that work better for your audience & supporters
• Testing Method 6 – Online Bulletin Boards to get audience responses over a longer time
• Testing Method 7 – Test & learn through paid ads to find what works at scale & adapt

Campaign Evaluation Approach

If you are interested in how we apply these methods in a campaign evaluation approach, take some time to look at the following:

• A campaign evaluation and testing case study from Germany
• Our overview of how to build an empirically tested narrative change strategy and campaign.

Access Full Resources

• Message Testing Methods for Narrative Change – Website
• Message Testing Methods for Narrative Change – PDF

Explore Further

• How to Test your Communications Guide
• How to Measure Narrative Change
• Measuring Narrative Change: Understanding Progress and Navigating Complexity
• What Makes Narrative Change So Hard?
• Measuring Narrative Change: Moving From Theory to Practice
• What Makes Narrative Change So Hard?
• Impact Narrative Tool
• How to Change the Narrative / Story: Guides, Worksheets and Templates
• Narrative Change: Start Here
• How Do We Know If We Have Transformed Narrative Oceans? Here’s how we’re starting to answer the persistent question about narrative change evaluation
• Narrative and Storytelling: Training and Planning Tools
• Changing Our Narrative about Narrative: The Infrastructure Required for Building Narrative Power
• How do you Measure the Impact of Activism and Advocacy Work?