Beyond Vanity Metrics: Toward better measurement of member engagement

Introduction

We all want to know if the work we do is making a difference. But while “vanity metrics” such as list size or pageviews sound big and impressive, they can be misleading. Ultimately, they can lead to flawed decisions that doom membership-driven organizations.

What are the pitfalls of “vanity metrics”? Why do we rely on them? What are some alternatives? In short, are there better ways to measure how engaged members are with your organization? These are just a handful of the questions tackled in the new report, Beyond Vanity Metrics: Toward a Better Measurement of Member Engagement, presented by Citizen Engagement Lab and the Mobilisation Lab at Greenpeace. In this report, you’ll find:

• The five common pitfalls of vanity metrics that create potential danger for people-powered organizations, and why we rely on them anyway.
• The two key questions to ask about any key metric to understand how it influences your work.
• The difference between the “Starters” and the “Seekers,” the two broad categories that most online campaigning organizations fall into, and how to tell which one best describes you.
• Where to go next, from existing alternatives to potential starting points and advice for moving forward.
• Concrete tips and questions to ask yourself to change the way your organization approaches metrics, alter the metrics you rely upon, and transform how you engage your membership.

Key Learnings

Key Learning: 5 Pitfalls of Vanity Metrics

1. Flawed understanding, leading to poor decisions
2. Short-sighted decision making
3. Bad staff incentives
4. Failure to engage members, leading to unsubscribes and people tuning out
5. Organizations that don’t live up to potential

Key Learning: 3 Lessons

1. There is no “perfect metric.” Every single metric has flaws or biases your thinking in some way.
2. Your mission should define your metrics. The best metric comes from a deep, shared understanding of the organization’s theory of change.
3. Get the sequencing right. For example:
   1. Define your organization’s mission.
   2. Understand what it will take to achieve that mission.
   3. Identify key metrics for measuring progress towards mission.
   4. If necessary, identify and track “leading indicators.”
   5. Regularly assess your metrics, and don’t be afraid to change them.

Key Learning: Alternatives

There are no “perfect metrics” but some starting points include:

1. Cohort Analysis: Rather than looking at all members as one unit, breaking into cohorts of people who share a common characteristic over a certain period of time (i.e. # of actions members have taken).
2. Mission-Centered Metrics: In certain situations a raw, aggregate number can still be useful if there is a clear connection between volume and a desired outcome calling them “mission-centered metrics” (i.e. # of people who attended a rally or “total victory experiences,” the # of members who have been a part of a winning campaign).

Access the report

Download Full Report

View Beyond Vanity Metrics on Slideshare

About

Authored by veteran campaigner Colin Holtz, with advisors Jackie Mahendra and Michael Silberman, Beyond Vanity Metrics is based on methodology that includes in-person information gathering, an online survey, and in-depth interviews with leading practitioners. Beyond Vanity Metrics is a project of Citizen Engagement Lab and the Mobilisation Lab at Greenpeace.