Background
Our petition tool now has the ability to allow members to sign our petitions through Facebook, rather than entering their email and other details directly. This is a new tool so we ran an AB test to assess its effectiveness.
Results of experiment
| Version | Participants | Facebook signatures | Non-Facebook signatures | Total signatures |
| Control | 100,383 | 78,739 | 78,739 | |
| Experiment | 99,856 | 7,524 | 68,540 | 76,064 |
The total number of signatures produced dropped by 3.4% when Facebook was included as a method for signing the petition.
I have also broken down the results based on whether the person clicked onto the page from Facebook.
| Version | Facebook signatures | Non-Facebook signatures | Total signatures |
| Control, Facebook | 11,388 | 11,388 | |
| Experiment, Facebook | 2,235 | 8,995 | 11,230 |
| Control, non-Facebook | 67,381 | 67,381 | |
| Experiment, non-Facebook | 5,290 | 59,573 | 64,863 |
For Facebook-sourced signatures, the gap between control and experiment drops to 1.4%, compared to 3.7% for non-Facebook-sourced signatures.
Conclusion
There were hopes that the ‘sign with Facebook’ option would lead to more signatures, which clearly has not taken place. While there is some extra value from users signing with their Facebook account, it appears that there is a statistically-significant decline in the action rate.
This decline seems significantly smaller for those visitors who have come directly from Facebook, suggesting that this issue is less of a concern for people in that category, but there is still no increase in signatures amongst this group.
We are interested in whether other organisations have done similar testing, or have found a way to use ‘Sign with Facebook’ to improve results.
