By Ben Raue
GetUp has recently implemented a new tool that allows us to run A/B tests in the background of all of our pages without needing to interfere with the structure of email sends.
As the first test of this new technology, we implemented an A/B test on the “donate” button on all of our donation pages.
At the moment, our “donate” button by default is blue. In the experiment, half of all visitors to donation pages were served a button coloured orange.
There are graphical changes that can be made to our donation buttons that result in larger donations and more donations.
Approximately 57,000 visits were recorded to donation pages as part of this experiment.
There was a slightly higher rate of conversions on the experimental page (orange button). The average size of donations was also larger. These two factors combined to increase the average amount of money raised per visit from $1.71 for the control to $1.86 for the new version. This was an increase of 8.7%.
|Version||$ raised||Conversions||Visits||Conv. %||$/click||Average donation|
The orange button produced an improvement in donations. It is not possible to say that the increase in conversions was statistically significant, and may require a longer experiment to produce such a result.
Overall the improvement is subtle.
It is important to note that we don’t know how much of the effect is due to orange buttons being generally better than blue buttons, and how much is due to the novelty of the change in the form.
We are now running another experiment on a new donation form that uses a multi-step design. This will likely change behaviour in more substantial ways.
Once this is completed, we will look at running further experiments on buttons with different colours and wording to further improve the design.
It may be worth running this test again in the future after the novelty of an orange button has worn off.