The Third Bear

Just Right.

A/B testing ActionKit survey questions

egj actionkit

ActionKit's built-in A/B testing platform for pages is great, but it doesn't yet support variations in survey question labels or survey questions themselves.  Fortunately there's a pretty easy way to do it yourself, while still running through the A/B platform to stop and start variations, view test results, and pick a winner.

The key is that WAWD recently made test variation enrollments available in templatesets, so we can just enroll a page in an "empty" test and then use template logic to conditionally display a different question.

Here's an example that just varies the label for a particular question.

  1. Create a new A/B test and enroll the relevant survey page in it. Don't enable any features for testing, even though it will prompt you to.  Add the appropriate number of empty variations -- if you want to test three different labels, add three empty variations.
  2. Note the ID of the new test. Let's say the test ID is 68.
  3. Now, in your survey page, put some code in the label of the question you want to test.
{% if tests.68.letter == "A" %}
How old are you?
{% elif tests.68.letter == "B" %}
What is your age?
{% elif tests.68.letter == "C" %}
How many birthdays have you had?
{% endif %}

If you try to view your page now, you'll probably see that literal code displayed to the end user.  To make it all work, you'll need to change your active templateset slightly, to tell ActionKit that a survey question's label should actually be parsed as template code instead of just rendered as-is.

Open up your active templateset's  `survey.html` template, and look for a block of code like this:

{% for question in form.surveyquestion_set.all %}
   <div class="input-parent {{ question.question_label|slugify }}">
    <label>{{ question.question_label }}</label>
    {{ question.question_html }}
   </div>
{% endfor %}

Change that {{ question.question_label }} line to {% include_tmpl question.question_label %} -- so the result should look like:

{% for question in form.surveyquestion_set.all %}
   <div class="input-parent {{ question.question_label|slugify }}">
    <label>{% include_tmpl question.question_label %}</label>
    {{ question.question_html }}
   </div>
{% endfor %}

You should now be all set!  You can now visit your test dashboard to start the test, view the stats for each variation, and stop & apply a variation when you have a winner.  And if you want to apply this test to a survey question that appears across multiple pages, just add more pages to the test as usual, and repeat step (3) above on each page.

You may be nervous about whether this templateset change will have consequences for the rest of your pages; if you ever happen to use question label text that "looks like" Django template code, you may end up with undesirable results on those pages.  There are a few ways you can limit the scope of this change.  One would be to create an entirely new templateset for this behavior that you only ever assign to pages that need to A/B test their survey questions.  Another would be to check the contents of the label a bit before treating it as a template, e.g.:

{% if "% if tests."|is_in:question.question_label %}
{% include_tmpl question.question_label %}
{% else %}
{{ question.question_label }}
{% endif %}

Extending this approach to vary the question content itself, or an entire battery of questions, would look pretty similar but involve baking somewhat more code into your templateset.  At minimum, you would need to {% include_tmpl question.question_html %} too, and rely on questions of type "custom HTML"; for a more user-friendly approach, you'd probably want to encode a variation in each question's field name, and then modify your for loop to skip over questions that aren't associated with the active test variation.