A/B Testing with Contentful and Optimizely

Overview

This tutorial gives an overview of an end-to-end experimentation setup with Contentful, Optimizely and the Optimizely App. The goals of the setup are:

• Separation of concerns: Controlling content in Contentful and experiments in Optimizely
• Speed: Server side selection of variations for fast delivery and to avoid "flash of content"
• No development dependencies: Enable server side experimentation without pushing new code

The following areas will be covered:

• Setup of the Optimizely app in Contentful
• Selection of content variations from the Contentful and Optimizely APIs
• Rendering of experiments in a React application

Optimizely

Optimizely is an experimentation platform that integrates with your content to produce powerful insights on all your apps powered by Contentful.

In the following tutorial you will walk through how to set up a simple application which allows for A/B testing using content from Contentful.

Requirements to get started

• An Optimizely account with a "Full Stack" experiment.
• The Optimizely app installed on your space. See the installation instructions.

How the Optimizely App changes the Contentful API response

The Optimizely app creates a new content type: the variation container. A variation container is a custom Contentful content type which consists of two or more values inside a content model that pertain to the same reference field. Let's look at the following example:

In our example without Optimizely, we have a content model with a reference field to hold "blocks". Notice the CTA is "Buy Now!":

Fig 1.1

Using Optimizely, the same content model is transformed. Where we originally received the CTA of "Buy Now!", we now are receiving a variation container holding all the different possible CTAs:

Fig 1.2

The variation container is simply a content type that nests the possible values for CTA.

Leveraging Optimizely to pick the right variation

We are going to create a pseudo-backend which proxies the Contentful API response and uses the Optimizely SDK to determine which variant to use. We will then reformat the response based on this decision.

The Optimizely client allows us to determine which variant we should use based on an identifier for the current user. In this example, we are passing userId to determine which variant this user should see from the ctaVariants experiment.

import Optimizely from 'optimizely';
import datafile from './optimizelyDataFile';
import { getUser } from 'user';

const optimizelyClient = new Optimizely({ datafile });

const user = getUser();

const variation = optimizelyClient.activate('ctaVariants', user.userId);

// variation => 'cta_b'

ctaVariants contains two possible variants, a control group called cta_a and a test group called cta_b. In the example code above, Optimizely has determined this user should see variant cta_b.

Let's look at the variation container JSON response. Notice that the content type exposes a meta and variations property which lists all possible variations.

{
    "sys": {
    "space": { ... },
    "id": "41nZggHEplcBOsrPXLOEU",
    "type": "Entry",
    "createdAt": "2019-07-17T14:32:24.306Z",
    "updatedAt": "2019-07-17T14:32:24.306Z",
    "environment": { ... },
    "revision": 1,
    "contentType": {
        "sys": {
        "type": "Link",
        "linkType": "ContentType",
        "id": "variationContainer"
        }
    },
    "locale": "en-US"
    },
    "fields": {
    "experimentTitle": "CTA Variation Experiment",
    "experimentId": "15249150297",
    "meta": {
        "cta_a": "2hoYOxnZVV3Imvry1DhmtG",
        "cta_b": "5IlB5PCsca3zZnzlVd7WvI"
    },
    "variations": [
        {
        "sys": {
            "type": "Link",
            "linkType": "Entry",
            "id": "2hoYOxnZVV3Imvry1DhmtG"
        }
        },
        {
        "sys": {
            "type": "Link",
            "linkType": "Entry",
            "id": "5IlB5PCsca3zZnzlVd7WvI"
        }
        }
    ],
    "experimentKey": "cta-experiment"
    }
}

cta_a and cta_b are variations of this experiment and they point to a Contentful content type which holds the actual content we want to show. Let's add to our backend code to get the variation:

import Optimizely from 'optimizely';
import datafile from './optimizelyDataFile';
import { getUser } from 'user';
import sdk from 'contentful-sdk';

const optimizelyClient = new Optimizely({ datafile });

const user = getUser();

// we are using the id of the variation container from the JSON sample above
const variationEntry = await sdk.getEntry('41nZggHEplcBOsrPXLOEU');

const variation = optimizelyClient.activate('ctaVariants', user.userId);

// variation => 'cta_b'

const ctaEntryId = variationEntry.meta[variation];

// ctaEntryId => '5IlB5PCsca3zZnzlVd7WvI'

We now have the entryId of the CTA content block we want to display! This means that the content we should show is the "20% off!" CTA from Fig 1.2.

Important notes about the example

In this example we used userId. Optimizely requires that you identify the current user so they can determine which experiment group to place them into. This ID should stick with the user and properly identify them throughout different sessions. If you use a random number as an ID, your A/B test will not work correctly and the results will be meaningless.

A datafile was used in the example above to create the Optimizely client. This is supplied by Optimizely and will be unique to your account. You can read more about the data file here.

Only published content on Contentful will be exposed in the variation container. Using a Contentful environment, you can test how a variation container works before promoting your experiment to production.

Conclusion

Concepts covered:

• The definition of a variation container.
• How a variation container is returned in the Contentful API response.
• How to leverage Optimizely, using their SDK, to transform a variation container into the correct display field for your presentation layer.