Digital experimentation: How to transform digital content experiences

If you can remember your first science experiment, you might also remember what you learned from it. A combination of vinegar, baking soda, and food coloring, for example, is a great way to simulate a volcanic eruption (and risk ruining a carpet). 

The point is, experimentation and data-driven observations help us not only change the state of things, but understand why they've changed — and that principle applies just as much in digital ecosystems as it does in the real world. 

In fact, the rapidly evolving digital landscape in which most businesses operate makes experimentation crucial. Shifting customer expectations and market trends, plus new technology innovations make it difficult to know intuitively what kind of content experiences customers want when they visit a website, use an app, or interact across other digital channels. 

That’s a problem that digital experimentation can solve. 

In this post, we’re going to explore the concept of digital experimentation, find out why it’s so important to modern content experiences, and discuss how brands should approach digital experiments with their own content. 

What is digital experimentation?

Digital experimentation refers to the process of testing different types of digital content in order to optimize that content for an intended purpose, and maximize return on investment (ROI). The intended purpose in question might be personalizing a page for a specific demographic, determining the optimal price for a product, simplifying the customer journey, and so on. 

Accordingly, the digital experimentation process requires a brand to develop,  perform tests on its content, and then analyze the results. The data that the content tests generate can be used to help adjust the brand’s website, mobile app, or any other digital customer touchpoint. 

Making decisions based on hard data is, obviously, an alternative to simply using intuition to decide what an audience might like. It also enables brands to apply metrics to their content’s performance in order to judge its success objectively. A brand might, for example, test different versions of a call-to-action (CTA) button, by changing variables like color, position on page, and so on, in order to determine which prompts the greater number of customers to click through. 

In theory, the greater the quality and quantity of digital experimentation data that content tests generate, the deeper the understanding a brand can develop of both its content and its audience behaviors, and use that valuable insight to influence customer behavior, or increase customer satisfaction and engagement. 

The insight you derive from experimentation has numerous downstream benefits too, from driving conversion rates higher to ensuring that your content is working as hard as possible to deliver ROI without you having to increase the effort you put into it.  

The benefits of digital experimentation 

Let’s drill down into the specific benefits of digital experimentation. 

Stronger decision-making

The objective data that digital experimentation delivers enables stronger, more efficient decision-making. For example, if a brand is launching a seasonal promotion on its ecommerce site, it can test different versions of a promotional page to decide which creates the greatest user engagement, and then quickly roll out the more effective version of the page to all touchpoints in order to maximize financial return.

Conversion optimization

The overall goal of experimentation is to increase conversion rate and ROI from content. However, that outcome is predicated on experiments increasing engagement: prompting users to click a link more frequently, browse more pages on your website, and progress further into your customer journeys. Increased engagement will ultimately lead to higher conversions on metrics such as adding products to the basket, signing up for an email list, actually purchasing your product, and so on. Digital experimentation helps brands zero in on the specific content elements that are driving user behavior, and then fine-tune those elements to optimize conversion rate further.

Understanding customers

Digital experimentation offers brands a perspective on their customers’ preferences and behavior. That insight goes beyond objective, quantifiable data such as click-through rate or time on page, and can feed into the creative process itself. That offers numerous downstream benefits: it can make content creation more efficient, it can inspire future marketing campaigns, it can even help brands develop products and services.  

Avoiding the HIPPO

Effective experimentation helps avoid falling into the trap of following the highest-paid person’s opinion (HIPPO). If you’re following the HIPPO, it’s likely you’re not relying on data driven insight — at least, as much as you could be. By implementing effective experimentation, and ensuring data continuously flows into the decision-making process, you’ll be able to prevent the organization being run from the top down, and avoid an environment in which people are scared to voice their opinions, or in which decisions are made based on seniority alone.  

Personalization

By experimenting with different forms of content, you’ll be able to derive data-driven insight into what works best for specific audiences, and personalize content accordingly in real time. In practice, that means being able to adjust headlines to capture attention, display personalized product recommendations, adjust page layouts, and even create new content automatically during customer journeys. 

Culture of experimentation 

Digital experimentation isn’t a box to be ticked for each new campaign. While there can be quick returns on performing content tests and gathering data, digital experimentation has the wider benefit of fostering a culture of ongoing innovation and exploration in relation to a brand’s tech stack and long-term digital transformation. As part of this culture of experimentation, content teams (and all stakeholders) are empowered to develop creative solutions to problems and consider different perspectives as they develop digital experiences. By fostering a culture of experimentation, brands not only get to develop innovative, engaging, new content, but drive significant business growth.   

Different types of digital experiments

So, what kind of digital experiment could you run in order to test your content? Let's look at the options.

A/B testing

In an A/B test, you create two or more versions of a piece of content, and then directly compare them to determine which is the more effective. For example, you could create two versions of a CTA content component: one of which uses a red button, and one which uses a blue button. Then, you’d test them against each other by bucketing users into one of the two variations in order to find out which delivers the higher conversion rate. If the test generates statistically significant results, you’ll be able to identify and implement the optimal CTA on the live site. 

Multivariate testing

Whereas A/B testing involves two different versions of an individual piece of content, multivariate testing involves the comparison of multiple variables in order to determine which produces more conversions. For example, a test involving two differently worded CTAs, in either red or blue, would test four combinations of content: a red version of both wordings, and a blue version of both wordings. 

Multi-armed bandit testing

In a multi-armed bandit test, you create multiple versions of a piece of content. The test begins with all traffic bucketed equally but then, over time, as some versions perform better than others, the algorithm moves traffic to the higher performing versions in order to maximize ROI as quickly as possible. 

While a multi-armed bandit test still allows you to experiment and gather data, and wait for statistical significance before selecting a winning option, you can begin to increase ROI ahead of that decision, thereby increasing the overall impact of the experiment. This approach also allows you to include more people in the test, which speeds up the overall experiment because you won’t have to account for the negative impact of the underperforming versions. 

Feature flagging

Feature flagging is an experimentation strategy that allows you to ring-fence certain content features, such as a new homepage redesign or a new product that you're launching. These features usually fall outside of the day-to-day content items that you're creating for your website (blogs, reviews, etc.), while the flagging process allows for more control of certain important processes such as roll-outs, where you might decide not to roll out a homepage redesign to every visitor straight away. Instead, you could roll out flagged features more slowly to check everything works, to avoid negative feedback, and so on. 

You can also run an A/B test on a feature flag by randomly bucketing users into either “feature turned on” and “feature turned off” buckets, and testing the outcomes. 

The type of test you ultimately choose for your experiment will depend on certain contextual factors. For example, while A/B testing may only compare versions of a given variable, it typically requires a lower volume of testers to generate statistically significant results than multivariate testing. Meanwhile, multivariate testing could end up making your post-test reporting unclear: for example, if you see a huge difference in visitors’ behavior following a multivariate test, how do you know exactly what variant caused the change? Multi-armed bandit testing, on the other hand, can help you assess content efficacy over a longer period of time, and even account for fluctuating content trends. 

How to implement digital experimentation strategies

Implementing digital experimentation strategies requires planning. The more thought you put into your experiments, the stronger and more reliable your tests will be, and the easier it will be to use them to make important decisions.

With that in mind, let’s explore the digital experimentation process step by step.

1) Exploration

You're running content experiments because you want to improve the content that you deploy across digital touchpoints — so you should begin by reviewing your current site performance, then setting new performance objectives along with metrics for success. 

For example, if you want to boost the number of page views for a particular product, you might decide to test the conversion rate of a homepage CTA and set key performance indicators (KPIs) to measure success. You might also seek to boost product sales, bounce rates, time on page, and so on. 

2) Ideation  

The objectives and metrics that you set will shape the next phase of your experimentation process: identifying the content you want to test and the experiments that you will perform.

At this point, you’ll need to develop a hypothesis that aligns with your business goals. For example, you might hypothesize that different homepage header text will encourage more visitors to scroll down the page. To test this hypothesis, you’d create an A/B test with two different versions of a page header, and then analyze the behavior of incoming traffic to the page. 

You don’t need to restrict your experiment to a single type of test: you could implement a multivariate test for the same objective, based on the same hypothesis. 

3) Prioritization

Prioritization is a way to optimize digital experimentation if you have multiple tests to run — for example, three separate A/B tests.

Prioritization not only helps you assign sufficient resources to your tests but manage them effectively once they’re in motion — thereby optimizing their eventual value and impact. This is a consequence of the “interaction effect”: part of prioritization is about understanding how experiments will interact with each other, and determining how best to track outcomes in order to run as many experiments as you can concurrently.

Common experiment prioritization frameworks include:

• PIE: The PIE prioritization framework requires you to consider three factors: potential, importance, and ease (PIE). You rate each test against those three factors, on a scale of one to ten, and assign the highest priority to the highest scoring. The PIE framework is subjective, but offers a quick and simple way to establish your test priorities. 

• ICE: In an ICE framework, priority is assigned based on impact, confidence, and effort (ICE) — which are, again, rated on a scale of one to ten. Like the PIE framework, ICE offers a quick, simple way of assigning priority but it’s also just as subjective, so the personal preferences of the internal testing team are going to be a factor. 

• RICE: An extension of the ICE framework, RICE adds a “reach” factor to the prioritization scoring. In this context, reach refers to how many people will potentially be impacted by the variable being tested.  

• PXL: The PXL framework offers a more objective prioritization process than other frameworks, by setting out a series of 10 yes/no questions for each test, with more important questions weighted more heavily than others. PXL isn’t as simple to apply as other frameworks because some questions may need to be tailored to the specifics of the test. 

4) Execution 

You’ll need to select a testing tool to help you carry out the tests that you’ve developed. The testing tool you select will offer functionalities that help you create, run, and manage your experiments, and analyze their results for statistical significance. Not all testing tools are going to offer the same experiment setup capabilities, integration options, or value for money. 

To learn more about the capabilities of an AI-supported testing platform, check out  Contentful Personalization. 

With your tool selected, and your test set up, you can launch your test and, once it’s completed, analyze your collected data. 

5) Continuation

Digital experimentation isn’t a box to be ticked as you develop content. Digital experiences should evolve with the needs of your audience and the market, and so you’ll need to continue to experiment and iterate continuously to ensure your offering remains as impactful as possible. 

With that in mind, reflecting on your experiments can be just as important as deriving data from them. Even failed experiments may deliver benefits, and not just because they offer insight that isn’t based on “gut instinct.” This type of outcome should be celebrated in the same way as a “successful” experiment because it forms a vital part of your "culture of experimentation,” in which everyone has the capacity to explore, experiment, learn, and, ultimately, make data-led decisions.

Transform content experiences with Contentful

Experimentation is fundamental to the digital transformation process. The tests you perform will help steer your brand through the challenges of a constantly evolving digital world, shape your digital strategies, and ensure the content you create remains fresh and engaging. 

Experimentation can be daunting, but with the right tools and the right expertise you can develop an accessible, efficient testing solution that streamlines your process and delivers actionable data, when you need it.

You can kick-start your process on Contentful by adding Contentful Personalization to your tech stack. Tightly coupled with your content management system (CMS) in order to reduce your content operations effort, Contentful Personalization aids collaboration between teams and makes running experiments second nature to content production. Powered by built-in AI, our tool not only generates experiment ideas for your content, but can automatically segment your audiences, based on user behavior. 

Even better, Contentful Personalization is accessible for non-technical teams, who can create and manage experiments independently, and foster a culture of experimentation from the ground up. 

Don’t let your content experiences lose momentum. The more rigorous and effective your digital experimentation strategy, the more you’ll learn about your audiences over time — and the easier it will be to transform your brand, your business, and your bottom line. 

Take a tour

Chat with our team

We'll show you how our composable content platform can enhance 1:1 customer connections and drive growth for your business.

Subscribe for updates

Build better digital experiences with Contentful updates direct to your inbox.