Segmented A/B testing in Umbraco

The web has provided businesses with unparalleled reach to large customer bases, but this great benefit was traditionally offset by the inability to approach customers individually. Your visitors are presented with a uniform experience, so it better be one that appeals to everyone!

Same with you communicating differently with your parents than you are with your friends (especially when the subject matter relates to your drinking habits), companies too are increasingly adjusting their language based on the audience they are talking to.

Differentiated campaigns with messages optimized to resonate with a particular segment are more effective than a generic message aimed at everyone but speaking strongly to no one.

- Segmentation tips from smartinsights.com

Chances are you are not doing much of personalisation yourself, and the reasons why are outlined below. This article introduces techniques (and two open-source packages) that will allow you and your company to add personalisation, a great tool for driving extra revenue, under your toolbelt.

What makes personalisation hard

The reality is that each visitor to your site is unique. People have different needs and desires, arrive to your site at different times and in all spectrum of moods, some already know the site, some see it for the first time.

We have just scratched the surface and we have already identified so many variables that might be affecting your visitors! Covering every possible combination won't be feasible, so how do you choose what content and for what audience to personalise?

Coming up with ideas is not the problem. Simply by following your intuition you are able to narrow down your search to just a few ideas you could try. The problem, as we soon find out is execution, or lack of thereof.

Too many people get stuck at the drawing board.

How many multilingual sites do you know that started ambitiously, translated in their eternity into several languages, only to end up having the increasingly outdated content trimmed down to just a handful of pages? This is personalisation too, personalisation done wrong.

What it boils down to is that regardless of the amount of preparation you put into this each adjustment to your site comes at a cost. Furthermore, you have to maintain all the variations going forward as well. That makes it a big commitment.

In the case of multilingual sites - is there a way to determine the right amount of translated content? Is it worth doing at all?

Work smart, not hard

With so many potential amendments you could make, but fully realizing the cost and future obligations, a lot of decision making will inevitably go into evaluating each proposed change. How do you narrow these down to those that have the best cost / benefit ratio? Or simply - where are you going the get the most bang out of your buck?

Most of the times you are able to estimate the costs that will go into implementing and maintaining your changes: from close to zero if it's a localized CMS managed change, to medium for localized layout / design changes, to fairly substantial if you are creating translated copies of the whole site.

All that is left is estimating the positive impact of your changes which, unless you have a Magic 8 Ball, is going to be rather tricky.

This is the breaking point: Without a simple but accurate measure it is easy to fall into a trap a lot of people refer to as "paralysis by analysis". That or they haphazardly change random pieces of their site. Sure, you'll notice when you did something that had a significant impact, but I would compare that to blindly shooting into the air thinking that eventually you'll get a duck.

A/B testing to the rescue

The way entertainment industry, and particularly niches that deal with masses (broadcasting networks, politics...), evaluate proposed changes is via the use of Focus Groups. These groups form a representative sample of their audience, or a particular segment they are interested in. The group's reaction to a proposed change is indicative of how the whole audience they represent would respond.

Because each segment is fairly homogeneous in their needs and attitudes, they are likely to respond similarly to a given marketing strategy. That is, they are likely to have similar feelings and ideas about a marketing mix comprised of a given product or service, sold at a given price, distributed in a certain way, and promoted in a certain way.

- Segmentation guide from omniture.com

The internet has came up with its own version of Focus Groups that is much cheaper, can cover 100% of your audience (without them even realizing they are subject to an experiment, so no bias!), and most importantly for some: it completely sidesteps human contact. It is A/B testing.

A/B testing provides you with efficient means to expose your visitors to one of your competing variations by bucketing them into these "virtual focus groups". You can now derive the performance of your variations by looking at the change in conversion rates of each group.

Last point to mention is that we are no longer in the realm of estimation anymore. We are instead measuring the real response people have. The plus side is that now we have real and accurate data to aid us in decision making. On the downside we need to have something real to show to our visitors - we need to put at least some effort into creating prototypes of ideas we want to test.

Breathing life into your sites

Perfect, now you know how to approach Personalisation. But how difficult and costly is it to implement? How much effort does it take to segment your customers? What will it take to create and run an experiment?

Turns out you are looking at a roughly 30 minute job and it won't cost you a cent! The barrier to entry is so low these days that even small sites with limitedt or no marketing budgets are experimenting with this. It is not a coincidence that Personalisation was a hot topic on this year's CodeGarden, with not just one, but two talks.

So, what does it specifically take to get started?

The second video is particularly interesting, as it features one of the two plugins this article relies upon: Personalisation Groups by Andy Butland. As the title suggests, this is used for personalisation, and you will use it in your templates to serve the content targeted for your segments. [You can also check out Andy's article about personalisation groups that he wrote back in Issue 4. - Ed]

Before you commit to having a particular piece of content permanently personalized and invest in necessary template adjustments and proper content variations, you will use segmented A/B testing to identify which personalisations, if any, are worth creating. For this we will use uSplit maintained by Endzone Software.

Disclaimer: the author of this article is one of the developers behind uSplit. While uSplit offers some unique capabilities, other A/B testing solutions such as Google Analytics Experiments or Optimizely will work for just as well.

Adding Personalisation to your Umbraco site

This section, and the one that follows, are meant to give you a general idea on the effort and high-level steps required to get these plugins up and running. Each is complex in details with enough corner cases that featuring a comprehensive tutorials would double the length of this article, and we have already passed the 1200-word mark!

When you give this a go, you want to follow the installation instructions on the Personalisation Groups GitHub page. You will have to do the following:

1. Install the Umbraco package (via backoffice). Search for Personalisation Groups. This step is important as it installs custom document types and data types.

2. Optionally install the NuGet package. This just makes updates easier, only adds the DLL, no DB update so not good enough on it's own! PM> Install-Package UmbracoPersonalisationGroups.

3. Create a special folder under your content root to store your segment definitions under. Use the Personalisation Groups Folder document type and name it something like "Personalisation Groups".

4. Update the Personalisation group picker data type and set the root node to be the "Personalisation Groups" folder you just created.

   It limits this content picker to operate only within the "Personalisation Groups" folder, making the search for visitor segments a bit easier in the Umbraco UI.

   For instance, this data picker is used by the Personalisation Groups uSplit plugin to allow you to select a segment for your experiment.

Done. Now all that remains to be done is to define your segments! You will start with a few, and grow and adjust these over time as you learn more and more about your visitors. We will demonstrate how to define a sample segment in an example at the end of this article but let's finish our first time setup first.

Enabling A/B testing of your content

uSplit gives you the ability to create, manage, and serve your experiments directly from Umbraco, while reporting your results to Google Analytics so that you can enjoy your existing analytics capabilities. Due to its reliance on Google Analytics and Google API, the setup requires additional steps to establish an authenticated channel between your Umbraco site and Google.

You should follow these detailed instructions (or check alternative installation options) inside the uSplit installation guide.

Google API

Firstly, your website needs to authorize itself when it contacts the Google API. Google uses this to attribute traffic to different projects (and to identify you as their client). Having these credentials merely allows you to talk to Google, it does not authorize you to access any private data. For that we will need to seek further permission from the owners of the resources we wish to access, but that is already handled by uSplit, more on that later.

To obtain the API credentials, you need to register a project, explicitly enable the Analytics API, and generate the correct set of API keys. The last step requires you to pick the right options at various stages of the process. Please see the installation guide for correct parameters.

You should end up with a Client Id and a Client Secret. If you have them the worst part is over!

Installing uSplit

Now that we have the pre-requisites necessary for uSplit to communicate with Google, we can proceed with installation. The preferred way to install uSplit is NuGet:

PM> Install-Package Endzone.uSplit

Installing this package will open a readme that will prompt you to do one additional manual installation step. Before you do that, install another NuGet package that will add a Personalisation Groups segmentation provider for uSplit:

PM> Install-Package Endzone.uSplit.PersonalisationGroups

After both package are installed lets return to the readme it asks to add a fragment to your Web.config.

<dependentAssembly>
  <assemblyIdentity name="log4net" publicKeyToken="null" />
  <codeBase version="1.2.11.0" href="bin/log4net/1.2.11.0/log4net.dll" />
</dependentAssembly>

This is needed because a particular GoogleApi library depends on a newer version of log4net than Umbraco depends on. A simple assembly redirect cannot be added, as the version referenced by Umbraco is not strongly signed. uSplit distributes the old log4net library on a side, and places it inside a special folder under bin. The dll directly inside the bin folder will get replaced by the newer version, but Umbraco will still be able to load the old one from the special folder, if you tell it where to find it!

From here we only need to update uSplit settings with the values we obtained earlier and with your Google Analytics account details. uSplit has already modified your appSettings sections of your Web.config and added five new configuration values. The first two are the GoogleAPI credentials we obtained earlier, and the last three are the Google Analytics identifiers of the account, property (website), and views.

You should end up with values similar to these:

<add key="uSplit:googleClientId" value="771327635790-aj37cdgcrl7c33nei5rvih72cefitd4m.apps.googleusercontent.com" />
<add key="uSplit:googleClientSecret" value="sbcFrLGMz6-U8--dF2RHI-1v" />
<add key="uSplit:accountId" value="63366167" />
<add key="uSplit:webPropertyId" value="UA-63366167-2" />
<add key="uSplit:profileId" value="104332457" />

Perfect, you should be able to rebuild and run your site now!

Finally, we recommend creating a special folder (unlike the "Personalisation Groups" one) that can be used to store all your temporary content variations for your experiments:

1. Create a new Variations Folder document type. It doesn't need any properties, but make sure it is allowed at root, and that all relevant document types (that you plan you plan to test and tweak) are allowed as children.

2. Create an instance of this document type under root. When uSplit asks you where you want to save your variation, simply select this folder.

3. Keeping variations in one place helps you keep them away from your master content (so that they do not affect menus for instance), and optionally allows you to create a simple redirect rule so that none of these variations are publicly available.

Granting access to the Google Analytics account

uSplit is now properly configured, but it needs to obtain a permission to access your Google Analytics data. You may grant these permissions on the "A/B Testing" dashboard (click on the "A/B Testing" tree on the Content section). Here you can invoke an authorization flow that will authenticate you with Google and grant uSplit access to the Google Analytics on your behalf. If you don't have access yourself simply ask someone else who has to do this.

That's it!

Phew, 30 minutes of hard work but you have added some pretty unique capabilities to your site! Let me give you a big pat on the back and let's have a look at a real problem you might encounter and the efficiency you just gained.

Seriously though, as you are about to find out, you just saved your business countless of wasted hours! Congratulations to that!

Case study

Let's look at a real-life example, something we've been asked to do in the past and what probably took us a bit longer to get right than it should have: A UK client from the hospitality industry asked us to enable them to, on a particular page, show personalised content to their US visitors.

We ended up doing what most would resort to - installing GeoLite 2 database, adding helpers that check the country based on the IP... but only after being exposed to the perils of relying on free online APIs and learning the hard way that creating blocking calls to random services perhaps wasn't the brightest idea after all! Let alone those who dreaded to explore the possibilities of hosting these APIs as standalone services themselves...

Don't reinvent the wheel!

Contrast these wasted hours (and client frustrations) with some 30 minutes required to install and setup Personalisation Groups and uSplit!

What is even more amazing is that this investment cuts the overhead of creating personalised content down to bare minimum, effectively allowing you to change the unit in which you estimate similar changes in the future - from hours to number of clicks.

What does the process look now that we have both of these plugins at our disposal?

Assessing the ideas

Before any development goes in there are some considerations you want to take into account:

• What is the cost of permanently implementing this idea?

• What is the simplest version if this that would allow to test the response?

• Define a success criteria: how big of an improvement in relevant metrics (e.g. goals in Google Analytics) will warrant implementing this and maintaining this variation?

Only once you have answers to the above, and you are happy with the estimated costs, can you proceed with the first step towards personalisation - testing the idea in the wild.

Measuring the response using segmented A/B testing

Apart from the mental exercise above, there are only five steps needed to validate each idea:

1. Define the segment we are targeting

2. Create an experiment and select the page you are interested in personalising

3. Create one or more variations you want to test (maybe you have more ideas what could work for the particular audience?)

4. Start the experiment and wait as data is collected

5. Analyse the results

Creating the segment

A segment is simply a set of various tests and rules. Each visitor belongs to all segments whose rules it satisfies. Following our example, we will create a "US visitors" segment:

1. Right-click the "Personalisation Groups" folder and create a "US visitors" Personalisation Group

2. Duration in group is Per visitor

3. We add a Country criteria, setting US as the country code parameter

Save and off you go! You are now able to target US visitors. If you have an important message to tell them regarding the upcoming elections now you can!

Setting up the experiment

Perfect, we can proceed with creating the actual experiment and adding a content variations we want to try. Sometimes you test just one (and see if there is a change in conversions compared to the original), but sometimes your designer gives you three options and you can have them all compete as part of a single experiment.

Crash course to uSplit

It is extremely cheap to create simple variations with uSplit. "Under the hood" variations of a page are stored as different instances of the same doctype. I will shortly demonstrate how to create these, but what this means is that you are editing your variations using the same editor you are used to. As a matter of fact, a legitimate way of creating a variation is simply by copying (duplicating) the page you want to test, modifying it, and selecting it as a variation.

When the page under test is requested, uSplit sets the PublishedContentRequest.PublishedContent property to an instance of VariedContent. This class contains a reference to the original page and all the variations that apply (only one per experiment, but multiple experiments might be running). It then simply redirects accessed properties to the first variation that has this property set, otherwise to the original. Same happens for the template by the way, so you can run multiple very distinct designs with no issues whatsoever. For details about how to use uSplit see a detailed walkthrough.

Creating the experiment:

1. Right-click on the A/B testing node (on the Content section), and click on "Create a new experiment".

2. Select the page you want to test (and personalise)

3. An Experiment page will appear. Under segmentation select "Personalisation Groups" and select the "US visitors" segment. (For those paying close attention: this is where the Personalisation group picker is used by the way.)

4. Optionally set any advanced experiment properties manually via the Google Analytics portal.

   One useful setting here is the objective: What metric / goal should Google use when comparing the performance of your variations? The default is Pageviews, but if you are optimizing for anything specific you want to see the direct impact on that goal.

Creating content variations

We have already mentioned one way of creating a content variation - by copying the original page. Once you are on the Experiment page you can achieve the same thing by clicking the "Clone Original" button. Either way you need to decide where to place the copy - it doesn't really matter though we suggest placing it inside the "Variations Folder" to keep it away from your normal content.

Alternatively you may also create a new page of the same document type, and add it as a variation by clicking the "Add existing page" button on the Experiment page. The advantage of this approach is that you get a blank document: you have total control over what you are overwriting (empty properties are ignored).

Let's create a single variation for now. Click on the "Clone Original" button and once a new item appears in the list of variations click on the "Edit" button to edit it. Make any adjustments you think will have a positive effect on the target audience. Save and publish.

Running the experiment

That was a lot of text but very few clicks, my friends. You can easily try a few ideas in one go, a few different images, different flavours of the main body, etc. There is nothing wrong with doing one at a time though, and for sites with lower traffic this might be a better strategy anyway.

Starting an experiment is irreversible. Before you hit "Commence experiment" let's do a quick rundown:

• Are happy with the variations? Are they all published?
• Is the correct segment set?
• Did you set a custom goal? (optional)

If all looks good then there is nothing holding you back - Commence the experiment!

Making sure things work

You can temporarily change the segment rules so that you match them - set the country condition to your country for instance, or change the "Match" type to "Any" and add a Querystring criteria for some key you then append to the URL.

Once you match the segment (or if none was selected) you should see a piece of JavaScript injected into your HTML to report the variation (close to the end of  <head>), and a custom "experiment" view event raised in Google Analytics (assumes you are using analytics.js, but you can customize all this).

Beyond A/B testing

Right now the experiment is live and all visitors coming from the US will either see the original page (as a control) or the variation we created.

Over time, as enough data is collected, Google will pick a winner of the experiment. It will be the variation that had the most success in terms of the objective of the experiment. You may also elect to end the experiment prematurely (if you can spot the trend earlier or the results are not very convincing).

If any of the variations look promising you can consider implementing this personalisation permanently. This relies on the Personalisation Groups plugin but explaining how to do that is outside of the scope of this article. There is a trove of information available on their GitHub page, alternatively watch one of the videos referenced earlier. It is still super simple, but most importantly - you know the investment will pay off!

Conclusion

If you ever had to pick music for a room full of people you will understand that making everybody satisfied is an impossible feat. At home you may simply ignore those with inferior taste in music, but once you put a bouncer at the door and start charging for entry... will you pass on those extra sales?

The solution to these problems is personalisation. Some solutions (like the one to the to the problem above) might seem counter-intuitive, but if you approach it right and base your work on real data it really guides you towards the right (and profitable) decisions.

Thanks to free and open-source plugins like Personalisation Groups and uSplit you gain capabilities that were traditionally available only as premium services. For minimal investment you can start experimenting with personalizing your site for particular groups of visitors.

The combination of the above plugins allows you to approach personalisation in a very efficient and rigorous manner. This is necessary because maintaining extra content is costly and it is hard to just guess the reaction of your audience. You want to base your decision making on something objective, like, say, asking your visitors directly.

Leveraging both plugins you can use the following process to deliver improvements to your sites:

• Pick an idea: Based on some heuristic select the next candidate to be considered for implementation. Ideally you should have hypothesis backing it up with some numbers like an unexpectedly high funnel dropoff. State what your expected outcome is and estimate the cost of proper implementation.

• Test the idea: Create just a prototype of the change (or start with just a small part) and test that. Considering translating your site to French? Start with the homepage and see how often people visit it.

• Reach an informed decision: Evaluate the data. Did a lot of people visit your French homepage? Perfect, translate more pages! Did nobody care? That is good too! Now you know and you can delete the prototype that was cheap to make, and you don't have useless content to maintain!