Test Variant

Set up A/B tests in Neocom to optimize recommendations and understand customer preferences

A fresh new A/B Testing feature, called “Test Variant”, has been deployed allowing you to learn more about your advisor, your customers, and about making the perfect product recommendation. 

While there are countless ways to use this feature, in this article, we’ll focus on the following:

1. Setting up Test Variant - A/B Testing
2. Example Use Cases
   
   1. Testing Single Question Variations
   2. Testing Multiple Question Flows
   3. Testing Different Mappings
3. Analysing the results 

Setting up Test Variant - A/B Testing

To set up A/B testing, navigate to the Conversation tab in your Neocom Admin Portal. While within an individual question, navigate to the horizontal navigation bar towards the top of the screen to the option Settings.

Once the menu opens, you can navigate to the Test Variant portion, create a new Test Variant for the question you are working on, or select from variants you've created for other questions. Once the Test Variant is entered or selected, Save your changes and the process is complete. 

You just want to test one variation?

Create a Test Variant for just one question, leaving the Test Variant field empty for the others. This will split users evenly—50% will see the test question, and 50% won’t.

You want to create two completely different flows?

Whether you’re testing different imagery, language styles, or entire questions, you can create two separate flows. Start by creating Flow A, adding a Test Variant, and naming it accordingly. Then, create Flow B, using the Duplicate Question Feature if needed, and name it appropriately. This will divide users so that 50% experience Flow A and 50% experience Flow B. If there are more than two flows, the audience will be split accordingly.

Example Use Cases

Testing Single Question Variation

If you want to test how a single question impacts your advisor’s effectiveness, follow these steps:

1. Create or duplicate a question. 
2. Add a Test Variant (A/B Test) 
3. Publish your advisor

A common use case: You could test the impact of adding a budget question. Half of your users will see the question, and the other half won’t. Then, analyze how this affects the average order value.

Testing Multiple Question Flows

If you want to test two completely differing question flows, you can follow the following steps: 

1. Create the first question flow and assign a Test Variant name. 
2. Create the second set of questions and assign a different Test Variant name - you can also duplicate questions you want to have in both or leave those you want in both without a Test Variant name. 
3. Publish your Advisor

Testing Different Mappings 

You can also use the Test Variant Feature to try out different Mapping approaches and compare the results. By following the steps below, you can create variations of the same question, but with a differing logic working in the background: 

1. Choose the question you want to test. 
2. Duplicate it as many times as needed. (you can test an unlimited number of variations) 
3. Assign a Test Variant value to each duplicate and adjust the mapping for each. 

Analysing the results 

Creating an A/B, or A/B/C/D/E Test only makes sense if you can analyze the results afterwards. That’s where Neocom Analytics comes into play.

By applying the right filters, you can compare the performance of each variation, examining metrics like bounce rates, average sales value, average basket value, and more. For a detailed guide on how to use Neocom Analytics and Users, refer to this Knowledge Base articles: Analytics and Users.

This analysis will not only help improve your current advisors but also shape future ones. You’ll gain valuable insights into your customers' preferences—whether it’s product selection, imagery, language, or any other important metric. These insights can then inform your marketing strategies, logistics, and beyond.