As an ecommerce store owner, your customers are your lifeblood. The more you cater your shop to them, the more successful you’ll be. That’s why measuring how they respond to various elements of your store is the key to increasing conversions.

You might test a few different product layouts, multiple versions of a homepage sale pop-up, or two different colors for your Buy now button. Finding the version of each component that produces the most sales is a practice with huge potential for growing your business.

Now, you’ll also be able to do this with different aspects of your ecommerce chatbot, thanks to our new A/B Test plugin for Pro users! Here’s what you need to know about how to test the chatbot for your store with this plugin, so you can make sure it’s as effective as possible in boosting sales.

What A/B testing is

A/B testing is a tool commonly relied on by businesses across all kinds of industries, including ecommerce. The idea is to compare the performance of two or more versions of the same asset or component, so you can tell which one is more effective for achieving a certain goal.

For a proper A/B test, all aspects of the two versions must be the same—except for one element that you’ll be testing. For example, you could compare the performance of:

  • Two landing pages with different sign-up forms to see which gets more leads
  • Two marketing emails with different subject lines to see which gets more opens
  • Two Facebook ads with different copy to see which gets more clicks
  • Two checkout-page designs to see which leads to lower cart-abandonment rates
Shopify Merchant?
Recover up to 7x more revenue from abandoned carts than with email reminders.
Coming soon: Conversational tools for Shopify that drive sales.
Get VIP Access


If there’s a significant difference in results between version A and version B, you’ll know it’s because of the single element that varied between the two. Once you have a clear winner, use only that version of the asset from then on, and test something else next.

Our new plugin allows you to run this same type of experiment with different elements of your bot, so you can see which version of it will promote more sales.

​​What A/B testing is
An example of A/B testing would be the same email with two different subject lines, to see which gets more opens.

Why you should use A/B testing for your bot

A/B testing will help you calculate and improve the number of conversions your bot achieves. First, it’s important to define what a conversion means for your unique ecommerce business at its current stage. In general, it’s some action a user takes that makes them more valuable to you. It could be completing a product-recommendation quiz or clicking a link. Or, you may only be interested in which configuration of your bot gets the user to make a purchase then and there.

Once you’re clear on what type you’re looking for, A/B testing can help you calculate conversions so you can see how many you’re actually getting. It’s hard to know how to improve if you don’t know where you stand. Analyzing this data will help you learn about your bot subscribers, what they like, and what they’re drawn to. You’ll be able to see just how effective your bot currently is at helping you achieve your business goals, instead of hoping or assuming.

Then, you’ll be able to improve conversions based on your findings. The A/B Test plugin will help you see what makes users lose interest in your bot on the road to conversions, versus what keeps them engaged and moving forward. You can learn from the data and make adjustments, refining your bot’s design until you’re getting the best results possible.

How to test your chatbot’s elements with the A/B Test plugin

To learn how to test for chatbot effectiveness across different elements, follow these six steps:

1. Create two flows to test. First, design variant A and variant B: two portions of your bot where everything is the same except the factor you want to test. (Note that you can test up to four variants at once with this plugin). Let’s say you wanted to see if including the user’s name in a message will make them more likely to visit your online shop. In separate blocks, create two messages that are exactly the same—except that one includes {{name}} for customization.

A/B testing
Create two flows that are exactly the same, except for one element.

2. Use Button Actions to link a user attribute. Next, you’ll need to link an attribute to the button you want users to select so the plugin can track how many actually do it. For our example, we could create a user attribute called Shopper which, when it equals true, means the user clicked the button that leads to our store website. We would then use the Button Actions feature to automatically apply this attribute to any user on either variant who selects the Yes! button and is taken to our shop.

A/B Testing
Use Button Actions, another Pro feature, to apply an attributes to users who take a specific action

3. Add the A/B Test plugin. Now, navigate to the Plugins menu in your Chatfuel dashboard and add the A/B Test plugin. We’ll select the block that contains the first version of our message for variant A, and the one that contains the second version for variant B. Then choose the user distribution, which is the percentage of all bot users that you’ll send to one variant versus the other. You can choose any numbers you like, but keep in mind that an even, 50/50 split will get results more quickly.

Link your blocks to the variants and choose the user distribution.

4. Set your goal. The last thing you’ll need for the plugin to function is a goal, which is the user attribute you selected in step two. In our case, the goal is for the attribute Shopper to be marked as true.

Set your goal based on the attribute you want to be marked as true.

5. Collect results. Your test is now set up! All that’s left is to wait for enough users to interact with your bot to get accurate results. You’ll need about 10 to experience each variant before you can analyze. Don’t edit either one after the test has begun, as it will skew your data. When the test is complete, Chatfuel will tell you how many users went through each variant and what percentage of those achieved your desired conversion (the goal we set in step four).

​​Collect A/B Testing results
These results clearly show that variant B is the version of the message we should use going forward.

6. Make changes based on the results. Once you’ve received and examined the data, the final step is to take action based on what it shows. If there’s a significant difference between the two results, as in the example above, adjust your bot’s flow so that all users receive only the more successful variant. If the results are very close, consider running the test again with a more major difference between the two. If you continue to receive inconclusive results, that element clearly doesn’t have an impact on user behavior. Focus instead on testing another aspect of your bot.

Shopify Merchant?
Recover up to 7x more revenue from abandoned carts than with email reminders.
Coming soon: Conversational tools for Shopify that drive sales.
Get VIP Access

Other ideas for using the A/B Test plugin

You can A/B test a chatbot to gauge the effectiveness of most any of its elements, including:

  • The flow. Do users who are led through a quiz flow end up clicking through to your ecommerce store more often than those who aren’t? Do those who receive images of your new product line click through more often than those who don’t?
  • Voice and tone. Does a bot with a more serious tone resonate with your users, or do they prefer one that’s lighthearted and makes jokes?
  • CTA, or call to action. Does one CTA get more conversions than another? Are users more enticed by a personalized response that involves their emotions, or a straightforward invitation to click or tap?
A/B testing
You could A/B test two different CTAs to see if one encourages users to click through to your store more than the other.
  • Emojis and GIFs. Do emojis encourage users to stick with a bot quiz till the end? Does a GIF help entice them to browse your products?
  • Plugins. Does attempting to collect an email address turn users off? Is adding a video more or less effective? Do users who are offered the Live Chat option tend to convert more than those who aren’t?


When it comes to ecommerce sales, it’s often the small details that have the biggest impact. This plugin allows you to test multiple versions of all kinds of details, so you can use the data to decide which work best. The testing process for your bot just got easier—and so did increasing sales and growing your business.

Maximize sales for your online store with A/B testing

As an ecommerce store owner, you only stand to benefit from running A/B tests for your chatbot on Facebook Messenger. You’ll learn more about your customers and how they interact with your brand, and you can drastically increase your conversion rates based on your findings.

Getting started on the road to gaining more sales with a bot is simple. Sign up with Chatfuel today to begin creating a chatbot for your own ecommerce store—no coding knowledge or special skills required. Next, take our free, online course to learn all the bot-building basics, then check out our guide for dozens of other ways to increase sales for your online store with a bot.

Ready to take your business to the next level?
Request a Demo