An A/B test is an experiment between two competing landing pages. The idea is to see which landing page generates a higher conversion rate for a given campaign. The conversion rate is the percentage of unique visitors that complete the form on the landing page.

What URL are A/B tests run on?

Both pages are shown on the same URL. By default KickoffLabs will use the URL of the A page being tested.

If the A page was published to www.example.com then the B page would also be shown on that URL. The page weight (% of time each page is scheduled to be shown) determines which version of the page ( A or B ) will be shown to each unique visitor.

Will one person see different versions of the page? 

No. We track, per browser, what version of a page was shown to someone. So the next time that person comes to the landing page they will see the same version they signed up on. This means that, for your testing, you will only see one version. It’s why we generate testing URLS for you to verify each version of the page on the final URL.

How is page weight determined?

In KickoffLabs you can choose to have us automatically determine the weight with a “Smart Test”. Or you can opt to manually set the percentage of times each page is shown on the URL.

How do KickoffLabs “Smart Tests” determine the weight?

In a standard A/B test pages with radical differences would typically lose potential conversions by showing bad variations of a landing page too often. We worked around that by using what’s called  “Multi-Arm Bandit Experiments”. It’s the method recommended by Google and described here:

The name “multi-armed bandit” describes a hypothetical experiment where you face several slot machines (“one-armed bandits”) with potentially different expected payouts. You want to find the slot machine with the best payout rate, but you also want to maximize your winnings. The fundamental tension is between “exploiting” arms that have performed well in the past and “exploring” new or seemingly inferior arms in case they might perform even better. There are highly developed mathematical models for managing the bandit problem, which we use in Google Analytics content experiments.

Using the Multi-Arm Bandit algorithm helps minimize this waste. Our early calculations proved that it could lead to nearly double the actual number of conversions for our customers running A/B tests.

Once a test is over, and there is a winner, can you stop it and show the winning variation?

Yes – You can stop a test at any time. When you stop a test the A version of the page will be on the test URL… but you can simply un-publish the A version and publish the B version to the final URL if you were happy with it’s performance.