I’m running a split test on a lead capture form using Visual Website Optimizer and seeing something strange. The test has been running since 3/7, and the results are below. The issue is that on 3/7 (day 1), the test for some reason had zero conversions, yet since 3/8, it has substantially outperformed the control. My question: is it appropriate to exclude the results from 3/7?
If you look at the full data set (3/7 – 3/20), you can see that the test is showing a lift of 12% at 74% confidence:
But, if you exclude 3/7, you get a 26% lift at 90% confidence (this has actually increased since I created the image):
I normally don’t exclude outliers out of convenience, but the fact that there were literally zero conversions for the test on 3/7 is so strange that I can’t help but think something was broken. 67% of conversions on this site happen on the first visit, so while we might expect a lower conversion rate on day 1, it shouldn’t be zero. Additionally, the conversion rate for the control on 3/7 is 2x its normal average. In other words, for that one day, the test is 100% worse than average and the control is 100% better…seems suspicious.
So, what do you think? Do you exclude day 1? Am I missing something? Leave a comment or hit me up on Twitter @AndrewSzatan.