SlideShare a Scribd company logo

A/B Testing Pitfalls - MeasureCamp London 2015

•
•
Michal Parizek
Michal Parizek

A quick presentation about A/B testing pitfalls you should be aware of. Things that influence your A/B tests and the results you get out them. I

Related slideshows

Cross Border Selling: Breaking the Language Barrier with Automated Translation
Cross Border Selling: Breaking the Language Barrier with Automated TranslationCross Border Selling: Breaking the Language Barrier with Automated Translation
Cross Border Selling: Breaking the Language Barrier with Automated Translation
 
Powering IBP with Modeling and Optimization
Powering IBP with Modeling and OptimizationPowering IBP with Modeling and Optimization
Powering IBP with Modeling and Optimization
 
Mobile metrics - eMetrics Berlin 2015
Mobile metrics - eMetrics Berlin 2015Mobile metrics - eMetrics Berlin 2015
Mobile metrics - eMetrics Berlin 2015
 
1 of 7
Download to read offline
A/B Testing Pitfalls Michal Parizek (@parezem), Avast #MeasureCamp , March 2015
Low-hanging Fruit (1) 2 business cycles (2) Big enough data sample (minimum of 200 orders per testing experience (3) No bugs in A/B test setup (4) Daily orders/revenue + cumulative orders/revenue (5) Check http://abtestguide.com/calc/ @parezem | #MeasureCamp
Traffic Mix & Seasonality Challenge A/B test results are tight to the traffic and circumstances of a testing period Solution (1) Make sure all testing experiences get the same traffic mix. (2) Avoid special commerce events for A/B testing (Christmas, Black Friday, Valentine’s day etc.). (3) In case you have a seasonal business, A/B test your hypotheses in both on and off season. 3 @parezem | #MeasureCamp
Cross-device A/B Testing Challenge Attribution in cross-device A/B testing. One user, different devices, not the same testing experience Solution (1) Use targeting only to one device type - not solve multiple same 
 device type issue (2) Wait when tools add “user-centric testing” @parezem | #MeasureCamp
Long Purchase Decision Making Process Challenge Customers from your A/B test made the actual decision before your A/B test was launched Solution (1) Target new visitors only. (2) Set micro-conversion goals when an A/B test focuses on an early part of the purchase process First visit on the LP T = 0 T + 28 Purchase!Third visit on the LP T + 12 Your A/B is launched T + 20 Research Finances, laziness, more research, discount coupons @parezem | #MeasureCamp
Research Online, Purchase Offline Challenge Your online A/B tests influence offline purchases too Solution (1) Show discount coupons for offline purchases - each testing experience has unique discount coupon.
 
 Experience A = DSC08A
 Experience B = DSC08B @parezem | #MeasureCamp
Optimising for Maximum CLTV Challenge You don’t want to get more average customers. You do want to get more excellent customers! Solution (1) A survey after a purchase - “Would you recommend us to your friend?” (2) Re-evaluate your A/B tests after few months. (3) Patterns with your historical data - predict CLTV. (4) Immediate insights: e.g. a share of auto-renewal customers buy again recommend stay loyal upgrade @parezem | #MeasureCamp

More Related Content

A/B Testing Pitfalls - MeasureCamp London 2015

  • 1. A/B Testing Pitfalls Michal Parizek (@parezem), Avast #MeasureCamp , March 2015
  • 2. Low-hanging Fruit (1) 2 business cycles (2) Big enough data sample (minimum of 200 orders per testing experience (3) No bugs in A/B test setup (4) Daily orders/revenue + cumulative orders/revenue (5) Check http://abtestguide.com/calc/ @parezem | #MeasureCamp
  • 3. Traffic Mix & Seasonality Challenge A/B test results are tight to the traffic and circumstances of a testing period Solution (1) Make sure all testing experiences get the same traffic mix. (2) Avoid special commerce events for A/B testing (Christmas, Black Friday, Valentine’s day etc.). (3) In case you have a seasonal business, A/B test your hypotheses in both on and off season. 3 @parezem | #MeasureCamp
  • 4. Cross-device A/B Testing Challenge Attribution in cross-device A/B testing. One user, different devices, not the same testing experience Solution (1) Use targeting only to one device type - not solve multiple same 
 device type issue (2) Wait when tools add “user-centric testing” @parezem | #MeasureCamp
  • 5. Long Purchase Decision Making Process Challenge Customers from your A/B test made the actual decision before your A/B test was launched Solution (1) Target new visitors only. (2) Set micro-conversion goals when an A/B test focuses on an early part of the purchase process First visit on the LP T = 0 T + 28 Purchase!Third visit on the LP T + 12 Your A/B is launched T + 20 Research Finances, laziness, more research, discount coupons @parezem | #MeasureCamp
  • 6. Research Online, Purchase Offline Challenge Your online A/B tests influence offline purchases too Solution (1) Show discount coupons for offline purchases - each testing experience has unique discount coupon.
 
 Experience A = DSC08A
 Experience B = DSC08B @parezem | #MeasureCamp
  • 7. Optimising for Maximum CLTV Challenge You don’t want to get more average customers. You do want to get more excellent customers! Solution (1) A survey after a purchase - “Would you recommend us to your friend?” (2) Re-evaluate your A/B tests after few months. (3) Patterns with your historical data - predict CLTV. (4) Immediate insights: e.g. a share of auto-renewal customers buy again recommend stay loyal upgrade @parezem | #MeasureCamp