Using machine learning to determine drivers of bounce and conversion (part 2)
•Download as PPTX, PDF•
3 likes•1,277 views
[2016 Velocity NY] There has been a lot of historical work that looks at the relationship between performance and conversions, but most of it has been after the fact or relied on linear models. Google partnered with SOASTA to train a machine-learning model on a large sample of real-world performance, conversion, and bounce data. Patrick Meenan and Tammy Everts offer an overview of the resulting model, able to predict the impact of performance work and other site metrics on conversion and bounce rates. The code used to generate the model is freely available.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Using machine learning to determine drivers of bounce and conversion (part 2)
Similar to Using machine learning to determine drivers of bounce and conversion (part 2) (20)
More from Tammy Everts
More from Tammy Everts (12)
Recently uploaded
Recently uploaded (20)
Using machine learning to determine drivers of bounce and conversion (part 2)
- 1. Using machine learning to determine drivers of bounce and conversion (part 2) Velocity 2016 New York
- 3. What we did (and why we did it)
- 7. Random forest Lots of random decision trees
- 8. Vectorizing the data • Everything needs to be numeric • Strings converted to several inputs as yes/no (1/0) • i.e. Device manufacturer • “Apple” would be a discrete input • Watch out for input explosion (UA String)
- 9. Balancing the data • 3% conversion rate • 97% accurate by always guessing no • Subsample the data for 50/50 mix
- 10. Smoothing the data ML works best on normally distributed data scaler = StandardScaler() x_train = scaler.fit_transform(x_train) x_val = scaler.transform(x_val)
- 11. Validation data • Train on 80% of the data • Validate on 20% to prevent overfitting –Training accuracy from validation set
- 12. Input/output relationships • SSL highly correlated with conversions • Long sessions highly correlated with not bouncing • Remove correlated features from training
- 13. Training random forest clf = RandomForestClassifier(n_estimators=FOREST_SIZE, criterion='gini', max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features='auto', max_leaf_nodes=None, bootstrap=True, oob_score=False, n_jobs=12, random_state=None, verbose=2, warm_start=False, class_weight=None) clf.fit(x_train, y_train)
- 15. Training deep learning model = Sequential() model.add(...) model.compile(optimizer='adagrad', loss='binary_crossentropy', metrics=["accuracy"]) model.fit(x_train, y_train, nb_epoch=EPOCH_COUNT, batch_size=32, validation_data=(x_val, y_val), verbose=2, shuffle=True)
- 17. Brute force FTW • 93 input “features” • Train 93 models with 1 input – Measuring the prediction accuracy of each • Train 92 models with 2 inputs – Top feature from first round – Measure combined prediction accuracy • Lather, rinse, repeat…
- 18. Visualizing the model • Take trained model (X inputs) • Vary inputs –100ms to 20 seconds in 100ms intervals • Apply the data smoothing from training set • model.predict_proba
- 19. What we learned
- 20. What’s in our beacon? • Top-level – domain, timestamp, SSL • Session – start time, length (in pages), total load time • User agent – browser, OS, mobile ISP • Geo – country, city, organization, ISP, network speed • Bandwidth • Timers – base, custom, user-defined • Custom metrics • HTTP headers https://docs.soasta.com/whatsinbeacon/
- 21. Finding 1 Maybe everything doesn’t matter after all
- 22. Bounce rate
- 23. Finding 2 DOM ready (aka DOM content loaded) and average session load time were the best indicators of bounce rate
- 24. Up to 89.5% accuracy
- 26. Finding 3 When it came to getting high predictability, conversion data was tougher than bounce data
- 27. 81% prediction accuracy was as high as we got
- 28. Finding 4 Pages with more scripts were more less likely to convert
- 30. Finding 5 The number of DOM elements matters… a lot
- 32. Finding 6 Mobile-related measurements weren’t meaningful predictors of conversions
- 34. Finding 7 Some conventional metrics were not as important as we thought
- 35. Feature Importance (bounce) Start render 69 ~top 3
- 36. Things to watch out for (other than dangling prepositions)
- 37. Yep, checkout pages are SLOW
- 39. Takeaways
- 40. 1. YMMV 2. Do try this at home 3. Gather your RUM data (lots of it) 4. Run the machine learning against it 5. If you get unexpected results, keep digging
Editor's Notes
- mPulse is built above the boomerang JavaScript library that collects web performance data from a user’s web browser and sends that back to the mPulse servers on a beacon. The simple definition of a beacon is that it is an HTTP(S) request with a ton of data included either as HTTP headers or as part of the Request’s Query String.
- “DOM ready” refers to the amount of time it takes for the page’s HTML to be received and parsed by the browser. Actual page elements, such as images, haven’t appeared yet. (It’s kind of like getting ready to cook. Your cookbook is open, your recipe is in front of you, and your ingredients are on standby.)
- “DOM ready” refers to the amount of time it takes for the page’s HTML to be received and parsed by the browser. Actual page elements, such as images, haven’t appeared yet. (It’s kind of like getting ready to cook. Your cookbook is open, your recipe is in front of you, and your ingredients are on standby.) It is the same as the DOM Content Loaded event in nav timing but the polyfill version that works across all browsers.
- DOM_ready + load time gets us up to 89.5% accuracy on the predictions Takeaway: External blocking scripts (such as third-party ads, analytics, and social widgets) and styles (such as externally hosted CSS and fonts) have the greatest impact on DOM ready times. Site owners should measure the impact that these external elements have on their pages and conduct ongoing monitoring to ensure that scripts and styles are available and fast. Whenever possible, scripts should be served asynchronously (in parallel with the rest of the page) or in a non-blocking fashion.
- DOM_ready + load time gets us up to 89.5% accuracy on predictions
- Sessions that converted contained 48% more scripts (including third-party scripts, such as ads, analytics beacons, and social buttons) than sessions that didn’t.
- Before around 300 scripts, it's possible that it learned the patterns of what some checkout flows looked like. Scripts are one of those things that may be more fixed than timings so it might be easier for deep learning to just learn what all sites checkout flows look like.
- max_params_dom_In (number of DOM elements) -- the more complex pages after (1000?) elements starts to fall off.
- Median_bandwidth_kbps was 44 User_agent_device_type was 79 Mobile_connection_type was 89 Shoppers who used low-bandwidth or mobile connections didn’t convert significantly less than shoppers on faster connections. This is interesting because it confirms that we’ve entered a “mobile everywhere” phase. Takeaway: Internet users don’t behave especially differently depending on what device they’re using. Site owners need to ensure they’re delivering consistent user experiences across device types.
- Start render tells you when content begins to display in the user’s browser. Of the 1M records, 720k did not have render start times included (because the browser didn't support it) which is why it ended up being a not-important feature. Pat re-ran the deep learning version of the importances on a filtered dataset that only includes records that also included a render time to see how it looked relative to the other times. Filtered down to just records that also include a start render, start render is basically the same importance as dom ready. There is a similar pattern to the others where it plateaus though it looks like the plateau starts pretty early (around 3 seconds) which generally makes sense since usually render < dom ready < onload. In all cases, there doesn't seem to be a point where it isn't worth making it faster. If anything, the gains become more significant as you get closer to zero.