SlideShare a Scribd company logo

ASTQB washington-sept-2015

•Download as PPTX, PDF•
•
Dan Boutin
Dan Boutin

The document discusses how SOASTA collects billions of real user beacons from user experiences on websites and mobile apps. It describes SOASTA's data science workbench, which loads the beacon data into a highly optimized schema in Amazon Redshift. This allows data scientists to analyze the data using the Julia programming language and build statistical models and functions. The workbench is aimed at helping customers understand user behavior and identify performance issues based on insights from the large volume of real user data.

1 of 49
September 14, 2015 “Let’s turn Real User Data into a Science!” Dan Boutin – Senior Product Evangelist
© 2014 SOASTA. All rights reserved. March 3, 2015 2 Agenda • Who are we, and why are we here? • Performance Testing over the Years – History Lesson • Now Let’s Talk Architecture! • Trade-Offs & Deep Dive • The Result! • Session II: It’s Your Turn!
© 2014 SOASTA. All rights reserved. September 17, 2015 3 Who are We and Why are we here? • We are….Performance Experts. • …with a Data Science component • We collect Billions of Real User Beacons • …What’s a beacon? • …Where do we get it?
© 2014 SOASTA. All rights reserved. September 17, 2015 4 Who are We and Why are we here? • How Are We Different? • User Experience Beacon Collection All Of The Data KG All Of The Detail All Of The Time Kept Forever • What do we do with these Billions of Real User Beacons? • …we keep them….which could have been a challenge….
© 2014 SOASTA. All rights reserved. September 17, 2015 5 100 Billion User Experiences Tested 10 Million Tests Performed Actual CloudTest view Who are We and Why are we here?
© 2014 SOASTA. All rights reserved. How did all this start? o 1989 o 1995 o 2007!
© 2014 SOASTA. All rights reserved. Fear Factor o “We don’t test in production.”
© 2014 SOASTA. All rights reserved. Automated Grid Provisioning Your environment must be flexible & scalable
© 2014 SOASTA. All rights reserved. You need a Kill Switch – No Fear Factor!
© 2014 SOASTA. All rights reserved. Real-time Performance Analysis You need drill down by Load Generation Location
© 2014 SOASTA. All rights reserved. Detailed Error Analysis You need detailed error analysis - LIVE
© 2014 SOASTA. All rights reserved. Multi-Test-Run Comparison Compare results of a LIVE test with previous test executions You need to know: Are we better than last time?
© 2014 SOASTA. All rights reserved. Detailed Transaction, Page, and URL Analysis • Detailed Transaction and Page Analysis of Web and Mobile Load Tests • Detailed URL Analysis of Web and Mobile Load Tests You need web & mobile analysis
© 2014 SOASTA. All rights reserved. Run Globally Distributed Load Tests with Akamai Your analytics should have visibility into your CDN
© 2014 SOASTA. All rights reserved. Detailed Page Component Breakdown Your analytics should have visibility into your CDN
© 2014 SOASTA. All rights reserved. Reflects growth in cloud hours – Amazon only! (17 other providers!) 7 Year Growth of cloud testing: SOASTA & Amazon ….goodbye Fear Factor…
© 2014 SOASTA. All rights reserved. Testing in Production – Why Not? o What is the value added? • CDN Tests (Not configured to serve up new content) • Batch Jobs are not present in the lab • Misconfigured App & Web servers • Thread & Connection Pool settings • Bandwidth Constraints • Radically different performance on different database sizes
© 2014 SOASTA. All rights reserved. Testing in Production – Why Not? o So what should the process look like?
© 2014 SOASTA. All rights reserved. o So what should the process look like?
Test Takeaway: Building the tests
CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015 Fix Your process! No more outdated test creation
© 2014 SOASTA. All rights reserved. o So what should the process look like?
CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015 Analyze the most common session paths of real users
© 2014 SOASTA. All rights reserved. How Do Users Move Through Your Site?
ASTQB washington-sept-2015
© 2014 SOASTA. All rights reserved. New Way to Pinpoint Performance Problems
© 2014 SOASTA. All rights reserved. Test Takeaway What did we learn? Revenue Brand Competitive advantage
© 2014 SOASTA. All rights reserved. o So what should the process look like?
mPulse What’s a Beacon? www.w3.org/TR/Beacon Total Beacons Collected since 6/2013: ~ 85 Billion Run rate over 3B per week and growing Projected ~ 175B by 1/1/166
Big Data Challenges Data Scientists spend too much time ‘data wrangling’ “Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.” NY Times – August 17th, 2014
Big Data Challenges Building a data science platform is very difficult Infrastructure •Choosing big data technologies and setting up a cluster can easily take 9 months or more Data Pipeline •Building a high performing big data schema requires specialized skills •Extracting, transforming, and loading of data (data wrangling) is an enormous time sink and a poor use of data scientists time Analysis and Workflow •Figuring out how you can ask questions of the data and how to visualize the results takes time that data scientists should be using to generate actionable insights from their studies
Julia Language & iJulia Notebook UI Julia is a rising star in scientific programming processing speed support for parallel processing compatibility with 400+ prebuilt statistical packages large number and growing number of visualization libraries. Trade-Offs & Deep Dive
Julia Language & iJulia Notebook UI www.julialang.org processing speed support for parallel processing compatibility with 400+ prebuilt statistical packages large number and growing number of visualization libraries. Where can I find Julia?
Trade-Offs & Deep Dive
© 2014 SOASTA. All rights reserved. September 17, 2015 35 Trade-Offs & Deep Dive o Amazon Redshift is a fully managed, petabyte- scale data warehouse service in the cloud. • Columnar Database • Extremely fast query times • Attractive Economics
© 2014 SOASTA. All rights reserved. September 17, 2015 36 Now Let’s Talk Architecture
Data Science WorkbenchData Science without the data wrangling, and much more Infrastructure Data PipelineAnalysis and Workflow • Data Science Workbench comes with the state-of-the-art technology you need to analyze your customer experiences • All of the real user beacon data is loaded into Data Science Workbench into a highly optimized schema ready for analysis • Data science is done with Julia, a remarkably fast and in-memory solution for analyzing huge data-sets • Access to an ever growing library of analysis functions and visualizations based on SOASTA’s and our customers’ expertise
ASTQB washington-sept-2015
© 2014 SOASTA. All rights reserved. September 17, 2015 39 The Result! • Every customer beacon unpacked, transformed and loaded nightly by SOASTA into a SOASTA designed Schema in Amazon Redshift. This process designed, supplied and supported by SOASTA • Amazon Redshift is an extremely inexpensive and powerful BIG DATA database that can scale to almost 2 Petabytes in size. Amazon estimates compute and storage costs of $1,000/TB/month for our implementation • An online, interactive explore, discover and develop interface based on the Julia scientific programming language developed at MIT and the iJulia Notebook UI • SOASTA developed Functions & Statistical Models
Let’s take a look at what we have! A peek at how we use these Technologies
Our Data Science Workbench
Discussion? Our Data Science Workbench
Our Data Science Workbench
Our Data Science Workbench
Let’s look deeper inside the DSWB
The future is here!
© 2014 SOASTA. All rights reserved. Trivia Questions on Technology that apply today @DanBoutinSOASTA
ASTQB washington-sept-2015
September 14, 2015 “Let’s turn Real User Data into a Science!” Dan Boutin – Senior Product Evangelist

More Related Content

ASTQB washington-sept-2015

  • 1. September 14, 2015 “Let’s turn Real User Data into a Science!” Dan Boutin – Senior Product Evangelist
  • 2. © 2014 SOASTA. All rights reserved. March 3, 2015 2 Agenda • Who are we, and why are we here? • Performance Testing over the Years – History Lesson • Now Let’s Talk Architecture! • Trade-Offs & Deep Dive • The Result! • Session II: It’s Your Turn!
  • 3. © 2014 SOASTA. All rights reserved. September 17, 2015 3 Who are We and Why are we here? • We are….Performance Experts. • …with a Data Science component • We collect Billions of Real User Beacons • …What’s a beacon? • …Where do we get it?
  • 4. © 2014 SOASTA. All rights reserved. September 17, 2015 4 Who are We and Why are we here? • How Are We Different? • User Experience Beacon Collection All Of The Data KG All Of The Detail All Of The Time Kept Forever • What do we do with these Billions of Real User Beacons? • …we keep them….which could have been a challenge….
  • 5. © 2014 SOASTA. All rights reserved. September 17, 2015 5 100 Billion User Experiences Tested 10 Million Tests Performed Actual CloudTest view Who are We and Why are we here?
  • 6. © 2014 SOASTA. All rights reserved. How did all this start? o 1989 o 1995 o 2007!
  • 7. © 2014 SOASTA. All rights reserved. Fear Factor o “We don’t test in production.”
  • 8. © 2014 SOASTA. All rights reserved. Automated Grid Provisioning Your environment must be flexible & scalable
  • 9. © 2014 SOASTA. All rights reserved. You need a Kill Switch – No Fear Factor!
  • 10. © 2014 SOASTA. All rights reserved. Real-time Performance Analysis You need drill down by Load Generation Location
  • 11. © 2014 SOASTA. All rights reserved. Detailed Error Analysis You need detailed error analysis - LIVE
  • 12. © 2014 SOASTA. All rights reserved. Multi-Test-Run Comparison Compare results of a LIVE test with previous test executions You need to know: Are we better than last time?
  • 13. © 2014 SOASTA. All rights reserved. Detailed Transaction, Page, and URL Analysis • Detailed Transaction and Page Analysis of Web and Mobile Load Tests • Detailed URL Analysis of Web and Mobile Load Tests You need web & mobile analysis
  • 14. © 2014 SOASTA. All rights reserved. Run Globally Distributed Load Tests with Akamai Your analytics should have visibility into your CDN
  • 15. © 2014 SOASTA. All rights reserved. Detailed Page Component Breakdown Your analytics should have visibility into your CDN
  • 16. © 2014 SOASTA. All rights reserved. Reflects growth in cloud hours – Amazon only! (17 other providers!) 7 Year Growth of cloud testing: SOASTA & Amazon ….goodbye Fear Factor…
  • 17. © 2014 SOASTA. All rights reserved. Testing in Production – Why Not? o What is the value added? • CDN Tests (Not configured to serve up new content) • Batch Jobs are not present in the lab • Misconfigured App & Web servers • Thread & Connection Pool settings • Bandwidth Constraints • Radically different performance on different database sizes
  • 18. © 2014 SOASTA. All rights reserved. Testing in Production – Why Not? o So what should the process look like?
  • 19. © 2014 SOASTA. All rights reserved. o So what should the process look like?
  • 21. CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015 Fix Your process! No more outdated test creation
  • 22. © 2014 SOASTA. All rights reserved. o So what should the process look like?
  • 23. CONFIDENTIAL – Not for Distribution © 2015 SOASTA. All rights reserved. January 13, 2015 Analyze the most common session paths of real users
  • 24. © 2014 SOASTA. All rights reserved. How Do Users Move Through Your Site?
  • 26. © 2014 SOASTA. All rights reserved. New Way to Pinpoint Performance Problems
  • 27. © 2014 SOASTA. All rights reserved. Test Takeaway What did we learn? Revenue Brand Competitive advantage
  • 28. © 2014 SOASTA. All rights reserved. o So what should the process look like?
  • 29. mPulse What’s a Beacon? www.w3.org/TR/Beacon Total Beacons Collected since 6/2013: ~ 85 Billion Run rate over 3B per week and growing Projected ~ 175B by 1/1/166
  • 30. Big Data Challenges Data Scientists spend too much time ‘data wrangling’ “Data scientists, according to interviews and expert estimates, spend from 50 percent to 80 percent of their time mired in this more mundane labor of collecting and preparing unruly digital data, before it can be explored for useful nuggets.” NY Times – August 17th, 2014
  • 31. Big Data Challenges Building a data science platform is very difficult Infrastructure •Choosing big data technologies and setting up a cluster can easily take 9 months or more Data Pipeline •Building a high performing big data schema requires specialized skills •Extracting, transforming, and loading of data (data wrangling) is an enormous time sink and a poor use of data scientists time Analysis and Workflow •Figuring out how you can ask questions of the data and how to visualize the results takes time that data scientists should be using to generate actionable insights from their studies
  • 32. Julia Language & iJulia Notebook UI Julia is a rising star in scientific programming processing speed support for parallel processing compatibility with 400+ prebuilt statistical packages large number and growing number of visualization libraries. Trade-Offs & Deep Dive
  • 33. Julia Language & iJulia Notebook UI www.julialang.org processing speed support for parallel processing compatibility with 400+ prebuilt statistical packages large number and growing number of visualization libraries. Where can I find Julia?
  • 35. © 2014 SOASTA. All rights reserved. September 17, 2015 35 Trade-Offs & Deep Dive o Amazon Redshift is a fully managed, petabyte- scale data warehouse service in the cloud. • Columnar Database • Extremely fast query times • Attractive Economics
  • 36. © 2014 SOASTA. All rights reserved. September 17, 2015 36 Now Let’s Talk Architecture
  • 37. Data Science WorkbenchData Science without the data wrangling, and much more Infrastructure Data PipelineAnalysis and Workflow • Data Science Workbench comes with the state-of-the-art technology you need to analyze your customer experiences • All of the real user beacon data is loaded into Data Science Workbench into a highly optimized schema ready for analysis • Data science is done with Julia, a remarkably fast and in-memory solution for analyzing huge data-sets • Access to an ever growing library of analysis functions and visualizations based on SOASTA’s and our customers’ expertise
  • 39. © 2014 SOASTA. All rights reserved. September 17, 2015 39 The Result! • Every customer beacon unpacked, transformed and loaded nightly by SOASTA into a SOASTA designed Schema in Amazon Redshift. This process designed, supplied and supported by SOASTA • Amazon Redshift is an extremely inexpensive and powerful BIG DATA database that can scale to almost 2 Petabytes in size. Amazon estimates compute and storage costs of $1,000/TB/month for our implementation • An online, interactive explore, discover and develop interface based on the Julia scientific programming language developed at MIT and the iJulia Notebook UI • SOASTA developed Functions & Statistical Models
  • 40. Let’s take a look at what we have! A peek at how we use these Technologies
  • 45. Let’s look deeper inside the DSWB
  • 46. The future is here!
  • 47. © 2014 SOASTA. All rights reserved. Trivia Questions on Technology that apply today @DanBoutinSOASTA
  • 49. September 14, 2015 “Let’s turn Real User Data into a Science!” Dan Boutin – Senior Product Evangelist