SlideShare a Scribd company logo

10 Tactics for Building an Optimization Culture

•
•
Optimizely
Optimizely

Slides from a presentation of '10 Tactics for Building an Optimization Culture' webinar, hosted by Brooks Bell and Optimizely. Full webinar recording with audio can be found here: http://optimizely.wistia.com/medias/xf4yk47rml https://www.optimizely.com/ http://brooksbell.com/

1 of 25
Download to read offline
10 Tactics for Building an Optimization Culture OPTIMIZELY & BROOKS BELL 3.4.2014
Welcome! Gregory Ng Chief Marketing Officer Brooks Bell @GregoryNg Kyle Rush Head of Optimization Optimizely @kylerush
10 Tactics for Developing a Successful Experimentation Culture
What we’ll cover ... 1. Best practices from top testing organizations 2. The people behind testing 3. How to organize your program 4. How to keep the process manageable 5. How to include others and build excitement
Successful experimentation culture must be data-driven.
What does it mean to be data-driven? Progress is compelled by data, rather than by intuition or personal experience. Evidence-based decision making. Using data to inform key decisions.
How data-driven are you, actually? • Do you dictate strategy without referencing data? • Do you rely on the person with the most experience to make the ultimate call? • Do you use words like “think” and “feel” when talking about strategy? • Does your team trust the data?
How do you establish a successful testing culture?
10 Tactics 1. Clear organizational structure 2. Identify the right talent 3. Establish stakeholders 4. Solicit test ideas from everyone 5. Prioritize tests 6. Share results and successes 7. Build a knowledge base 8. Don’t rush into MVT tests 9. Balance learning and “wins” 10.Go beyond CRO
#1 – Choose the right organization structure • No “one-size-fits-all” • One or multi-person team • Centralized or decentralized? Members of the Obama for America Team
#1 – Choose the right organization structure • Army of One • Ringleader • Fellowship • Kingdom
#2 – Pick the Right Talent • Qualitative and quantitative combination • Curiosity, creativity • People skills • Analytical mindset VP of Marketing E-Commerce Director Program Manager, Retail Optimizely’s customers come from a variety of backgrounds Product Manager
#3 – Establish Stakeholders Early On • Identify the correct stakeholders • Secure an executive sponsor • Develop an approval process for tests • Communicate and share successes
#4 – Solicit Test Ideas from Everyone • Make it easy to submit ideas • Capture interest after sharing results • Follow up for every idea • Create a scoreboard • Attribute experiments to team members Example: Optimizely Scorecard Vote on predicted winning variations
#5 - Know how to Prioritize Tests ESTABLISH A SCORECARD •Relevance to business goals (KPIs) •Potential impact •Potential for learning •Level of effort •Time to significance and available traffic •Likelihood of implementation
#6 – Share Results and Successes • • • • • Make it easy to understand Clarify the primary goal Share 100 percent of results, regardless of outcome Standing meetings Keep score Presenting A/B test findings at company all-hands
#7 – Build a Knowledge Base • Gain broader insights • Lay a foundation for growth • Eliminate single point of failure • Increase accountability • Standardize processes and documentation
#8 – Don’t Rush Into MVT Tests • Ideal for fine-tuning optimization • More complicated but less effective • Fewer learnings
#9 – Balance Learning and “Wins” • • • • Always start with a hypothesis Never settle for a flat result Invitation for deep analysis Segment results, target accordingly
#10 – Go Beyond CRO Optimization doesn’t need to occur in isolation. Almost anyone can test. For instance ...
#10 – Go Beyond CRO • Designers Variation A Variation B Walk designers through A/B tests so they can inform their designs with data
#10 – Go Beyond CRO • Engineers Launch of OFA 2012 New Donation API A/B testing the new platform revealed an architectural bug that was missed in QA
#10 – Go Beyond CRO • Email and Content Marketers A/B Tested Emails and Landing Pages for Subscribers #dataculture
1. Clear organizational structure 2. Identify the right talent 3. Establish stakeholders 4. Solicit test ideas from everyone In summary ... 5. Prioritize tests 6. Share results and successes 7. Build a knowledge base 8. Don’t rush into MVT tests 9. Balance learning and “wins” 10.Go beyond CRO
Thank you for reading!

More Related Content

10 Tactics for Building an Optimization Culture

  • 1. 10 Tactics for Building an Optimization Culture OPTIMIZELY & BROOKS BELL 3.4.2014
  • 2. Welcome! Gregory Ng Chief Marketing Officer Brooks Bell @GregoryNg Kyle Rush Head of Optimization Optimizely @kylerush
  • 3. 10 Tactics for Developing a Successful Experimentation Culture
  • 4. What we’ll cover ... 1. Best practices from top testing organizations 2. The people behind testing 3. How to organize your program 4. How to keep the process manageable 5. How to include others and build excitement
  • 6. What does it mean to be data-driven? Progress is compelled by data, rather than by intuition or personal experience. Evidence-based decision making. Using data to inform key decisions.
  • 7. How data-driven are you, actually? • Do you dictate strategy without referencing data? • Do you rely on the person with the most experience to make the ultimate call? • Do you use words like “think” and “feel” when talking about strategy? • Does your team trust the data?
  • 8. How do you establish a successful testing culture?
  • 9. 10 Tactics 1. Clear organizational structure 2. Identify the right talent 3. Establish stakeholders 4. Solicit test ideas from everyone 5. Prioritize tests 6. Share results and successes 7. Build a knowledge base 8. Don’t rush into MVT tests 9. Balance learning and “wins” 10.Go beyond CRO
  • 10. #1 – Choose the right organization structure • No “one-size-fits-all” • One or multi-person team • Centralized or decentralized? Members of the Obama for America Team
  • 11. #1 – Choose the right organization structure • Army of One • Ringleader • Fellowship • Kingdom
  • 12. #2 – Pick the Right Talent • Qualitative and quantitative combination • Curiosity, creativity • People skills • Analytical mindset VP of Marketing E-Commerce Director Program Manager, Retail Optimizely’s customers come from a variety of backgrounds Product Manager
  • 13. #3 – Establish Stakeholders Early On • Identify the correct stakeholders • Secure an executive sponsor • Develop an approval process for tests • Communicate and share successes
  • 14. #4 – Solicit Test Ideas from Everyone • Make it easy to submit ideas • Capture interest after sharing results • Follow up for every idea • Create a scoreboard • Attribute experiments to team members Example: Optimizely Scorecard Vote on predicted winning variations
  • 15. #5 - Know how to Prioritize Tests ESTABLISH A SCORECARD •Relevance to business goals (KPIs) •Potential impact •Potential for learning •Level of effort •Time to significance and available traffic •Likelihood of implementation
  • 16. #6 – Share Results and Successes • • • • • Make it easy to understand Clarify the primary goal Share 100 percent of results, regardless of outcome Standing meetings Keep score Presenting A/B test findings at company all-hands
  • 17. #7 – Build a Knowledge Base • Gain broader insights • Lay a foundation for growth • Eliminate single point of failure • Increase accountability • Standardize processes and documentation
  • 18. #8 – Don’t Rush Into MVT Tests • Ideal for fine-tuning optimization • More complicated but less effective • Fewer learnings
  • 19. #9 – Balance Learning and “Wins” • • • • Always start with a hypothesis Never settle for a flat result Invitation for deep analysis Segment results, target accordingly
  • 20. #10 – Go Beyond CRO Optimization doesn’t need to occur in isolation. Almost anyone can test. For instance ...
  • 21. #10 – Go Beyond CRO • Designers Variation A Variation B Walk designers through A/B tests so they can inform their designs with data
  • 22. #10 – Go Beyond CRO • Engineers Launch of OFA 2012 New Donation API A/B testing the new platform revealed an architectural bug that was missed in QA
  • 23. #10 – Go Beyond CRO • Email and Content Marketers A/B Tested Emails and Landing Pages for Subscribers #dataculture
  • 24. 1. Clear organizational structure 2. Identify the right talent 3. Establish stakeholders 4. Solicit test ideas from everyone In summary ... 5. Prioritize tests 6. Share results and successes 7. Build a knowledge base 8. Don’t rush into MVT tests 9. Balance learning and “wins” 10.Go beyond CRO
  • 25. Thank you for reading!