SlideShare a Scribd company logo

Make User Experience Part of The KPI Conversation With Universal Measures

•
•
UserZoom
UserZoom

Join Dr. Andrea Peer and learn: -How Universal Measures makes tangible the abstract concept of experience for your organization -How practitioners can make experience a critical KPI for their organization -Ways to establish experience score goals for all lines of business -The benefits Universal Measures brings to executives and stakeholders

1 of 58
Download to read offline
MAKE USER EXPERIENCE PART OF THE KPI CONVERSATION WITH UNIVERSAL MEASURES Dr. Andrea Peer | UserZoom
Agenda Where Did KPIs Come From? WEBINAR SPEAKER Dr. Andrea Peer Onboarding Customer Success Manager at UserZoom Contact Info Connecting Experience to ROI Consider an eXperience Score as a KPI Universal Measures to grow experience practice Q&A 1 2 3 4 5
Quick Housekeeping • Control panel on the side of your screen if you have any comments during the presentation • Time at the end for Q&A • Today’s webinar will be recorded for future viewing • All attendees will receive a copy of the slides/recording • Continue the discussion using #uzwebinar Let’s make sure you’re all set up for the webinar!
Where did KPIs come from?
Where did KPIs come from? INDICATORS 20th Century - Fredrick Taylor gave us Scientific Management •  Operational performance •  Efficiency and effectiveness 1920 – DuPont Analysis, ROI •  Financial Indicators 1930’s Tableau de Board…..Dashboard 1992 - Balanced Score Card •  Strategic Performance
Cue the world of dashboards and information vomit
So Many KPIs Financial Indicators System Produced indicators Experience indicators Return On Investment Transactions Net Profit/LossCash Flow Capital Reserves / Debt Site Traffic Bounce Rate Click Analytics CTA Engagement CSat NPS Task completion Time on Task Satisfaction Ease of Use SUS SUPRQ SUM Journey Metrics
One Score Obsession Snapstreaks quantify a friendship in Snapchat Financial Score communicates your financial health The number you need to increase customer loyalty Helps to focus and ground Simplifies abstract concepts
Poll Results
Poll Results
Connecting experience measures to ROI
The KPI Conversation Financial Indicators System Produced indicators Experience indicators Return On Investment Transactions Net Profit/LossCash Flow Capital Reserves / Debt Site Traffic Bounce Rate Click Analytics CTA Engagement CSat NPS Task completion Time on Task Satisfaction Ease of Use SUS SUPRQ SUM Journey Metrics
Tying UX to ROI 1 CHAPTER Justifying Cost-Justifying Usability Randolph G. Bias School of Information, The University of Texas at Austin Clare-Marie Karat IBM TJ Watson Research Center 1.1 INTRODUCTION Sir Walter Scott wrote, in Marmion, "Oh! what a tangled web we weave When first we practise to deceive!" (Scott, 1805) Sometimes it is quoted as "... what a wicked Web we weave." This year is the bicentennial anniversary of the publication of Scott's book. Did he anticipate the World Wide Web, 200 years ago? Scott's words are relevant to today's networked world and the development activities that support it, includ- ing designing for ease of use. Usability is not the end-all-be-all; rather, it must be considered alongside (and in equal measure with!) functionality and schedule. The problem is not that Web site developers are wicked; rather, they are too often in a hurry and are not oper- ating in a reward structure that motivates attention to usability. So the Web is not wicked. But tangled?. Oh yeah. Most of us are wowed by the Internet. It is amazing what we can accomplish while sitting at home in our pajamas. But when we say "us" or "we," we are refer- ring to that small subset of the population who have some expectation of what will happen if we "right click," or those of us who have heard of, for example, Doug Engelbart. When one of "us" sits down with one of "them" (the great unwashed masses), the madcap hilarity begins. "They" are so stupid! Can you believe it?, some of them don't even know that an underline, on a Web page, means the text is a link to another page (except when it isn't). "They" don't even v1, Feb 2005. Includes material adapted from the chapter: Cost benefits framework and case studies, in Bias, R.G. & Mayhew, D.J. (2005) Cost-Justifying Usability: An Update for the Internet Age. 1 Cost benefits evidence and case studies Nigel Bevan Serco Usability Services nbevan@usability.serco.com Contents 1 – Potential benefits of usability 2 1.1 Development savings 4 1.2 Increase e-commerce sales 5 1.3 Product sales 7 1.4 Usage benefits for employers 8 1.5 Reduce support and maintenance costs 10 2 – Estimating costs 11 2.1 User centered design methods 11 2.2 Costs of user centered design 13 3 – Making the cost-benefit case 14 4 – Case studies 16 4.1 Israel Aircraft Industries 16 4.2 Inland Revenue/EDS 23 4.3 Comparison 26 4.4 Taking up the methods 27 Acknowledgements 28 References 28 This paper summarises the benefits that can be obtained from use of user centered design, and discusses how to select appropriate methods and justify their cost benefits. It includes two case studies of the cost benefits of using a usability maturity model to improve the usability capability of an organisation. Usability Return on Investment (ROI) By Jakob Nielsen, J.M. Berger, and Shuli Gilutz 3rd edition 48105 WARM SPRINGS BLVD. FREMONT, CA 94539–7498 USA WWW.NNGROUP.COM Copyright © Nielsen Norman Group; All Rights Reserved. To buy a copy, download from http://www.nngroup.com/reports/roi NN/g, 2008 Bevan, 2005 Bias, 2005
Tying UX to ROI Scribecreatedbywww.PopulationDesign.com © 2010 Human Factors International, Inc. All rights reserved. www.HumanFactors.com | 800.242.4480 USA & Canada | +1.641.472.4480 International
Tying UX to ROI Is ROI an Effective Approach for Persuading Decision-Makers of the Value of User-Centered Design? Susan Dray, Moderator Dray & Associates, Inc. Minneapolis, MN USA dray@acm.org Clare-Marie Karat IBM T. J. Watson Research Center Hawthorne, NY USA ckarat@us.ibm.com Daniel Rosenberg Oracle Redwood Shores, CA USA daniel.Rosenberg @oracle.com David Siegel Dray & Associates, Inc. Minneapolis, MN USA david.siegel@ acm.org Dennis Wixon Microsoft Redmond, WA USA denniswi@ microsoft.com ABSTRACT This panel examines the utility and effectiveness of various ways of making the business case for user-centered design (UCD). Most of the discussion in our field has assumed that measuring and demonstrating ROI for usability is the key to this effort. However, experience shows that the most brilliant ROI analysis may not win the day in the real world of business. Our panelists range from people who claim that ROI is an important persuasive tool as long as the communication about ROI is happening within a healthy business relationship, to people who claim that a focus on ROI can actually be destructive. We also explore the idea that there are important business contexts where ROI simply does not fit. Through the presentations by the panelists and through discussion of a business case scenario, we explore some alternatives to ROI in making the business case for user-centered design. Author Keywords Return on Investment, ROI, cost-justification, UCD ACM Classification Keywords H5.3. Group and Organization Interfaces (eval/methods) INTRODUCTION Over the years, there has been much discussion in the field of usability about how to objectively and convincingly demonstrate the benefits that usability produces. This is not simply an academic exercise, but rather is part of our profession’s effort to prove its value and thus promote adoption of UCD methods. Most of the discussion in the field has focused on measuring and demonstrating return on investment (ROI) for usability, on the assumption that a well-constructed ROI analysis is an effective persuasive tool. Although we certainly hope that accuracy has something to do with effectiveness of an ROI analysis, it is also possible that the most brilliant ROI analysis will not win support for UCD in the real world of business. If we are to understand the strengths and weakness of the ROI paradigm as an approach to persuasion, we must examine questions such as: • What are the inherent strengths and weaknesses of ROI arguments? • What makes for an effective ROI argument, beyond the quality of the calculations? • What types of factors, besides ROI, make compelling cases for UCD? • Can we identify situations where ROI arguments, or other approaches are inherently more or less compelling? • What are the most challenging UCD benefits to create a compelling argument for? How can ROI and other approaches address them? The following sections give an overview of contrasting perspectives on answers to questions like these. HUMAN DYNAMICS, GOALS, AND BUSINESS CONTEXT Beyond the quality of the data on which the ROI calculation is based, a number of factors relating to the dynamics of the business are critical in determining the effectiveness of ROI arguments. First and foremost are the people involved. The person who hears the argument must have a strategic business vision or at least be a pragmatist who will experiment with new methods. If the usability professional is able to make a credible presentation of the ROI argument for usability and build trust with the visionary person, then the stage is set for the next critical factor. A second factor is how well the ROI arguments tie to the business goals and strategies of the organization. If the usability goals can help to achieve organizational goals or even take them a step further, this approach can demonstrate to the organization a new type of partnership they find very valuable. A final critical factor involves world events that are affecting the business world. One is the global marketplace and the movement of jobs and work to areas of the world where people can work at lower cost. Other significant events are the impending retirement of the "Baby Boom" generation with a resulting skills shortage, and increasing life expectancy rates in many parts of the world. If an ROI argument can be heard by visionaries, and will help them to achieve business goals or make strategies successful, and if Copyright is held by the author/owner(s). CHI 2005, April 2–7, 2004, Portland, Oregon, USA. ACM 1-59593-002-7/05/0004. CHI 2005 | Panels April 2-7 | Portland, Oregon, USA 1168 Dray, 2005
Tying UX to ROI Accomplishment Joy Freedom Delight Connection
Poll Question
Poll Question
Consider an eXperience Score as a Key Performance Indicator
What is your eXperience Score? Your experience score is the indicator that your product and service is having the desired experience outcome.
EXPERIENCE SCORES •  Reduce the fuzziness of the abstract concept of experience •  Define what experience means for your product(s) and service(s) •  Provide 1 score for experience that your organization can track •  Make experience research actionable eXperience Score As A KPI
But wait…Experience CANNOT be accurately conveyed by 1 score! eXperience Score As A KPI
eXperience Score As A KPI
eXperience Score is part of a bigger journey to grow your experience practice. Cue…Universal Measures
1.  Create your eXperience score 2.  Determine the story you want to tell with the score to grow experience practice for your organization 3.  Create your executive dashboards and product scorecards to ensure the experience score is front and center and it supports the story you are driving 4.  Analyze the relationship between your eXperience score and other business KPIs. Universal Measures
1.  Create your eXperience score 2.  Determine the story you want to tell with the score to grow experience practice for your organization 3.  Create your executive dashboards and product scorecards to ensure the experience score is front and center and it supports the story you are driving 4.  Analyze the relationship between your eXperience score and other business KPIs. Universal Measures
How does your organization define & measure the ideal experience with your product(s) or service(s)?
What is your eXperience Score? Your experience score is the indicator that your product and service is having the desired experience outcome.
Define Your Experience Goal What is the intended experience with your product and service? What can people do? See? Know? What and how can people achieve? How can people be? Feel? Understand?
Create your experience score Pick at least 1 measure in each cell
Create your experience score Pick at least 1 measure in each cell Task Completion/ Success Ease of Use SUSJourney Score
What is a eXperience Score? Final eXperience Score - 61% SUS 40% Journey 85% Avg Task Completion 60% Avg Ease of Use 60% System Scores Task Scores EXAMPLE
What is a eXperience Score? System Scores Final eXperience Score - 63% Joy Factor 1 45% Joy Factor 2 80% Joy Factor 3 65%EXAMPLE Avg Task Completion 60% Task Scores
What is a eXperience Score? System Scores Final eXperience Score - 53% SUPRQ 40% Avg Task Completion 60% Confidence 60% Task Scores EXAMPLE
Immediate Considerations •  Can I weight variables in my algorithm that are more important? •  How many variables are too many? •  What if a variable isn’t relevant to my research method? Tactical Questions •  Do I have to change my scales? •  How do you do the calculation? eXperience Score Questions
Long Term Considerations •  Can I change my variables over time? •  Do I report one score across all of my products or per product? •  Do all my products have the same variables? •  What is a good growth goal for my eXperience Score? eXperience Score Questions
1.  Create your eXperience score 2.  Determine the story you want to tell with the score to grow experience practice for your organization 3.  Create your executive dashboards and product scorecards to ensure the experience score is front and center and it supports the story you are driving 4.  Analyze the relationship between your eXperience score and other business KPIs. Universal Measures
Common Stories for Executive Dashboards - The black box of dev story - The pre prod investment story - The post prod baseline and gap story - The portfolio opportunity story - The full spectrum story - The user data powerhouse story - The experience investment vs value story What story do you want to tell?
1.  Create your eXperience score 2.  Determine the story you want to tell with the score to grow experience practice for your organization 3.  Create your executive dashboards and product scorecards to ensure the experience score is front and center and it supports the story you are driving 4.  Analyze the relationship between your eXperience score and other business KPIs. Universal Measures
Creating your Executive Dashboard & Product Scorecards 2 Types of Dashboards Executive UX Dashboard Product UX Scorecard Purpose: Show impact of UX on org and highlight UX win/work Target Audience: Senior Leadership Frequency: Monthly or quarterly Purpose: Show result of UX effort for a specific product line Target Audience: Development team leaders/PMs/POs/etc. Frequency: Bi-Monthly, Monthly
Executive Dashboard •  Let’s deep dive on an executive dashboard that works for your organization. Let’s craft the story you want to tell to your executives. •  What connection can we make to your analytics dataset? •  What connection is important to make to your key business measures? •  What other reports in the business do we need to connect to? •  Determine the process/channel to serve the dashboards to your internal stakeholders •  Determine frequency Product Scorecard •  Let’s deep dive on a product scorecard that works for your organization. Let’s craft the story you need to tell for development and product teams. •  Evaluate PDLC process and systems you can integrate with (ie: Jira, Rally) •  Determine the process/channel to serve the dashboards to your internal stakeholders •  Determine frequency based on PDLC integration strategy Creating your Executive Dashboard & Product Scorecards
42 Experience Scorecard - Quarterly Report (Q1 2016) ROI Impact 20 product lines, 5 personas impacted Prior to Development 53 Severe impact discoveries found and 60% of the discoveries addressed 20 Moderate impact discoveries found and added to the UX backlog Post Development (Beta) 3 Severe impact discoveries found and addressed 2 Moderate impact discoveries found and added to the UX backlog Post Development (Full Prod) 5 Severe impact discoveries found and added to backlog for next sprint 7 Moderate impact discoveries found and added to the UX backlog UX Performance Cadence: 1 research engagement per design sprint Avg Field Time: 5 hours Avg Participants per Study: 32 Persona Representation: 10% Bob, 50% Julie, 40% Chip Research Conversation: 40% Avg SUS +15% Avg Sat +10% UX Score 75%
43 Product Selector Still at the will of consumer knowledge. Improvement over No/Building/Good selector. In the Product Selector Modal... 84% prefer 'Average' (over ‘Typical’) What Product Selector Does Well… 39% said “Average + ‘9/10’ normalizes the experience” Device Selection Ease of Adding Device: Desktop 6.5, Mobile 6.1 Satisfaction. w/ Device Presentation: Desktop 5.7, Mobile 5.5 Cost Awareness How clear is payment 1? Avg: 5.8 How clear is payment 2? Avg: 6.2 Experience Scorecard - Project Product: Page communicates value proposition but is a bit top heavy. Cart Concept: The productX performs well with minimum usability issues. Client Call: Agreeable at first; comfort reduces when paired with Product. Page Ratings (x/7) 85 Non Customer prospects: 60 Desktop | 25 Mobile What’s Next Suggestion: The main difference between product X and Y are abc. Make those two points bold or bring them out more by shortening sentences into smaller fragments. → A/B Test results indicate design A recommended over design B as more value measures are favorable with design A. Pull xxx from design B as a secondary recommendation. 45 % 20 % 20 % 15 % Completely OK Warranted Makes me uncomfortable Wouldn’t do this at all How do you feel about being asked for a XYZ? 5.7 5.5 5.1 4.9 Info Satisfaction Clarity of info Trust Likelihood to Click “Start Shopping” 1 2 3 4 5 6 7 UX Score 60%
1.  Create your eXperience score 2.  Determine the story you want to tell with the score to grow experience practice for your organization 3.  Create your executive dashboards and product scorecards to ensure the experience score is front and center and it supports the story you are driving 4.  Analyze the relationship between your eXperience score and other business KPIs. Universal Measures
Connecting your eXperience Score to Business KPIs Financial Indicators System Produced indicators Experience indicators Return On Investment Transactions Net Profit/LossCash Flow Capital Reserves / Debt Site Traffic Bounce Rate Click Analytics CTA Engagement CSat NPS Task completion Time on Task Satisfaction Ease of Use SUS SUPRQ SUM Journey Metrics
Connecting your eXperience Score to Business KPIs System Produced indicators Experience indicators NPS eXperience Score “R = 0.6” “eXperience Score of 70% will increase NPS by 1 point”
Connecting your eXperience Score to Business KPIs System Produced indicators Experience indicators eXperience Score “R = 0.8” “eXperience Score of 74% will increase CTA Engagement by 5 points” CTA Engagement
Connecting your eXperience Score to Business KPIs Financial Indicators Experience indicators Return On Investment Task completion Time on Task Satisfaction Ease of Use SUS SUPRQ SUM Journey Metrics “R = 0.65” “eXperience Score of 82% will increase ROI by x amount”
Universal Measurement Investment
•  Executive leadership •  Small amount of time at meetings/forums where executives reports on product investment/impact are evaluated •  Long term – some minor infrastructure investments to make the measures more easily available •  Product Managers •  Incorporate the universal measures metric into your current product success tracking •  Bring UX into the PDLC – process modifications •  UX Strategist / Leaders •  Champion the effort – education, demonstrate, negotiate Investment to Create your Universal Measures
•  UX Researchers •  Consistently bring the experience score measures into your practice •  Establish task libraries for quick access •  Collect and analyze the experience scores to serve up to the forums where the experience score is shared •  UX Designers •  Collect and analyze the experience scores to serve up to the forums where the experience score is shared •  Incorporate experience score language when evaluating designs •  Development Teams •  Digital Analytics Teams Investment to Create your Universal Measures
•  Development Teams •  Establish checkpoints for experience score investment •  Long term - Minor modifications in ticket tracking system •  UserZoom tracking code into code base •  Digital Analytics Teams •  URL alignment •  Segment alignment •  Sharing of analytics data for correlation studies Investment to Create your Universal Measures
Business executives •  1 experience measure to track for all lines of business •  Establish experience score goals for all lines of business •  1 experience measure that is related to your ROI measures •  Quickly see impact of UX investment with the executive dashboard Product managers •  1 experience measure to track with each major product iteration •  Establish experience score goals for your product •  Helps determine product experience investment decisions with a consistent view of investment and value •  1 experience measure that is related to the success measures of the product What do Universal Measures mean for…
UX Strategist / Leaders •  Provides a consistent way to measure UX across lines of business •  1 experience measure to track for all lines of business •  Establish experience score goals for all lines of business •  1 experience measure that is related to your ROI measures •  Quickly show impact of UX investment and tell your UX maturity story UX Research and Designer •  Provides a consistent way to show research results across lines of business •  1 experience measure to track for all lines of business •  Helps determine investment decisions based on research findings •  Support UX debt tracking technique to increase adoption of research What do Universal Measures mean for…
Development Teams •  Quickly determine what UX research is a must-act-upon find •  1 experience measure to track and try to achieve •  Consistent way to approach UX •  Supports efforts to establish a consistent way to incorporate UX into products Digital Analytics Teams •  1 experience measure to track for all lines of business •  Establish experience score goals for all lines of business •  Correlate your experience measure to your analytics measures •  Identify experience measure as a leading indicator for analytics measures •  Experience cost savings prior to large scale A/B testing What do Universal Measures mean for…
eXperience Score •  Reduces the fuzziness out of the abstract concept of experience •  Defines what experience means for your product(s) and service(s) •  Provides 1 score for experience that your organization can track The Story •  Strategy to grow experience practice within your organization Executive Dashboard & Product Scorecard(s) •  Makes experience research actionable Universal Measures Summary
Q&A Session Type your questions into the chat panel
THANK YOU FOR JOINING TODAY’S WEBINAR! Discover our upcoming webinars: userzoom.com/webinar Connect with Dr. Andrea Peer: https://www.linkedin.com/in/andreapeer/ marketing@userzoom.com US HQ: 866-599-1550

More Related Content

Make User Experience Part of The KPI Conversation With Universal Measures

  • 1. MAKE USER EXPERIENCE PART OF THE KPI CONVERSATION WITH UNIVERSAL MEASURES Dr. Andrea Peer | UserZoom
  • 2. Agenda Where Did KPIs Come From? WEBINAR SPEAKER Dr. Andrea Peer Onboarding Customer Success Manager at UserZoom Contact Info Connecting Experience to ROI Consider an eXperience Score as a KPI Universal Measures to grow experience practice Q&A 1 2 3 4 5
  • 3. Quick Housekeeping • Control panel on the side of your screen if you have any comments during the presentation • Time at the end for Q&A • Today’s webinar will be recorded for future viewing • All attendees will receive a copy of the slides/recording • Continue the discussion using #uzwebinar Let’s make sure you’re all set up for the webinar!
  • 4. Where did KPIs come from?
  • 5. Where did KPIs come from? INDICATORS 20th Century - Fredrick Taylor gave us Scientific Management •  Operational performance •  Efficiency and effectiveness 1920 – DuPont Analysis, ROI •  Financial Indicators 1930’s Tableau de Board…..Dashboard 1992 - Balanced Score Card •  Strategic Performance
  • 6. Cue the world of dashboards and information vomit
  • 7. So Many KPIs Financial Indicators System Produced indicators Experience indicators Return On Investment Transactions Net Profit/LossCash Flow Capital Reserves / Debt Site Traffic Bounce Rate Click Analytics CTA Engagement CSat NPS Task completion Time on Task Satisfaction Ease of Use SUS SUPRQ SUM Journey Metrics
  • 8. One Score Obsession Snapstreaks quantify a friendship in Snapchat Financial Score communicates your financial health The number you need to increase customer loyalty Helps to focus and ground Simplifies abstract concepts
  • 12. The KPI Conversation Financial Indicators System Produced indicators Experience indicators Return On Investment Transactions Net Profit/LossCash Flow Capital Reserves / Debt Site Traffic Bounce Rate Click Analytics CTA Engagement CSat NPS Task completion Time on Task Satisfaction Ease of Use SUS SUPRQ SUM Journey Metrics
  • 13. Tying UX to ROI 1 CHAPTER Justifying Cost-Justifying Usability Randolph G. Bias School of Information, The University of Texas at Austin Clare-Marie Karat IBM TJ Watson Research Center 1.1 INTRODUCTION Sir Walter Scott wrote, in Marmion, "Oh! what a tangled web we weave When first we practise to deceive!" (Scott, 1805) Sometimes it is quoted as "... what a wicked Web we weave." This year is the bicentennial anniversary of the publication of Scott's book. Did he anticipate the World Wide Web, 200 years ago? Scott's words are relevant to today's networked world and the development activities that support it, includ- ing designing for ease of use. Usability is not the end-all-be-all; rather, it must be considered alongside (and in equal measure with!) functionality and schedule. The problem is not that Web site developers are wicked; rather, they are too often in a hurry and are not oper- ating in a reward structure that motivates attention to usability. So the Web is not wicked. But tangled?. Oh yeah. Most of us are wowed by the Internet. It is amazing what we can accomplish while sitting at home in our pajamas. But when we say "us" or "we," we are refer- ring to that small subset of the population who have some expectation of what will happen if we "right click," or those of us who have heard of, for example, Doug Engelbart. When one of "us" sits down with one of "them" (the great unwashed masses), the madcap hilarity begins. "They" are so stupid! Can you believe it?, some of them don't even know that an underline, on a Web page, means the text is a link to another page (except when it isn't). "They" don't even v1, Feb 2005. Includes material adapted from the chapter: Cost benefits framework and case studies, in Bias, R.G. & Mayhew, D.J. (2005) Cost-Justifying Usability: An Update for the Internet Age. 1 Cost benefits evidence and case studies Nigel Bevan Serco Usability Services nbevan@usability.serco.com Contents 1 – Potential benefits of usability 2 1.1 Development savings 4 1.2 Increase e-commerce sales 5 1.3 Product sales 7 1.4 Usage benefits for employers 8 1.5 Reduce support and maintenance costs 10 2 – Estimating costs 11 2.1 User centered design methods 11 2.2 Costs of user centered design 13 3 – Making the cost-benefit case 14 4 – Case studies 16 4.1 Israel Aircraft Industries 16 4.2 Inland Revenue/EDS 23 4.3 Comparison 26 4.4 Taking up the methods 27 Acknowledgements 28 References 28 This paper summarises the benefits that can be obtained from use of user centered design, and discusses how to select appropriate methods and justify their cost benefits. It includes two case studies of the cost benefits of using a usability maturity model to improve the usability capability of an organisation. Usability Return on Investment (ROI) By Jakob Nielsen, J.M. Berger, and Shuli Gilutz 3rd edition 48105 WARM SPRINGS BLVD. FREMONT, CA 94539–7498 USA WWW.NNGROUP.COM Copyright © Nielsen Norman Group; All Rights Reserved. To buy a copy, download from http://www.nngroup.com/reports/roi NN/g, 2008 Bevan, 2005 Bias, 2005
  • 14. Tying UX to ROI Scribecreatedbywww.PopulationDesign.com © 2010 Human Factors International, Inc. All rights reserved. www.HumanFactors.com | 800.242.4480 USA & Canada | +1.641.472.4480 International
  • 15. Tying UX to ROI Is ROI an Effective Approach for Persuading Decision-Makers of the Value of User-Centered Design? Susan Dray, Moderator Dray & Associates, Inc. Minneapolis, MN USA dray@acm.org Clare-Marie Karat IBM T. J. Watson Research Center Hawthorne, NY USA ckarat@us.ibm.com Daniel Rosenberg Oracle Redwood Shores, CA USA daniel.Rosenberg @oracle.com David Siegel Dray & Associates, Inc. Minneapolis, MN USA david.siegel@ acm.org Dennis Wixon Microsoft Redmond, WA USA denniswi@ microsoft.com ABSTRACT This panel examines the utility and effectiveness of various ways of making the business case for user-centered design (UCD). Most of the discussion in our field has assumed that measuring and demonstrating ROI for usability is the key to this effort. However, experience shows that the most brilliant ROI analysis may not win the day in the real world of business. Our panelists range from people who claim that ROI is an important persuasive tool as long as the communication about ROI is happening within a healthy business relationship, to people who claim that a focus on ROI can actually be destructive. We also explore the idea that there are important business contexts where ROI simply does not fit. Through the presentations by the panelists and through discussion of a business case scenario, we explore some alternatives to ROI in making the business case for user-centered design. Author Keywords Return on Investment, ROI, cost-justification, UCD ACM Classification Keywords H5.3. Group and Organization Interfaces (eval/methods) INTRODUCTION Over the years, there has been much discussion in the field of usability about how to objectively and convincingly demonstrate the benefits that usability produces. This is not simply an academic exercise, but rather is part of our profession’s effort to prove its value and thus promote adoption of UCD methods. Most of the discussion in the field has focused on measuring and demonstrating return on investment (ROI) for usability, on the assumption that a well-constructed ROI analysis is an effective persuasive tool. Although we certainly hope that accuracy has something to do with effectiveness of an ROI analysis, it is also possible that the most brilliant ROI analysis will not win support for UCD in the real world of business. If we are to understand the strengths and weakness of the ROI paradigm as an approach to persuasion, we must examine questions such as: • What are the inherent strengths and weaknesses of ROI arguments? • What makes for an effective ROI argument, beyond the quality of the calculations? • What types of factors, besides ROI, make compelling cases for UCD? • Can we identify situations where ROI arguments, or other approaches are inherently more or less compelling? • What are the most challenging UCD benefits to create a compelling argument for? How can ROI and other approaches address them? The following sections give an overview of contrasting perspectives on answers to questions like these. HUMAN DYNAMICS, GOALS, AND BUSINESS CONTEXT Beyond the quality of the data on which the ROI calculation is based, a number of factors relating to the dynamics of the business are critical in determining the effectiveness of ROI arguments. First and foremost are the people involved. The person who hears the argument must have a strategic business vision or at least be a pragmatist who will experiment with new methods. If the usability professional is able to make a credible presentation of the ROI argument for usability and build trust with the visionary person, then the stage is set for the next critical factor. A second factor is how well the ROI arguments tie to the business goals and strategies of the organization. If the usability goals can help to achieve organizational goals or even take them a step further, this approach can demonstrate to the organization a new type of partnership they find very valuable. A final critical factor involves world events that are affecting the business world. One is the global marketplace and the movement of jobs and work to areas of the world where people can work at lower cost. Other significant events are the impending retirement of the "Baby Boom" generation with a resulting skills shortage, and increasing life expectancy rates in many parts of the world. If an ROI argument can be heard by visionaries, and will help them to achieve business goals or make strategies successful, and if Copyright is held by the author/owner(s). CHI 2005, April 2–7, 2004, Portland, Oregon, USA. ACM 1-59593-002-7/05/0004. CHI 2005 | Panels April 2-7 | Portland, Oregon, USA 1168 Dray, 2005
  • 16. Tying UX to ROI Accomplishment Joy Freedom Delight Connection
  • 19. Consider an eXperience Score as a Key Performance Indicator
  • 20. What is your eXperience Score? Your experience score is the indicator that your product and service is having the desired experience outcome.
  • 21. EXPERIENCE SCORES •  Reduce the fuzziness of the abstract concept of experience •  Define what experience means for your product(s) and service(s) •  Provide 1 score for experience that your organization can track •  Make experience research actionable eXperience Score As A KPI
  • 22. But wait…Experience CANNOT be accurately conveyed by 1 score! eXperience Score As A KPI
  • 24. eXperience Score is part of a bigger journey to grow your experience practice. Cue…Universal Measures
  • 25. 1.  Create your eXperience score 2.  Determine the story you want to tell with the score to grow experience practice for your organization 3.  Create your executive dashboards and product scorecards to ensure the experience score is front and center and it supports the story you are driving 4.  Analyze the relationship between your eXperience score and other business KPIs. Universal Measures
  • 26. 1.  Create your eXperience score 2.  Determine the story you want to tell with the score to grow experience practice for your organization 3.  Create your executive dashboards and product scorecards to ensure the experience score is front and center and it supports the story you are driving 4.  Analyze the relationship between your eXperience score and other business KPIs. Universal Measures
  • 27. How does your organization define & measure the ideal experience with your product(s) or service(s)?
  • 28. What is your eXperience Score? Your experience score is the indicator that your product and service is having the desired experience outcome.
  • 29. Define Your Experience Goal What is the intended experience with your product and service? What can people do? See? Know? What and how can people achieve? How can people be? Feel? Understand?
  • 30. Create your experience score Pick at least 1 measure in each cell
  • 31. Create your experience score Pick at least 1 measure in each cell Task Completion/ Success Ease of Use SUSJourney Score
  • 32. What is a eXperience Score? Final eXperience Score - 61% SUS 40% Journey 85% Avg Task Completion 60% Avg Ease of Use 60% System Scores Task Scores EXAMPLE
  • 33. What is a eXperience Score? System Scores Final eXperience Score - 63% Joy Factor 1 45% Joy Factor 2 80% Joy Factor 3 65%EXAMPLE Avg Task Completion 60% Task Scores
  • 34. What is a eXperience Score? System Scores Final eXperience Score - 53% SUPRQ 40% Avg Task Completion 60% Confidence 60% Task Scores EXAMPLE
  • 35. Immediate Considerations •  Can I weight variables in my algorithm that are more important? •  How many variables are too many? •  What if a variable isn’t relevant to my research method? Tactical Questions •  Do I have to change my scales? •  How do you do the calculation? eXperience Score Questions
  • 36. Long Term Considerations •  Can I change my variables over time? •  Do I report one score across all of my products or per product? •  Do all my products have the same variables? •  What is a good growth goal for my eXperience Score? eXperience Score Questions
  • 37. 1.  Create your eXperience score 2.  Determine the story you want to tell with the score to grow experience practice for your organization 3.  Create your executive dashboards and product scorecards to ensure the experience score is front and center and it supports the story you are driving 4.  Analyze the relationship between your eXperience score and other business KPIs. Universal Measures
  • 38. Common Stories for Executive Dashboards - The black box of dev story - The pre prod investment story - The post prod baseline and gap story - The portfolio opportunity story - The full spectrum story - The user data powerhouse story - The experience investment vs value story What story do you want to tell?
  • 39. 1.  Create your eXperience score 2.  Determine the story you want to tell with the score to grow experience practice for your organization 3.  Create your executive dashboards and product scorecards to ensure the experience score is front and center and it supports the story you are driving 4.  Analyze the relationship between your eXperience score and other business KPIs. Universal Measures
  • 40. Creating your Executive Dashboard & Product Scorecards 2 Types of Dashboards Executive UX Dashboard Product UX Scorecard Purpose: Show impact of UX on org and highlight UX win/work Target Audience: Senior Leadership Frequency: Monthly or quarterly Purpose: Show result of UX effort for a specific product line Target Audience: Development team leaders/PMs/POs/etc. Frequency: Bi-Monthly, Monthly
  • 41. Executive Dashboard •  Let’s deep dive on an executive dashboard that works for your organization. Let’s craft the story you want to tell to your executives. •  What connection can we make to your analytics dataset? •  What connection is important to make to your key business measures? •  What other reports in the business do we need to connect to? •  Determine the process/channel to serve the dashboards to your internal stakeholders •  Determine frequency Product Scorecard •  Let’s deep dive on a product scorecard that works for your organization. Let’s craft the story you need to tell for development and product teams. •  Evaluate PDLC process and systems you can integrate with (ie: Jira, Rally) •  Determine the process/channel to serve the dashboards to your internal stakeholders •  Determine frequency based on PDLC integration strategy Creating your Executive Dashboard & Product Scorecards
  • 42. 42 Experience Scorecard - Quarterly Report (Q1 2016) ROI Impact 20 product lines, 5 personas impacted Prior to Development 53 Severe impact discoveries found and 60% of the discoveries addressed 20 Moderate impact discoveries found and added to the UX backlog Post Development (Beta) 3 Severe impact discoveries found and addressed 2 Moderate impact discoveries found and added to the UX backlog Post Development (Full Prod) 5 Severe impact discoveries found and added to backlog for next sprint 7 Moderate impact discoveries found and added to the UX backlog UX Performance Cadence: 1 research engagement per design sprint Avg Field Time: 5 hours Avg Participants per Study: 32 Persona Representation: 10% Bob, 50% Julie, 40% Chip Research Conversation: 40% Avg SUS +15% Avg Sat +10% UX Score 75%
  • 43. 43 Product Selector Still at the will of consumer knowledge. Improvement over No/Building/Good selector. In the Product Selector Modal... 84% prefer 'Average' (over ‘Typical’) What Product Selector Does Well… 39% said “Average + ‘9/10’ normalizes the experience” Device Selection Ease of Adding Device: Desktop 6.5, Mobile 6.1 Satisfaction. w/ Device Presentation: Desktop 5.7, Mobile 5.5 Cost Awareness How clear is payment 1? Avg: 5.8 How clear is payment 2? Avg: 6.2 Experience Scorecard - Project Product: Page communicates value proposition but is a bit top heavy. Cart Concept: The productX performs well with minimum usability issues. Client Call: Agreeable at first; comfort reduces when paired with Product. Page Ratings (x/7) 85 Non Customer prospects: 60 Desktop | 25 Mobile What’s Next Suggestion: The main difference between product X and Y are abc. Make those two points bold or bring them out more by shortening sentences into smaller fragments. → A/B Test results indicate design A recommended over design B as more value measures are favorable with design A. Pull xxx from design B as a secondary recommendation. 45 % 20 % 20 % 15 % Completely OK Warranted Makes me uncomfortable Wouldn’t do this at all How do you feel about being asked for a XYZ? 5.7 5.5 5.1 4.9 Info Satisfaction Clarity of info Trust Likelihood to Click “Start Shopping” 1 2 3 4 5 6 7 UX Score 60%
  • 44. 1.  Create your eXperience score 2.  Determine the story you want to tell with the score to grow experience practice for your organization 3.  Create your executive dashboards and product scorecards to ensure the experience score is front and center and it supports the story you are driving 4.  Analyze the relationship between your eXperience score and other business KPIs. Universal Measures
  • 45. Connecting your eXperience Score to Business KPIs Financial Indicators System Produced indicators Experience indicators Return On Investment Transactions Net Profit/LossCash Flow Capital Reserves / Debt Site Traffic Bounce Rate Click Analytics CTA Engagement CSat NPS Task completion Time on Task Satisfaction Ease of Use SUS SUPRQ SUM Journey Metrics
  • 46. Connecting your eXperience Score to Business KPIs System Produced indicators Experience indicators NPS eXperience Score “R = 0.6” “eXperience Score of 70% will increase NPS by 1 point”
  • 47. Connecting your eXperience Score to Business KPIs System Produced indicators Experience indicators eXperience Score “R = 0.8” “eXperience Score of 74% will increase CTA Engagement by 5 points” CTA Engagement
  • 48. Connecting your eXperience Score to Business KPIs Financial Indicators Experience indicators Return On Investment Task completion Time on Task Satisfaction Ease of Use SUS SUPRQ SUM Journey Metrics “R = 0.65” “eXperience Score of 82% will increase ROI by x amount”
  • 50. •  Executive leadership •  Small amount of time at meetings/forums where executives reports on product investment/impact are evaluated •  Long term – some minor infrastructure investments to make the measures more easily available •  Product Managers •  Incorporate the universal measures metric into your current product success tracking •  Bring UX into the PDLC – process modifications •  UX Strategist / Leaders •  Champion the effort – education, demonstrate, negotiate Investment to Create your Universal Measures
  • 51. •  UX Researchers •  Consistently bring the experience score measures into your practice •  Establish task libraries for quick access •  Collect and analyze the experience scores to serve up to the forums where the experience score is shared •  UX Designers •  Collect and analyze the experience scores to serve up to the forums where the experience score is shared •  Incorporate experience score language when evaluating designs •  Development Teams •  Digital Analytics Teams Investment to Create your Universal Measures
  • 52. •  Development Teams •  Establish checkpoints for experience score investment •  Long term - Minor modifications in ticket tracking system •  UserZoom tracking code into code base •  Digital Analytics Teams •  URL alignment •  Segment alignment •  Sharing of analytics data for correlation studies Investment to Create your Universal Measures
  • 53. Business executives •  1 experience measure to track for all lines of business •  Establish experience score goals for all lines of business •  1 experience measure that is related to your ROI measures •  Quickly see impact of UX investment with the executive dashboard Product managers •  1 experience measure to track with each major product iteration •  Establish experience score goals for your product •  Helps determine product experience investment decisions with a consistent view of investment and value •  1 experience measure that is related to the success measures of the product What do Universal Measures mean for…
  • 54. UX Strategist / Leaders •  Provides a consistent way to measure UX across lines of business •  1 experience measure to track for all lines of business •  Establish experience score goals for all lines of business •  1 experience measure that is related to your ROI measures •  Quickly show impact of UX investment and tell your UX maturity story UX Research and Designer •  Provides a consistent way to show research results across lines of business •  1 experience measure to track for all lines of business •  Helps determine investment decisions based on research findings •  Support UX debt tracking technique to increase adoption of research What do Universal Measures mean for…
  • 55. Development Teams •  Quickly determine what UX research is a must-act-upon find •  1 experience measure to track and try to achieve •  Consistent way to approach UX •  Supports efforts to establish a consistent way to incorporate UX into products Digital Analytics Teams •  1 experience measure to track for all lines of business •  Establish experience score goals for all lines of business •  Correlate your experience measure to your analytics measures •  Identify experience measure as a leading indicator for analytics measures •  Experience cost savings prior to large scale A/B testing What do Universal Measures mean for…
  • 56. eXperience Score •  Reduces the fuzziness out of the abstract concept of experience •  Defines what experience means for your product(s) and service(s) •  Provides 1 score for experience that your organization can track The Story •  Strategy to grow experience practice within your organization Executive Dashboard & Product Scorecard(s) •  Makes experience research actionable Universal Measures Summary
  • 57. Q&A Session Type your questions into the chat panel
  • 58. THANK YOU FOR JOINING TODAY’S WEBINAR! Discover our upcoming webinars: userzoom.com/webinar Connect with Dr. Andrea Peer: https://www.linkedin.com/in/andreapeer/ marketing@userzoom.com US HQ: 866-599-1550