How to train your product owner
•
6 likes•691 views
Machine Learning is transforming every industry with innovative techniques receiving deserved attention. But turning innovation into value requires integrating into practical technology products, often with the leadership of product managers. We'll talk about how to help your friendly neighborhood Product Owner: identify where ML can make a difference, develop metrics to validate and refine it, identify data to feed it, prioritize work to develop it, and structure teams to deliver it in a satisfying way.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to How to train your product owner
Similar to How to train your product owner (20)
More from David Murgatroyd
More from David Murgatroyd (12)
Recently uploaded
Recently uploaded (20)
How to train your product owner
- 1. How to Train Your Product Owner David Murgatroyd ( @dmurga) MassTLC ML Dev Day January 24, 2018 (please don’t sue me for copyright infringement, DreamWorks!)
- 3. Agile & Machine Learning David Murgatroyd (@dmurga) (ML Chapter Lead in Quest) Your Product Owner
- 4. Agile & Machine Learning David Murgatroyd (@dmurga) (ML Chapter Lead in Quest) Your Product Owner Your Product
- 5. @dmurga
- 6. @dmurga What do Product Owners Do? Lead team in: ‣ establishing vision/hypotheses for what product should become ‣ prioritizing work to get there
- 7. @dmurga What do Product Owners Do? Lead team in: ‣ establishing vision/hypotheses for what product should become ‣ prioritizing work to get there Your Product
- 9. @dmurga Outside In Machine Learning Your Product
- 10. @dmurga Outside In … Now to Then Machine Learning Machine Learning Your Product Your Future Product
- 11. @dmurga Outside In … Now to Then Machine Learning Problem Metrics Data Models Prioritizing OrganizingDesign Machine Learning Your Product Your Future Product
- 12. Train Your PO… Outside In
- 13. @dmurga Machine Learning Problem Metrics Data Models Outside In Design Your Product
- 14. @dmurga Machine Learning Problem Metrics Data Models Picking a Problem Design Your Product 1. Tweak It 2. Think It
- 15. @dmurga Problem Metrics Data Models Picking a Problem: Tweak It Design ‣ What’s the business goal of your product? ‣ What fuzzy decision does it make that impacts that goal? Your Product
- 16. @dmurga ‣ Perception: a person can do it in less than a second. ‣ Prediction: done over and over ‣ Personalization: similar need but met in different ways Problem Metrics Data Models Picking a Problem: Think It Design Your Product ?
- 17. @dmurga
- 18. @dmurga
- 19. @dmurga 1. The role of ‘why’ of ML output? 2. Coping with errors? 3. Designing for varied output? 4. New user behaviors that ML enables? Problem Metrics Data Models Designing with ML Design Your Product
- 20. @dmurga Design ‣ Trading off why vs. right ‣ Granularity of ‘why’ ‣ Creepy vs. personal ‣ First step: designer gives concrete example on concrete data Problem Metrics Data Models Design: the role of ‘why’ Your Product
- 21. @dmurga ‣ What quality is needed to user maintain trust or delight them? ‣ Rank multiple outputs? ‣ Provide fallback behavior? ‣ Explicit feedback mechanisms? Problem Metrics Data Models Design: coping with errors Design Your Product
- 22. @dmurga ‣ Get realistic output samples from a variety of users (e.g., synthetic, persona) ‣ Watch for subtle assumptions Problem Metrics Data Models Design: coping with variety Design Your Product
- 23. @dmurga ‣ Users might anthropomorphize ML Products ‣ Users might express more of themselves or test the limits of the systems ‣ Avoid “user bubbles” by encouraging discovery and crafting metrics Problem Metrics Data Models Design: new user behavior Design Your Product
- 24. @dmurga ‣ Alignment to business value ‣ Effort to measure ‣ Useful for team’s decisions ‣ Useful for model’s training Problem Metrics Data Models Metrics: Properties Design Your Product Machine Learning
- 25. @dmurga Problem Metrics Data Models Metrics: Matrix of Metrics Design Where / What Heuristic Modeled Online Online Heuristic Online Modeled Offline Offline Heuristic Offline Modeled Your Product Machine Learning
- 26. @dmurga Problem Metrics Data Models Metrics: Online vs Offline Design Online More aligned with business value Offline Generally less effort to measure Your Product Machine Learning
- 27. @dmurga Problem Metrics Data Models Metrics: Heuristic vs Modeled Design Heuristic Modeled More useful for team’s decisions interpretable More useful for training the model directly Your Product Machine Learning
- 28. @dmurga Problem Metrics Data Models Metrics: Matrix of Metrics Design Where / What Heuristic Modeled Online Online Heuristic Online Modeled Offline Offline Heuristic Offline Modeled Your Product Machine Learning
- 29. @dmurga ‣ Just look at some data before/after ‣ Alignment to business value: OK ‣ Effort to measure: OK (one-off) ‣ Useful for team’s decisions: OK ‣ Useful for model’s training: Bad Problem Metrics Data Models Metrics: … vs Subjective Design Your Product Machine Learning
- 30. @dmurga 1. What data do you need? 2. Where can you get it? 3. What biases does it carry? Problem Metrics Data Models Data Design Your Product
- 31. @dmurga ‣ Raw input data ‣ Metadata / reference ‣ Has metadata that is a source for measurement Problem Metrics Data Models What data do you need? Design Your Product
- 32. @dmurga ‣ Data is the new Wireframe ‣ Product Owner provides example inputs / outputs ‣ Use these to also vet metrics Problem Metrics Data Models Specifying with Data Design Your Product Machine Learning
- 33. @dmurga ‣ Logging ‣ Proxies and other 3rd Parties ‣ Annotation ‣ Watch out for drift in the data set ‣ Take advantage of PO’s domain knowledge! Problem Metrics Data Models Where can you get it? Design Your Product
- 34. @dmurga ‣ Is it skewed? ‣ Is it tainted? ‣ Is it likely to stereotype? Problem Metrics Data Models What biases does it carry? Design Your Product
- 35. @dmurga 1. Teach tasks rather than techniques 2. Teach trade-offs rather than tools 3. Start simple Problem Metrics Data Models Models Design Your Product Machine Learning
- 36. @dmurga ‣ Classification ‣ Clustering ‣ Regression ‣ (and Embeddings if you must :- ) Problem Metrics Data Models Models: Tasks Design Your Product Machine Learning
- 37. @dmurga ‣ Simplicity ‣ Interpretability ‣ Confidence ‣ Accuracy ‣ Adaptability ‣ Speed ‣ Space ‣ Scale Problem Metrics Data Models Models: Trade-offs Design Your Product Machine Learning
- 38. @dmurga ‣ Rules before baseline models ‣ Baselines before adapted models ‣ Adapted models before end-to-end models Problem Metrics Data Models Models: Err toward Simplicity Design Your Product Machine Learning
- 39. Train Your PO… Now to Then
- 40. @dmurga Outside In … Now to Then Machine Learning Problem Metrics Data Models Prioritizing OrganizingDesign Machine Learning Your Product Your Future Product
- 41. @dmurga Now to Then Prioritizing Organizing Machine Learning Your Future Product
- 42. @dmurga Prioritizing Prioritizing Organizing 1. Exploration 2. Experimentation 3. Error Analysis 4. ML Product Lifecycle 5. Longer term Goals (OKRs) Your Backlog
- 45. @dmurga Prioritizing: Error Analysis Prioritizing Organizing Bug Error Your Backlog
- 46. @dmurga Prioritizing: Error Analysis Prioritizing Organizing Your Backlog
- 47. @dmurga Prioritizing: Prod Lifecycle Prioritizing Organizing Your Backlog Stage Characteristics Exploration by hand examples/rules Pre-MVP (0.1% / early Beta) measurable & inspectable MVP (1%, Beta) accurate, not slow, & documented v1 (100% / GA) simple & fast Post-v1 handle new domains
- 48. @dmurga Prioritizing: Goals (OKRs) Prioritizing Organizing Your Backlog Stage OKRs Exploration amount of analysis Pre-MVP (0.1% / early Beta) having metrics, amount of experiments MVP (1%, Beta) moving core metrics, amount of experiments v1 (100% / GA) moving all metrics Post-v1 moving all metrics on new data
- 49. @dmurga Organizing Prioritizing Organizing 1. Team Structure 2. Roles Your Backlog
- 50. @dmurga Organizing: Team Structure Organizing ‣ Aligned to Product/Org Maturity ‣ Exploration: Centralized Team Your Backlog Prioritizing
- 51. @dmurga Organizing: Team Structure Organizing ‣ Aligned to Product/Org Maturity ‣ Exploration: Centralized Team ‣ Pre-MVP: Cross-functional Your Backlog Prioritizing
- 52. @dmurga Organizing: Team Structure Organizing ‣ Aligned to Product/Org Maturity ‣ Exploration: Centralized Team ‣ Pre-MVP: Cross-functional ‣ MVP to v1: Cross-functional with separate work stream Your Backlog Prioritizing
- 53. @dmurga Organizing: Team Structure Prioritizing Organizing ‣ Aligned to Product/Org Maturity ‣ Exploration: Centralized Team ‣ Pre-MVP: Cross-functional ‣ MVP to v1: Cross-functional with separate work stream ‣ Post v1: Dedicated Sibling Team Your Backlog
- 54. @dmurga ‣ Applied ML Eng ‣ ML Tool Eng ‣ Core Researcher Organizing: Roles Prioritizing Organizing
- 55. @dmurga Outside In … Now to Then Machine Learning Problem Metrics Data Models Prioritizing OrganizingDesign Machine Learning Your Product Your Future Product
- 56. Thanks! Questions? David Murgatroyd (@dmurga) Suggestions: What about different kinds of testing? What are common features ML-based products have? More on identifying metrics? Machine Learning vs. Data Science? Hiring in Boston, NYC, London, and Stockholm!