SlideShare a Scribd company logo

Analytics - How and why you are making a mess of it and ruining your analysts’ lives

•Download as PPTX, PDF•
•
deltaDNA
deltaDNA

Presented by Dr Dan Bergin-Holly at the 18th Games Industry Analytics Forum in London on 10th May 2018.

1 of 24
Analytics - How and why you are making a mess of it and ruining your analysts’ lives
Analytics - How and why youare making a mess of it and ruining your analysts’ lives
Who the F**k areyou? I’m Dr. Dan! ● Product Lead on Angry Birds Transformers for Exient Ltd ● Background inPerformance Technology and live theatre ● Previously Product Manager at SEGA,Producer atSixminute ● PhD in something vaguely interesting ● Married - so you know at least one person likes me
People Process Tools With Data from RealAnalysts!
People
People You Hired the Wrong People! You hired a data scientist when what you really wanted was a monetisation designer or game economy expert You hired a data scientist who ended up being a monetisation designer because your current designers are lazy or scared of math (you hired the wrong designers) You hired a data scientist with a big shiny degree in machine learning and big data operations so that they could export SQLqueries into excel and make the occasional chart - without telling them (possibly because you didn’t know) You hired someone right outta university - and didn’t take the time to guide them through the difficult transition towards everything being broken and horrible here in the real world.
People You keep your analysts in a separate building - away from the design and programming team! Your designers begin to resent all the suggestions coming from ‘the numbers people’ who have no idea what it’s like to be in the trenches, at the coalface, really working on a game. Your analysts wonder why your designers keep iterating on the different boosters for the archery character when none of the players actually own an archer in game Cross-functional teams are da bomb! Put your analysts where the dev team is - after all, they’re all working on the game right? - This includes not fobbing off all your data work to your publisher!
People You are blind to theanalyst’s suggestions! A lot of data is telling you that the FTUE funnel is broken and that the game doesn’t run on lower spec android phones Your analyst has repeatedly shown you that the biggest win would be to make the game performant on a certain subset of devices Your 6 year old niece who plays the game keeps saying the bunny rabbits should beblue You prioritise colour iterations on rabbits for the next 3 sprints
Tools
Tools You build a shonky homebrew analytics system that never works Your programmers will hate you and your analyst Your analyst will hate you and your programmers Your investors will hate you and your bad data Your bank manager will be indifferent to your suffering
Tools When is it OK to build your own Analytics Solution? You are Daddy Warbucks. You can dedicate significant AND CONTINUOUS resources to the development and maintenance of a high quality piece of business critical software.
Tools When is it OK to build your own Analytics Solution? You have decided to quit your job and are now making an analytics company. You can dedicate significant AND CONTINUOUS resources to the development and maintenance of a high quality piece of business critical software.
Tools Right at the end of development (or even after launch) you suddenly thought ‘oh no we better get some data’ and rush to get some analytics into the game. It doesn’t get goodtesting. Your analyst doesn’t get time to verify that the data will be sufficient to answer your key initial questions. You can’t answer any key questions - and lose precious time. Remember! Telemetry failures can be incredibly difficult to detect and fix, and often only rear their head once the game is ‘in the wild’ - reduce risk by implementing early! You put your analytics inlate
Tools That’s right. For one reason or another you made a mess of it. The data is clearly not firing correctly, and pretty much every day that goes by not only are you adding useless data to a database, you may even be damaging your existing data set and making it worthless. One day, your analyst presents you with a data table that includes an entry being populated by the words ‘TO DO’ - and a bill for all the additional server space You put your analytics inwrong
Tools After two months of other fixes you finally get around to fixing your data. On your next release, something drastically changes in the game - you struggle to find out how or why, because all your baseline data from the last two months is complete trash. You took too long to fix your analytics
Process
Process You ignore your analyst and product team’s pleas for some measure of sanity and prioritization in favour of making an apple watch/appleTV/ARKit/Google Chromecast version of your game because you know a guy, who knows a guy, who said they totally got store featuring when they did it. Seriously though - this annoys everyone in your team. Don’t do it.
Process This is a direct result of some of the previous issues I’ve mentioned (or your team experiencing those issues in other studios and bringing that baggage with them) This can lead to glaring problems in your game going unfixed for far too long - for example, nobody collecting their rewards in your game, and thus rendering your entire progression design totally redundant. Where does this comefrom? ● Unreliable tools ● Rarely checking results of changes ● Management’s attitude Nobody Trusts Your Data
Process You still need to make your best educated guess in absence of any data to back you up. The fact that you have data is not an excuse for habitual fence sitting. You can’t fob everything off to another A/B test. Remember! At the end of the day someone needs to be the shot-caller on a project - and that means actually calling the shots. You stopped taking responsibility for making tough calls because data exists
Process You blame analysts for thedata! An A/B test showed promise so you rolled out a big change to the game and it's all gone wrong. When discussing it, you keep coming back to ‘But the A/B Test said…..’ Remember! A/B Tests aren’t always the be all and end all of a feature. Funnel progression can’t tell you everything about your tutorial. Remember when using telemetry data about your game, it's only one piece of the puzzle. It’s information that can (and should) be supplemented.
Process You ask big questions then ignore the answers You have an idea for afeature. You ask the analysts to answer “some vital question”™ to prove you’re right. After badgering your analytics team into spending hours on “some vital question”™ they deliver data that strongly suggests you are absolutelywrong. You promptly ignore the answers they gave you and do what you were going to do anyway. Your analysts feel undervalued - and begin talking to recruiters….
Summary
Summary Know what you need from a potential hire. Be sound. Don’t roll your own. Test early andoften. Listen to the data - but don’t treat it as an oracle.
Thanks! Questions? Comments? Rotten Fruit? Mail: dan@danyoutohell.com Tweet: @danyoutohell Address by name tomy face: Dan

More Related Content

Analytics - How and why you are making a mess of it and ruining your analysts’ lives

  • 2. Analytics - How and why youare making a mess of it and ruining your analysts’ lives
  • 3. Who the F**k areyou? I’m Dr. Dan! ● Product Lead on Angry Birds Transformers for Exient Ltd ● Background inPerformance Technology and live theatre ● Previously Product Manager at SEGA,Producer atSixminute ● PhD in something vaguely interesting ● Married - so you know at least one person likes me
  • 4. People Process Tools With Data from RealAnalysts!
  • 6. People You Hired the Wrong People! You hired a data scientist when what you really wanted was a monetisation designer or game economy expert You hired a data scientist who ended up being a monetisation designer because your current designers are lazy or scared of math (you hired the wrong designers) You hired a data scientist with a big shiny degree in machine learning and big data operations so that they could export SQLqueries into excel and make the occasional chart - without telling them (possibly because you didn’t know) You hired someone right outta university - and didn’t take the time to guide them through the difficult transition towards everything being broken and horrible here in the real world.
  • 7. People You keep your analysts in a separate building - away from the design and programming team! Your designers begin to resent all the suggestions coming from ‘the numbers people’ who have no idea what it’s like to be in the trenches, at the coalface, really working on a game. Your analysts wonder why your designers keep iterating on the different boosters for the archery character when none of the players actually own an archer in game Cross-functional teams are da bomb! Put your analysts where the dev team is - after all, they’re all working on the game right? - This includes not fobbing off all your data work to your publisher!
  • 8. People You are blind to theanalyst’s suggestions! A lot of data is telling you that the FTUE funnel is broken and that the game doesn’t run on lower spec android phones Your analyst has repeatedly shown you that the biggest win would be to make the game performant on a certain subset of devices Your 6 year old niece who plays the game keeps saying the bunny rabbits should beblue You prioritise colour iterations on rabbits for the next 3 sprints
  • 10. Tools You build a shonky homebrew analytics system that never works Your programmers will hate you and your analyst Your analyst will hate you and your programmers Your investors will hate you and your bad data Your bank manager will be indifferent to your suffering
  • 11. Tools When is it OK to build your own Analytics Solution? You are Daddy Warbucks. You can dedicate significant AND CONTINUOUS resources to the development and maintenance of a high quality piece of business critical software.
  • 12. Tools When is it OK to build your own Analytics Solution? You have decided to quit your job and are now making an analytics company. You can dedicate significant AND CONTINUOUS resources to the development and maintenance of a high quality piece of business critical software.
  • 13. Tools Right at the end of development (or even after launch) you suddenly thought ‘oh no we better get some data’ and rush to get some analytics into the game. It doesn’t get goodtesting. Your analyst doesn’t get time to verify that the data will be sufficient to answer your key initial questions. You can’t answer any key questions - and lose precious time. Remember! Telemetry failures can be incredibly difficult to detect and fix, and often only rear their head once the game is ‘in the wild’ - reduce risk by implementing early! You put your analytics inlate
  • 14. Tools That’s right. For one reason or another you made a mess of it. The data is clearly not firing correctly, and pretty much every day that goes by not only are you adding useless data to a database, you may even be damaging your existing data set and making it worthless. One day, your analyst presents you with a data table that includes an entry being populated by the words ‘TO DO’ - and a bill for all the additional server space You put your analytics inwrong
  • 15. Tools After two months of other fixes you finally get around to fixing your data. On your next release, something drastically changes in the game - you struggle to find out how or why, because all your baseline data from the last two months is complete trash. You took too long to fix your analytics
  • 17. Process You ignore your analyst and product team’s pleas for some measure of sanity and prioritization in favour of making an apple watch/appleTV/ARKit/Google Chromecast version of your game because you know a guy, who knows a guy, who said they totally got store featuring when they did it. Seriously though - this annoys everyone in your team. Don’t do it.
  • 18. Process This is a direct result of some of the previous issues I’ve mentioned (or your team experiencing those issues in other studios and bringing that baggage with them) This can lead to glaring problems in your game going unfixed for far too long - for example, nobody collecting their rewards in your game, and thus rendering your entire progression design totally redundant. Where does this comefrom? ● Unreliable tools ● Rarely checking results of changes ● Management’s attitude Nobody Trusts Your Data
  • 19. Process You still need to make your best educated guess in absence of any data to back you up. The fact that you have data is not an excuse for habitual fence sitting. You can’t fob everything off to another A/B test. Remember! At the end of the day someone needs to be the shot-caller on a project - and that means actually calling the shots. You stopped taking responsibility for making tough calls because data exists
  • 20. Process You blame analysts for thedata! An A/B test showed promise so you rolled out a big change to the game and it's all gone wrong. When discussing it, you keep coming back to ‘But the A/B Test said…..’ Remember! A/B Tests aren’t always the be all and end all of a feature. Funnel progression can’t tell you everything about your tutorial. Remember when using telemetry data about your game, it's only one piece of the puzzle. It’s information that can (and should) be supplemented.
  • 21. Process You ask big questions then ignore the answers You have an idea for afeature. You ask the analysts to answer “some vital question”™ to prove you’re right. After badgering your analytics team into spending hours on “some vital question”™ they deliver data that strongly suggests you are absolutelywrong. You promptly ignore the answers they gave you and do what you were going to do anyway. Your analysts feel undervalued - and begin talking to recruiters….
  • 23. Summary Know what you need from a potential hire. Be sound. Don’t roll your own. Test early andoften. Listen to the data - but don’t treat it as an oracle.
  • 24. Thanks! Questions? Comments? Rotten Fruit? Mail: dan@danyoutohell.com Tweet: @danyoutohell Address by name tomy face: Dan