Growing Up The Right Way: An Example of How to Scale When You're Aren't a Startup Anymore by Derek Zell
•Download as PPTX, PDF•
1 like•65 views
The current environment we live in demands the ability to scale at the drop of a hat. In this SauceCon 2018 session, we will cover the questions to ask when you are faced with the scaling vs. cost dilemma, things like “I have a limited budget, where should I spend it?”, “How do I take advantage of my team’s diverse, yet non-correlating, coding experience?”, and “Where should I start to make the biggest bang for the buck? We’ll cover the options and tools you have to make sure you’re scaling without forsaking quality, and why it is ok to fail (as long as you do it fast). I will share real world examples of our company being forced to grow up and the pains and glorious successes that went right along with it. Plus there will be treats and who doesn’t like treats?
Report
Share
Related slideshows
Growing Up The Right Way: An Example of How to Scale When You're Aren't a Startup Anymore by Derek Zell
- 1. Derek Zell Engineering Manager – QA, R&E, and DevOps Criteo Growing Up the Right Way: An Example of How to Scale When You Aren’t a Start-up Anymore
- 2. 2 • Who am I?
- 3. 3 • What will we cover, Derek? • Background – How we got to the problem • Approach – How we tried to fix the problem • Results - Implementing the fix(es) • Conclusion – What the heck does this mean for you?
- 4. 4 • Background – Who We Were • Started as a travel performance marketing company in 2004 • Expanded into Auto and Retail • In-Grain, Organic Ads • Online auction site that allowed advertisers and brands to level the playing field
- 5. 5 • Background – What We Did/Do
- 6. 6 • Background – Who We Were
- 7. 7 • Break for Treats
- 8. 8 • Background – Problems Arise Problems • Publishers making changes (99%) • Our creatives not working/displaying incorrectly (1%) • Sheer number of placements forced us to be reactionary Mo’ Placements, Mo’ Problems
- 9. 9 • Decision #1 – The QUESTIONS • What matters most? • Eliminating the need for manual validation of placements • Where are we growing the fastest? • Publisher placements • Where can we (QA) add the most value? • Making sure our ads show ALL THE TIME • What is the end goal? • Finding issues to make sure we keep making the money
- 10. 10 • Realization (simplified) We need test automation!
- 11. 11 • Approach • All the things considered - Test tool suites - Offshore scripting - Development teams scripting - QA Automation Team
- 12. 12 • Decision #1* QA Automation Team + Selenium WebDriver • Why a QA Automation Team? • I am selfish • Teams retain their core competencies • More manageable and cost effective • Why Selenium? • Free • Code independent • Hiring made easier • User community • Framework allowed for adding new placements quickly *DISCLAIMER – not a commercial
- 13. 13 • Background – Again? Mo’ Placements, Mo’ Problems: The Sequel Problems • 4 hours to complete cycle • Now an issue meant potentially losing $ • Now an issue meant potentially losing a relationship = $$$$$$$$$$$
- 14. 14 • Decision #2 – The QUESTIONS • What matters most? • Keeping our placements up and serving ads • Where are we growing the fastest? • Publisher placements (STILL) • Where can we add the most value? • Making sure our ads show all the time • What is the end goal? • Finding issues FASTER to keep making the money
- 15. 15 • Realization (simplified) We need test automation! We need scalable yet still cost effective test automation!
- 16. 16 • Approach • All the things considered - Creating our own server farm - Using production server capacity to run scripts - Utilizing a 3rd party
- 17. 17 • Decision #2* Sauce Labs • Why? • Scalable scripting– allowed concurrent runs/threading • Also Scalable infrastructure – more VMs with a call • No infrastructure or maintenance costs • OS/Browser combinations • Annoying, pushy (and right) account exec *DISCLAIMER – STILL not a commercial
- 19. 19 • Implementation • First 6 months • Script runs took less than 40 minutes • Response to placement downtime dropped over 75% • BLACK FRIDAY/CYBER MONDAY – found 7 issues on Publisher sites
- 20. 20 • Break for Treats
- 21. 21 • Decision - Validated
- 22. 22 • Questions to Ask Yourself • What matters most? • Where are you going to grow the fastest? • Where can you add the most value? • What do you want to be when you grow up? Then…. • What/who do I need to get that done?
- 23. 23 • Side Note – Extra Credit • Making your results work for you 1. Prove your solution works • Low hanging fruit • Gain a champion – HOTH • If you fail, fail quickly and cheaply 2. Take results back and ask for more • % of savings “Be so good they can’t ignore you” – Steve Martin
- 24. 24 • Conclusion “The same solution doesn’t work for everyone. However, by identifying what is important to your team and your company, you can find what fits for you” - Me
- 25. 25 • Thank you (and maybe you clapping?) Questions?
Editor's Notes
- Welcome to what is sure to be one of the least technical presentations here, but we wanted to run thru a case study that showed issues that everyone at one time or another could run into as companies grow
- 20+ years in the industry, I have held every imaginable position in QA, I have an undergrad in International Business and Japanese and an MBA with an emphasis on e-Commerce…and you all know there is a story behind every picture and mine is that I like beer.
- Glad you asked… I will attempt to use MY real world example to weave an amazingly interesting story that has you on the edge of your post-lunch induced coma’ed seats and along the way I hope to give you useful information that just might pertain to and help you or someone you love
- Non-intrusive, partnerships with sites that sell the products already WIN/WIN
- In grain, NON INTRUSIVE, ads that make sense to the viewer. You searched for cheese, we are going to show you cheese damnit
- From 4 people to 150 in 10 years, but growth really hit in 2014 when I joined….ok, that happened to coincide with the retail launch, but we all know the truth. Look at that handsome devil there in the front – arm raised in triumph, most likely hung over from the night before. Ah the glory days
- Whew, are you tired? I am winded. And I promised snacks. Anyone allergic to peanuts, chocolates, nuget, gluten or happiness?
- Ok, now the good stuff This crazy exponential growth was starting to cause issues. Since we were injecting ads into these sites, whenever they made a change, they forgot about little old us so our ads would break
- Animation!!! You are welcome! These questions are the “secret sauce” of this presentation What mattered most was letting QA focus on core competency of functional testing
- This is where it gets different for everyone. Some teams have money/budget, some teams have technical expertise, some teams have a large number of employees that can help delay decisions. We had none of these things.
- I learned at Simon’s presentation that you all use it now so this is going to sound really familiar to every one of you. Selenium allowed me to hire the right person, not the right person who new X coding language. It takes a certain type of engineer to work in test automation and expanding the pool of potential candidates is a life saver when you need to hire fast.
- So everything should be going smooth now, right? Not so much. People develop these crazy things called “expectations”. Once you prove your worth, you have to continue proving your worth. Exciting stuff. Now it isn’t good enough to find issues, now every hour an issue exists costs $$s or more importantly a relationship
- Secret sauce
- Again, this is where it gets different for everyone. Some teams have money/budget, some teams have technical expertise, some teams have a large number of employees that can help delay decisions.
- HOTH – Star Wars nerds in QA? What? Pretty UI interface – really necessary? We will get to that in a sec We also set up these to alert thru PagerDuty when a script failed
- Saved over $200,000 in lost revenue in Q4
- HOME STRETCH!
- November 2016 we were purchased by Criteo. I was ready to fight - And I came to find out that The “kings of scale” with 10s of thousands of placements chose the same solution we had a few years before 75 VMs – threaded scripts now take less than 10 minutes (Oct – Dec)
- In QA there is always a need to prove yourself. QA is seen as the necessary evil. Here is the reason we created HOTH so we could show off what we did and how we were helping
- Famous quote