[@IndeedEng] Engineering Velocity: Building Great Software Through Fast Iteration
•
0 likes•5,422 views
Video available: http://www.youtube.com/watch?v=zCy077_dyJo&feature=youtu.be Since 2005, Indeed has created and cultivated a strong engineering culture with a focus on ownership, real-world impact, and constant incremental delivery. Our experience has demonstrated that rapid iteration is essential to discovering the most valuable functionality for our users. In the next @IndeedEng talk, Dan Heller will share some of the architectural solutions, tools, and processes Indeed has created to support constant incremental delivery of new features and enhancements. Speaker: Dan Heller has been working in software development for 13 years including time at Google, IBM, and long-forgotten startups. He has been at Indeed for the last 4 years, helping people get jobs by building products for Indeed’s employers and advertisers.
Report
Share
[@IndeedEng] Engineering Velocity: Building Great Software Through Fast Iteration
- 1. Engineering Velocity Building Great Software Through Rapid Iteration
- 2. Dan Heller Engineering Manager Employer Applications
- 6. Indeed #1 Job Site Worldwide More people find jobs on Indeed than anywhere else Over 100M monthly visitors Over 3B monthly searches
- 11. ≤
- 12. Rapid Iteration Leads to Better Products
- 14. Indeed Product Development Principles Measure Everything Test Everything (A/B Experiments) Iterate Quickly
- 16. Indeed Product Development Cycle Deliver MeasureLearn
- 17. Indeed Product Development Cycle Quickly! Deliver MeasureLearn
- 18. Indeed Product Development Cycle MeasureLearn MeasureLearn Deliver
- 20. Engineering Velocity (n) [en-juh-neer-ing vuh-los-i-tee] Our ability to quickly deliver implementations of ideas to job seekers & employers
- 22. Software Architecture Processes and Tools Risk Management Deliver
- 23. Software Architecture Processes and Tools Risk Management Deliver
- 30. Job Search Web App Doc Retrieval Company Info Authentication Location Index Search
- 31. Job Search Web App Index Search Doc Retrieval Company Info Location Authentication
- 32. Doc Retrieval Company Info Job Search Web App Authentication Location Index Search
- 33. Technical Challenges High Performance ● Speed is a feature ● 100ms response time
- 34. Technical Challenges Forward & Backward Compatibility ● Any client and any service can be updated at any time
- 35. High Performance SOA Framework Boxcar Image used with permission of O Scale Trains Magazine (oscalemag.com), photo credit Don McFall of Old Line Graphics http://go.indeed.com/boxcar
- 36. http://go.indeed.com/boxcar Boxcar: Load Distribution without Middleware
- 37. Represent complex data types Extremely compact and performant Forwards and backwards compatible Boxcar: API Compatibility
- 38. Boxcar: API Compatibility Request and response data encoded as Protocol Buffers Compatible evolution of service interfaces Reduces release order dependencies
- 39. Doc Retrieval Company Info Job Search Web App Authentication Location Index Search Boxcar!
- 41. Service Oriented Architecture for Web Frontends
- 49. Each page has different features different operational requirements different owners Each page should be its own application
- 52. Technical Challenges Provide a consistent user experience One implementation of shared functionality
- 53. Option: Shared Code Libraries Reduced flexibility of technology decisions Requires a release of all projects Occasional inconsistent user experience
- 57. <head> <title>Johnson & Johnson Careers</title> <link type="text/css" href="common.css"> <link type="text/css" href="comp.css"> </head> <body> <div id="page-header"> ...... </div> <div id="page-body"> <div id="page-frame"> <div id="comp-header"> ...... </div> </div> </div> </body>
- 58. <head> <title>Johnson & Johnson Careers</title> <link type="text/css" href="common.css"> <link type="text/css" href="comp.css"> </head> <body> <div id="page-header"> ...... </div> <div id="page-body"> <div id="page-frame"> <div id="comp-header"> ...... </div> </div> </div> </body>
- 59. <head> <title>Johnson & Johnson Careers</title> <link type="text/css" href="common.css"> <link type="text/css" href="comp.css"> </head> <body> <div id="page-header"> ...... </div> <div id="page-body"> <div id="page-frame"> <div id="comp-header"> ...... </div> </div> </div> </body>
- 60. REQUEST HEADERS Cookie, User-Agent, Referer, etc X-Indeed-Shell: 1 X-Indeed-Logged-In: [01] X-Indeed-Email: chris@indeed.com X-Indeed-Locale: en_US X-Indeed-Geo: US Navshell Web App
- 61. RESPONSE HEADERS Set-Cookie X-Indeed-Page-Title: Indeed Careers X-Indeed-Include-Js-Footer: /js/app.js X-Indeed-Include-Css: /style/main.css Navshell Web App
- 62. Navshell No duplicate code Centralized navigation changes Plain HTTP with strong contract Decoupled front-ends
- 64. Web-Based Delivery of Mobile App Features
- 65. Mobile Mobile Browser Mobile App
- 66. Mobile Mobile Browser Mobile App
- 68. ● One code base for mobile web, iOS, Android ● Simultaneous release of features across platforms Shared Code Across Platforms
- 69. ● No waiting for App Store approval ● No waiting for job seekers to upgrade Release On Our Own Schedule
- 70. Easy To Run Experiments ● A/B testing in native apps extremely challenging ● Reuse existing infrastructure for performing and evaluating experiments
- 71. Software Architecture Processes and Tools Risk Management Deliver
- 72. <Indeed map of the world>
- 73. one search. all jobs. una búsqueda. todos los empleos. Jedno vyhledávání. Všechna pracovní místa. Ein Klick. Alle Jobs. 一站搜尋，工作齊全 tek arama. bütün işler. uma busca. todos os empregos. één klik. alle vacatures. あらゆる仕事 まとめて検索 una ricerca. tutti i lavori.
- 74. Implement Commit Deploy to QA Deploy to Production Verify Before Internationalization
- 75. Implement Commit Translate Deploy to ProductionVerify Deploy to QA CommitExtract Internationalization Process Overhead
- 76. +1 day (20%)
- 77. +1 day (20%)
- 78. <Indeed map of the world>
- 79. <Indeed map of the world> <but with only US in orange>
- 82. Machine Translation Low quality translations are better than no translations But we needed something better
- 83. Human Translation Service Tuesday afternoon: Commit Code Fetch Machine Translations
- 84. Human Translation Service Tuesday afternoon: Commit Code Fetch Machine Translations Request Human Translations Wednesday morning: Retrieve Human Translations
- 85. Implement Commit Translate Deploy to ProductionVerify Deploy to QA CommitExtract Localization before Translation Automation
- 86. Implement Commit Deploy to ProductionVerify Deploy to QA Extract Localization with Translation Automation Retrieve
- 88. Software Architecture Processes and Tools Risk Management Deliver
- 89. Probability of issue Duration of issue Number of users affected Risk
- 90. Releases
- 91. Smaller Releases More releases → less functionality per release Probability of issue
- 94. Real-time monitoring with Datadog ● Agent reports summarized data ● System and customized business metrics ● Customizable dashboards
- 95. Healthcheck Framework ● Applications and services report internal state & health ● Divert traffic away from unhealthy instances ● Disable features for failing dependencies
- 96. Monitoring and Alerting ● Identify runtime issues ● Alert Ops when an issue arises
- 97. Proctor ● System for managing A/B experiments ● Segment visitors and assign test groups ● Manage feature visibility
- 99. Proctor Manage Experiments → Manage Behavior
- 100. Proctor "rule" : "loggedIn" "rule" : "${smartphone && 'JP' == country}" "rule" : "${(ua.OS.family == 'ios' && ua.OS.majorVersion >= 6)}" etc.
- 101. Proctor if (proctor.isNewThingEnabled()) { newThing.generateResults(); } else { oldThing.generateResults(); }
- 102. Feature Toggles Probability of issue "rule" : "fn:endsWith(confirmedUserEmail, '@indeed.com')"
- 103. Decouple Features from Releases Duration of issue Rollback features and releases independently
- 104. Number of users affected Small allocation changes 0% → 1% → 5% → 50% → 100%
- 105. Software Architecture Processes and Tools Risk Management Deliver
- 108. No Failed Projects
- 109. No Failed Projects
- 110. Fail Fast
- 111. No Deadline Crunches
- 112. There’s always another release in a few days
- 114. Large changes hidden behind feature toggles
- 115. Incremental progress validated in production environment
- 116. We built all of this incrementally, to meet our own challenges
- 117. Yours are different
- 118. v = d / t
- 119. With rapid iteration & quick delivery, we can cover more ground
- 120. Q&A