Quality Assurance to Test Engineering – Insights From our Journey by Oksana Shmaliy & Dan Longcoy

Editor's Notes

1. Hi – my name is Dan, I have been Testing Manager for 10+ years. Hi – my name is Oksana. I have been in Software development for the last 20 years and have broad range of experiences in web development, test engineering, project management, and dev ops. Today we are going to talk about the changes that have been happening in testing industry, evolution of quality assurance to test engineering. Dan and I will share our own experiences that may help leaders to understand what is needed to embrace such transformation, how to prepare your team and learn where the evolution of test engineering is leading us.
2. In order to understand what has been happening in the testing industry and why companies embracing more and more test engineering practices and changing their development practices, lets take a look at trends and changes that have been happening in testing since 1970s. Testing process in each decade is influenced by what had been happening in the computer industry, project management, changes in software languages and architecture. 1970s – characterized by focus on development of testing techniques such as Cause-effect graph created by Elmendorf, Decision Table Testing, Mutation Testing, Basis Path testing, Introduction of Code Inspections and this is the time when Waterfall model was first introduced as well. 1980s – characterized by maturity of testing discipline and creation of tools (Rational software founded in 1982 and create ClearQuest tool), establishing understanding and principles of Quality (What is Total Quality Control by Ishakawa and Out of the Crisis by Deming where he laid out 14 management principles that define quality). V-model is introduced in 1986. This is also an object-oriented era that introduced Use Cases (1987 Ivar Jacobson). 1990s – full focus on quality and processes behind. ISO 9126 published – standard on Software Quality, W-model introduced to address shortcomings of V-model. Elements of new techniques appears that will define the next era of 2000s – Computer Aided Software Testing (1991) and Rational Unified Process published in 1992 – iterative development process., Scrum – 1993. Moving to object-oriented testing. Thought - The trend introduced by previous decade will continue to evolve in the next decade until new ideas or technologies are introduced. The most interesting decades for us are 2000s and 2010s. Let’s review them in more details in the next couple of slides.
3. 2000s characterized by important changes happening in three areas – software development, methodology and testing. This is the time when we see development of automation unit testing framework. The first unit testing automation framework was developed by Kent Beck for SmallTalk in 1994, but unit testing started gaining popularity after Kent Beck and Erich Gamma developed Junit framework around 1998. Methodology: Agile Manifesto 2001 – 12 principles 2003 – Behavior Driven Development by David North, first starting with Jbehave later Rbehave that was adopted by Rspec 2009 – first DevOps Days conference – an acknowledgment of a need to change culture to move to a better collaboration for Dev, Test, and Operations There were many interesting changes in Testing: Frist Automated Testing tool developed by IBM, Rational Functional Tester – 2002 2004 – development of Selenium 2005-2007 – Rspec development 2006 – recognition of importance of Automation Testing, first Google Test Automation Conference in London 2008 – Cucumber developed by by Aslak Hellesøy
4. Internet era created a new educated consumer, influencing Business to become customer centric. To understand what customers need and like created demand for shorter development and testing cycles, releasing product faster, getting customer feedback and ability to iterate quickly on new product ideas. Product development shifted to Minimum Viable product concept. Google responded by creating Google Experiments - automated A/B Testing platform for web based applications in 2013. The biggest shift also happened in infrastructure, with AWS services offerings finalized by 2009, enabling tech teams to reduce time to provision new infrastructure, therefore reducing overall delivery cycle. Demand to have shorter development cycles had direct impact on how companies started approaching testing with focus on reducing testing cycles.
5. To understand how to prepare QA organization for change and identify opportunities of reducing testing cycles, it is important to understand internal characteristics and external factors surrounding QA organization. Based on our experiences, here are three main factors that influence QA org externally: Relationship with other units within technology organization – Tech org dependent on QA knowledge of business and products Delivery methodologies Culture – trust, team work, ownership. Here are three things that influence QA org internally: What testing school org is following Skillset Practices and tools Let’s look at these three factors in details
6. In 2003 Bret Pettichord came up with the definition of four schools of testing. Analytical – applying logical and mathematical approach for testing, every line of code needs to be tested. Factory – software is done via project, testing is a measure of progress of a project, it has to be managed, every requirement has to be tested Quality Assurance – managing quality via managing process. QA is a Gatekeeper Context-Drive was introduced in 1999 and is all about exploratory testing
7. Skillset. Two distinct group of skills. Let’s look at Jill – she has been with the business for more than 10 years, she came to testing from the business. Jill knows products very well and very often closes gaps in requirements or helps new business product owners to understand existing products features. She is testing system manually, but knows core systems well and can validate mainframe changes. Jack on the other hand is new to the company, has been with the company for 2+ years. He has programming background, learned Ruby and automation quickly. He also expanded his knowledge and because of his programming skills he contributes to building automation execution and reporting tools. The gap between these two groups is obvious – we need both skills but how do we enable transfer of skills?
8. Automation practices and tools: Frameworks – Ruby and Cucumber and Specflow (specific to C#) Initial automation focused on FE testing, lots of validations that could have been covered by Unit Testing Automation Testing is integral part of CI process, however, current process is clunky and supported by set of different tools and sub-processes that slows overall CI execution The other challenge – few experts in the area of automation who is capable of build new frameworks
9. It was obvious that QA transformation was needed. Biggest factors that were driving this change were: To support Transformation and Modernization company strategy (reducing cost of testing efforts and creating shorter testing cycles) and closing skillset gaps – differences between two groups of testers. Some other important factors were around creating repeatable process and improving people mobility within organization.
10. What is Test Engineering? With moving toward Test Engineering we are still preserving value of product and legacy system knowledge, but we create an opportunity for people to grow and acquire new skills to be able to write automation scripts. The biggest change is in the mindset – shifting to programming mindset, able to solve problems using programmer’s skill. Knowledge of programming language, understanding of application architecture, database and what role APIs play in modern application architecture become critical in moving towards Test Engineering.
11. We took Change Management approach to transform organization. It took us between 4 to 6 months to complete this process. Leadership drove many discussions from the top. With the executive level driving the need for change, the clarity for the need is improved and better understood by the individual contributors. We followed ADKAR methodology of change. Awareness: of the business reasons for change. We had numerous conversations and town halls led by both QA Director and Technologies leaders. Desire: to engage and participate in the change. The team members have to have the desire to change. Some of our team members self selected to chose other careers and/or just retire. Knowledge: about how to change. Knowledge is the goal/outcome of training and coaching. As an organization, we paired Testers up with mentors that were building automation already. QA organization partnered with an outside company to conduct an in-house Automation Testing training. Ability: Ability is the goal/outcome of additional coaching, practice and time. An independent assessment was completed for the manual testers at the beginning of training and at the end to measure their growth. During the class, some people realized that this is not something that they would like to continue doing, so they self selected to choose new careers that met their current needs/goals. Reinforcement: to ensure change continues to occur. Reinforcement is the goal/outcome of adoption of QA Transformation, recognition of successful change, and making continued adjustments where/when appropriate. Continual training needs and opportunities for the Test Engineers is something we are still looking to improve. We encourage the team members to share opportunities that they see within testing that will benefit us. We are leveraging our more experienced Test Engineers to help junior Testers in the work with Automation. It’s paramount to continue to invest in your associates, especially following a transformation of this magnitude. When you are transforming your QA organization you will be changing and impacting other unit within overall Technology organization, therefore, engaging your partners from Development, Business analysis, project management is extremely important. There will be disruption to current delivery process and having your partner support will be crucial in leading this transformation.
12. Offshore resources  helped our organization through transition. Testing is becoming more of a commodity and can be shipped to other companies. We wanted to make sure to partner with companies that have our companies best interest in mind and will work well with the culture of our company. Two contracting firms are providing us with staff augmentation. These contracting firms leverage those offshore resources that have worked with us previously and understand our company to do preliminary screenings. The hit ratio of those we’ve interviewed has been very high. There is some overlap in Tester availability. We are still growing and improving our offshore process and interactions with that team. However, we can now have continuity of testing by entering test cases in the afternoon, running our batch flow at night and having offshore team to validate results before onsite team starts. When building our offshore partnership, emphasis on getting people who knows automation or who can learn it quickly was crucial. Without having your offshore team building automation, you will not be able to sustain transition to Test Engineering culture.
13. We continue to grow and climb the wall of new challenges, increasing automation testing presence in all applications we support.
14. Here are some results that we saw as a result of transition towards TE.
15. Let’s say, the organization has successfully completed QA transformation. What’s next? In order to stay relative, each org needs to understand new trends and directions that are being set up now that will influence future. Let’s consider Google example to understand the evolution of Test Engineering practices and where it led. When Google just started the company was small and there was lots of manual testing with elements of automation FE testing. With increased number of products and integrations, company could not afford long testing cycles and started moving testing upstream focusing on more automation testing – unit testing, API testing, developing testing frameworks. Two separate roles began to emerge in testing – Test Engineering and Software Engineer in Test. The first role developed extensive knowledge of products and was writing automation scripts. The second group focused on building frameworks and developing tools to support automation. New tools like protrector, EarlGrey, karma, expresso and others tools. All these efforts led to dramatically reducing testing time, but interestingly overall velocity did not increase. Other phases in development cycles became bottleneck. So SET started building tools to accelerate all other aspects of product development: Extending IDEs to make writing and reviewing code easier, shortening the “write code” cycle Automating release verification, shortening the “release code” cycle. Automating real time production system log verification and anomaly detection, helping automate production monitoring. Automating measurement of developer productivity, helping understand what’s working and what isn’t. A new trend clearly emerged, switching focus from reducing testing cycles to improving overall engineering productivity. Interesting fact, last GTAC conference was conducted in 2016 and in 2017 Google declared that they are going to re-purpose this conference to address overall engineering productivity. As you can see from the two roles that were successfully introduced at Google, one roles, SET has been evolving further, with this role Google closing the gap in other areas of software development cycle.
16. Testing is landing where it belongs – handled by developers via TDD to produce extensive test coverage. Software testing now relies more on Developers taking ownership of code quality and supplementing it by specialized automated integration and e2e testing. Continuous Integration and deployment practices orchestrated by a wide variety of tools provide feedback on code quality through the entire process of getting code to production. So what are the new trends ? In 2011 Alberto Savoia spoke at GTAC conference with the topic of Testing is Dead. He challenged known ways of testing and suggested that most testers should follow a new test-mentality, one which includes shifting their focus and priority from “Are we building it right?” to “Are we building the right it?” He introduced the concept of Pretotyping testing ideas. This is similar to idea of MVP. The automation that started with testing has clearly moved to the right as well. With the emergence of DevOps culture, engineers adopted Infrastructure as Code approach and are focusing on full automation of building and configuring infrastructure. In this new era testing team has an opportunity to provide a different perspective about quality of product that comes from analysis of production logs and monitoring, discovering real users issues and providing this feedback to developers to address application issues that could not be uncovered via testing, but are impacting real users.
17. In Conclusion, if any organization is considering a journey of transformation these are four important topics they need to consider: Have a clear definition of Why transformation is needed Understand current QA organization – skillset, type of testing used, etc. and what needs to be changed People – following Change Management practices will accelerate change and reduce resistance for change Future – thinking ahead and knowing what will be next, will help shape current direction and practices and allow you to plant new seeds for skills needed ahead.