SlideShare a Scribd company logo

DevOps Tactical Adoption Theory: Continuous Testing

•Download as PPTX, PDF•
•
Berk Dülger
Berk Dülger

DevOps Tactical Adoption Theory tries to make the transition process as smooth as possible. It hypothesis each step towards DevOps maturity should bring a visible business value empowering management and team commitment for the next step. The innovative idea here, it is not required to add the tools/processes to stack from sequential beginning to end, but seeking benefit. The reason behind the theory is to encourage practitioners to apply each step one-by-one and then having the benefits in projects. Consequently, each step is tested in terms of utility and proved method validity for the further steps. In contrast to previous adoption models, our model indicates concrete activities rather than general statements. Theory built on the claim that many DevOps transition projects considered problematic, impractical or even unsuccessful causing concept to become a goal more than a technique. Basically, theory consists of different areas of interest describing various actions on a schema. In the session, it is planned to demonstrate “DevOps Tactical Adoption Theory” with focus on Delivery Pipeline/Testing Practices sectioned "Continuous Testing in DevOps".

1 of 38
DevOps Tactical Adoption Theory: Continuous Testing Berk Dülger – Consultant, Account Manager
TEST CONSULTING TEST OUTSOURCING TEST TRAINING
Learning Organizations Chapter I
Learning Organizations Learning organization is one in which people at all levels, individually and collectively, are continually increasing their capacity to produce results they really care about. Peter Senge
Learning Organizations’ Five Discipline Systems thinking: Ability to see whole Personal mastery: The process of life long learning Shared vision: A vision that shared & committed by everybody Mental model: Generalized & assumed view of how the world works Team learning: The process of developing the ability to create the desired results as a team
Systems Thinking
State of Agile/DevOps Adoption Chapter II
Lean/Agile Adoption in Europe
Global DevOps Rates
Principles and Practices Chapter III
Principles and Practices Lean is the basis of Agile Lean tells you to optimize the end-to-end process which creates value for your customer from the initial idea to collecting cash. Lean principles focus on flow more than anything else: bottlenecks in the process must be removed and wasteful activities need to be identified and avoided.
DevOps is not a goal, but a process of continuous improvement
Sustainable success requires both bottom-up practices and top-down management support
DevOps Practices Practices should address problems like, • Manual Efforts • Long feedback times • Long MTTR • Too much downtime • Lead time • Unrepetable work …
Automation in DevOps What can be automated? • Dev environments • Builds on pull requests and merge • Static code analysis • Code style checking • Dynamic code analysis • Verification • Archiving artifacts • Deployment
DevOps Tactical Adoption Theory: Continuous Testing
Tactical Adoption Theory Chapter IV
Tactical DevOps Adoption DevOps Tactical Adoption Theory tries to make the transition process as smooth as possible. It hypothesis each step towards DevOps maturity should bring a visible business value empowering management and team commitment for the next step. The idea here, it is not required to add the tools/processes to stack from sequential beginning to end, but seeking benefit.
Tactical DevOps Adoption The reason behind the theory is to encourage practitioners to apply each step one-by-one and then having the benefits in projects. Consequently, each step is tested in terms of utility and proved method validity for the further steps.
Large-Scale Adoption • Begin with an end in mind • Start with a pilot project • Make someone/unit responsible • Evangelize, build communities • Gain executive buy-in • Make people believe • Drive tool standardization • Automate, automate, automate: Build, Test, Deploy • Demostrate the value!
Either two ways; Choose to improve all categories for single project Or, choose one category to improve across all projects (i.e. Testing)
Continuous Testing Chapter V
Testing in DevOps There is no DevOps without Continuous Testing
Test Automation Pyramid Business Facing Technology Facing
Testing in DevOps Checkforexpected It’s not ideal to automate everything Findtheunexpected
Unit Integration UI High Medium Low Low Medium High Medium Long / High Short / Low Test Type Business Logic Coverage Code Coverage Execution Time / Costs
Testing in DevOps Unit Testing aims to test small chunks of your code in isolation from the rest of the world. UI Testing, different name of system testing, where you test the entire system together to ensure it does what it is supposed to do under real life conditions. (Unless by UI testing you mean usability / look & feel etc. testing)
Testing in DevOps You need both of these in most of projects, but at different times: unit testing during development (ideally from the very beginning, TDD!!!), and UI testing later, once you actually have some complete end-to-end functionality. If you already have a system running, but no tests, practically you have legacy code. Start to get the best test coverage achievable with the least effort first, which means high level functional tests. Adding unit tests is needed too, but it takes much more effort and starts to pay back later.
Continuous Testing Anti-Patterns Long and slow deployment pipelines Test Data Management is not a big deal We can skip non-functional tests Can be done by anyone Don’t need to refactor/maintain automated tests
Long and Slow Deployment Pipeline Anti-Pattern Tips: • If next stage(Automated Acceptance Tests) takes a significant amount of time (e.g. More than 30 minutes), embed a small subset of them into commit stage. So, feedback interval will be decreased to act fast on major incidents • Run tests in parallel (TestNG for Java and MbUnit for .Net might be good choices) • Focus on multi-threading for race conditions • Design atomic scenarios
Long and Slow Deployment Pipeline Anti-Pattern Tip: Prefer wide and shallow architecture rather than deep and narrow.
Test Data Management is not a Big Deal Anti-Pattern Four Design Techniques for Successful Test Automation Data Management A typical maturity level of data management for test automation process is outlined here; 1. Fully Integrated Test Data 2. Partially Independent Test Data 3. Storing Test Data in an External Source 4. Dynamic Test Data Management (Micro Services, GUI ?)
Non-Functional Testing Anti-Pattern Tips: • Select most business critical cases (Either widely used or critical for a business) • Test against a production replica environment, for example staging (As much as possible) • Do care about data. Effects computational cost • Focus on subject matter practices (Anything!) • Use automated-acceptance tests with counters (As a first step maybe)
Can be Done by Anyone Anti-Pattern Reasons Behind The Idea • Test automation is a development activity (Performance, Security Testing etc. as well ) • Convincing people to have a carreer in the field • Positioning the personnel and task in the right place • … May prefer a different job title, like ‘Software Development Engineer in Test’ (SDET)
Automation Maintanance is not Required Anti-Pattern Automation code is passive, meaning effected by any change in product code. Even with a perfect automation architecture, many times it is not that possible, you will need to redesign against living product. Sounds like Software Gardening!
Continuous Testing – E-commerce Case Study Inflection Point 2-3 Test Cases per Man/Day Nearly No Maintance Effort 3-5 Test Cases per Man/Day Less Maintance Effort (%20) 2-3 Test Cases per Man/Day Moderate Maintance Effort (%70) 3 Test Cases per Man/Day Moderate Maintance Effort (%50) ~1 Test Cases per Man/Day Heavy Maintance Effort (%90) Maximum number of test cases ~350 – 500 depending of SUT Based on metrics from 14 consultancy projects
QA Intelligence Survey 2015 Kristian Karl – TestIstanbul 2016 Mike Cohn Test Automation Pyramid Google Search Trends - DevOps searchsoftwarequality.techtarget.com/tip/Use-Agile-software-testing-principles-to-plan-your-tests blog.martinfenner.org/images/Agile-vs-iterative-flow.jpg slideshare.net/IBMDevOpsforEnterpriseSystems/lessons-learned-from-large-scale-adoption-of-devops-fori-bm-z-systems-software slideshare.net/ThoughtWorks/when-enterprise-meets-devops/15 PRIORITIZE_PILLAR_OF_PRACTICES15ESSENTIALCollaborationBuild_for slideshare.net/SkeltonThatcher/continuous-delivery-antipatterns-from-the-wild-matthew-skelton-continuous-lifecycle-london-2016 confengine.com/agile-india-2016/proposal/1680/how-to-explore-the-learning-organization-within-the-agile-organization References
Thank You Berk Dülger berk.dulger@keytorc.com https://tr.linkedin.com/in/berkdulger

More Related Content

DevOps Tactical Adoption Theory: Continuous Testing

  • 1. DevOps Tactical Adoption Theory: Continuous Testing Berk Dülger – Consultant, Account Manager
  • 2. TEST CONSULTING TEST OUTSOURCING TEST TRAINING
  • 4. Learning Organizations Learning organization is one in which people at all levels, individually and collectively, are continually increasing their capacity to produce results they really care about. Peter Senge
  • 5. Learning Organizations’ Five Discipline Systems thinking: Ability to see whole Personal mastery: The process of life long learning Shared vision: A vision that shared & committed by everybody Mental model: Generalized & assumed view of how the world works Team learning: The process of developing the ability to create the desired results as a team
  • 11. Principles and Practices Lean is the basis of Agile Lean tells you to optimize the end-to-end process which creates value for your customer from the initial idea to collecting cash. Lean principles focus on flow more than anything else: bottlenecks in the process must be removed and wasteful activities need to be identified and avoided.
  • 12. DevOps is not a goal, but a process of continuous improvement
  • 13. Sustainable success requires both bottom-up practices and top-down management support
  • 14. DevOps Practices Practices should address problems like, • Manual Efforts • Long feedback times • Long MTTR • Too much downtime • Lead time • Unrepetable work …
  • 15. Automation in DevOps What can be automated? • Dev environments • Builds on pull requests and merge • Static code analysis • Code style checking • Dynamic code analysis • Verification • Archiving artifacts • Deployment
  • 18. Tactical DevOps Adoption DevOps Tactical Adoption Theory tries to make the transition process as smooth as possible. It hypothesis each step towards DevOps maturity should bring a visible business value empowering management and team commitment for the next step. The idea here, it is not required to add the tools/processes to stack from sequential beginning to end, but seeking benefit.
  • 19. Tactical DevOps Adoption The reason behind the theory is to encourage practitioners to apply each step one-by-one and then having the benefits in projects. Consequently, each step is tested in terms of utility and proved method validity for the further steps.
  • 20. Large-Scale Adoption • Begin with an end in mind • Start with a pilot project • Make someone/unit responsible • Evangelize, build communities • Gain executive buy-in • Make people believe • Drive tool standardization • Automate, automate, automate: Build, Test, Deploy • Demostrate the value!
  • 21. Either two ways; Choose to improve all categories for single project Or, choose one category to improve across all projects (i.e. Testing)
  • 23. Testing in DevOps There is no DevOps without Continuous Testing
  • 24. Test Automation Pyramid Business Facing Technology Facing
  • 25. Testing in DevOps Checkforexpected It’s not ideal to automate everything Findtheunexpected
  • 26. Unit Integration UI High Medium Low Low Medium High Medium Long / High Short / Low Test Type Business Logic Coverage Code Coverage Execution Time / Costs
  • 27. Testing in DevOps Unit Testing aims to test small chunks of your code in isolation from the rest of the world. UI Testing, different name of system testing, where you test the entire system together to ensure it does what it is supposed to do under real life conditions. (Unless by UI testing you mean usability / look & feel etc. testing)
  • 28. Testing in DevOps You need both of these in most of projects, but at different times: unit testing during development (ideally from the very beginning, TDD!!!), and UI testing later, once you actually have some complete end-to-end functionality. If you already have a system running, but no tests, practically you have legacy code. Start to get the best test coverage achievable with the least effort first, which means high level functional tests. Adding unit tests is needed too, but it takes much more effort and starts to pay back later.
  • 29. Continuous Testing Anti-Patterns Long and slow deployment pipelines Test Data Management is not a big deal We can skip non-functional tests Can be done by anyone Don’t need to refactor/maintain automated tests
  • 30. Long and Slow Deployment Pipeline Anti-Pattern Tips: • If next stage(Automated Acceptance Tests) takes a significant amount of time (e.g. More than 30 minutes), embed a small subset of them into commit stage. So, feedback interval will be decreased to act fast on major incidents • Run tests in parallel (TestNG for Java and MbUnit for .Net might be good choices) • Focus on multi-threading for race conditions • Design atomic scenarios
  • 31. Long and Slow Deployment Pipeline Anti-Pattern Tip: Prefer wide and shallow architecture rather than deep and narrow.
  • 32. Test Data Management is not a Big Deal Anti-Pattern Four Design Techniques for Successful Test Automation Data Management A typical maturity level of data management for test automation process is outlined here; 1. Fully Integrated Test Data 2. Partially Independent Test Data 3. Storing Test Data in an External Source 4. Dynamic Test Data Management (Micro Services, GUI ?)
  • 33. Non-Functional Testing Anti-Pattern Tips: • Select most business critical cases (Either widely used or critical for a business) • Test against a production replica environment, for example staging (As much as possible) • Do care about data. Effects computational cost • Focus on subject matter practices (Anything!) • Use automated-acceptance tests with counters (As a first step maybe)
  • 34. Can be Done by Anyone Anti-Pattern Reasons Behind The Idea • Test automation is a development activity (Performance, Security Testing etc. as well ) • Convincing people to have a carreer in the field • Positioning the personnel and task in the right place • … May prefer a different job title, like ‘Software Development Engineer in Test’ (SDET)
  • 35. Automation Maintanance is not Required Anti-Pattern Automation code is passive, meaning effected by any change in product code. Even with a perfect automation architecture, many times it is not that possible, you will need to redesign against living product. Sounds like Software Gardening!
  • 36. Continuous Testing – E-commerce Case Study Inflection Point 2-3 Test Cases per Man/Day Nearly No Maintance Effort 3-5 Test Cases per Man/Day Less Maintance Effort (%20) 2-3 Test Cases per Man/Day Moderate Maintance Effort (%70) 3 Test Cases per Man/Day Moderate Maintance Effort (%50) ~1 Test Cases per Man/Day Heavy Maintance Effort (%90) Maximum number of test cases ~350 – 500 depending of SUT Based on metrics from 14 consultancy projects
  • 37. QA Intelligence Survey 2015 Kristian Karl – TestIstanbul 2016 Mike Cohn Test Automation Pyramid Google Search Trends - DevOps searchsoftwarequality.techtarget.com/tip/Use-Agile-software-testing-principles-to-plan-your-tests blog.martinfenner.org/images/Agile-vs-iterative-flow.jpg slideshare.net/IBMDevOpsforEnterpriseSystems/lessons-learned-from-large-scale-adoption-of-devops-fori-bm-z-systems-software slideshare.net/ThoughtWorks/when-enterprise-meets-devops/15 PRIORITIZE_PILLAR_OF_PRACTICES15ESSENTIALCollaborationBuild_for slideshare.net/SkeltonThatcher/continuous-delivery-antipatterns-from-the-wild-matthew-skelton-continuous-lifecycle-london-2016 confengine.com/agile-india-2016/proposal/1680/how-to-explore-the-learning-organization-within-the-agile-organization References

Editor's Notes

  1. Sounds familiar