How to Optimise Continuous Testing
•Download as PPTX, PDF•
1 like•583 views
Organisations turn to Agile and DevOps to improve customer experience by maximising the speed of delivery without sacrificing quality. As the champions of quality, testers achieve this goal through continuous testing. Test Automation plays a major role in continuous testing; it is the backbone of the continuous test process. To achieve continuous testing, automation must be applied at every stage of the development process. Developing a smart automation strategy and using the right tools is critical in achieving continuous testing since test scripts must be scalable and easy to maintain.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to How to Optimise Continuous Testing
Similar to How to Optimise Continuous Testing (20)
More from Sauce Labs
More from Sauce Labs (20)
Recently uploaded
Recently uploaded (20)
How to Optimise Continuous Testing
- 1. Achieving Continuous Testing through Test Optimization and Automation June 28th, 2018
- 2. Gerie Owen VP, Knowledge and Innovation US Giovanni Rago Solutions Engineer, EMEA Webinar Presenters
- 3. Agenda Continuous Testing Test Optimization Introduction SummaryTest Automation Q&A
- 4. Agenda Continuous Testing Test Optimization Introduction SummaryTest Automation Q&A
- 5. About QualiTest QA and testing is all that we do! World’s largest pure play testing company Over 3,000 testing professionals worldwide Business Assurance QA & Testing Focus Outcome Based Testing (OBT) Contractual guarantees for quality improvement & cost reduction Managed Testing Solutions (MTS) Long term, SLA-based testing outsourcing Industry & Technology Expert Global Service Delivery Model On-Shore, Near-Shore, Off-Shore and Crowd Testing
- 6. Sauce Labs Analytics Comprehensive Coverage Secure Most Reliable Experience DebuggingHighly Scalable
- 7. Agenda Continuous Testing Test Optimization Introduction SummaryTest Automation Q & A
- 8. ContinuousTesting | ContinuousTesting is the basis for Continuous Delivery | Continuous Delivery requires not only increased velocity, but more importantly increased quality. | Testing Cycles must be shortened yet quality must be increased. | Increases in microservices architectures require high-quality functional testing including automated API testing. | Builds, integrations and deployments are automated; testing must be integrated into the pipeline.
- 9. ContinuousTesting | A core capability in Agile and DevOps where all testing activities run continuously in an integrated fashion with development and delivery. Immediate bug fixing is enabled, test environments are provisioned instantly, and unit, functional, and nonfunctional tests are run in an automated way, orchestrated by Continuous Integration and Continuous Delivery tools. The ForresterWave™: ModernApplication FunctionalTest AutomationTools, Q4 2016
- 10. ContinuousTesting | ContinuousTesting requires risk analysis and process improvement as well as implementation of automation throughout the entire software development process. | “Shift Left” i.e., moving automated testing to the lowest levels of the testing pyramid is an important part of the approach.
- 11. ContinuousTesting vsTest Automation | ContinuousTesting requires automation, however, it encompasses much more. | ContinuousTesting is an approach to managing risk by focusing not only on improving testing efficiency but more importantly, increasing the effectiveness of our test processes.
- 12. | Assesses business risk coverage as its primary goal | Establishes a safety net that helps the team protect the user experience | Requires a stable test environment to be available on demand | Integrates seamlessly into the software delivery pipeline and DevOps toolchain | Delivers actionable feedback appropriate for each stage of the delivery pipeline Attributes of ContinuousTesting
- 13. ContinuousTesting | ContinuousTesting Minimizes Business RiskThrough: | Test Optimization | Automation throughout the Continuous Delivery Pipeline
- 14. Agenda Continuous Testing Test Optimization Introduction SummaryTest Automation Q & A
- 15. Test Optimization | Test Optimization encompasses risk assessment of the entire test scope: | Analyze test case for quality, effectiveness, automation potential | Evaluating the test suites to determine if they are providing the maximum amount of coverage with the fewest number of test cases.
- 16. OptimizeTest Scope | Select end-to-end tests that focus on the critical and high-risk features of the application | Understand all integrations involved in critical user workflows including the technologies employed in those applications (web, mobile, message/API-layer etc.) | Implement service virtualization for integrations and components that aren’t readily available in order to run end-to-end test continuously | Use the most realistic data possible for each test, include synthetic data generation where needed | Include exploratory testing to find user-experience and other bugs not easily found by automated tests
- 17. OptimizingTest Cases | Tests should be focused on the functionality they are validating. | Tests should be as simple as possible; the more complex the test, the more validation is required for the test code itself. | Even though service virtualization can be used to fill in for missing components in integration testing, tests should as independent as possible.
- 18. OptimizingTest Suites | Include test cases that validate high risk areas of code | Include test cases that validate customer-centric features | Include test cases that validate critical business workflows | Use FunctionalAutomation tools that create multi-purpose scripts | Cross-PlatformTesting • Scripts should run on Desktop,Web, Mobile & Server | Cross-BrowserTesting • Scripts should run on multiple browsers including Chrome, Edge, Safari and any others required | PerformanceTesting • Scripts should collect performance statistics including page loads etc.
- 19. Agenda Continuous Testing Test Optimization Summary Introduction Q & ATest Automation
- 20. Test Automation | Test automation is critical for Continuous Integration and Continuous Delivery | Fast feedback is not just a nice-to-have | Automated tests require a strategy | Which cases to automate? | How to automate them? | When to run automated tests? (At which point in the delivery pipeline?) | An automation framework can be a software project of its own | Specific technical know-how is needed | Maintenance needs to be accounted for | Good software engineering practices apply
- 21. Dependable, Consistent, ReproducibleTests | Test environment should mimic production to the highest degree possible | Having a system free of artifacts or stale data is paramount for reproducible results | Every session should be “fresh”: clean system cache, browser cache, cookies, etc. | Make sure your execution workstation and system under test have enough bandwidth and memory to support the number of simultaneous testing threads
- 22. Beware the Maintenance Overhead | …big deal in UI testing! | Can explode and needs to be managed carefully | Can be kept under control by following best practices | PageObject model | In general: loose coupling, good software development practices.
- 23. Enabling BetterTesting | Strive to build up speed | Monitor test performance over time | Leverage parallelism whenever possible | Fast test suites mean more testing, better QA | Trust in your framework is precious | Flaky tests produce noise, noisy builds reduce user compliance | An automation friendly SUT makes a big difference | Get your developers to help out!
- 24. Enabling BetterTesting | Use the right tool for the job | Keep functional tests separate from performance tests | If reliable testing can be achieved faster with a unit test, use a unit test | Place functional tests in the appropriate place in the build | Create shortest and most succinct tests possible by creating a state closest to the single feature being tested | API Calls to insert data | Deep linking | Setup andTeardown Methods | …enable atomic and autonomous tests!
- 25. KeepingTests Atomic | Atomic | Test a single feature | Short and succinct
- 27. KeepingTests Autonomous | Autonomous | Tests should run independently from other tests | One test’s actions should not drive another test’s acceptance criteria
- 28. KeepingTests Autonomous | (Bad) Example:
- 29. KeepingTests Autonomous | (Good) Example:
- 30. Benefits of Atomic and AutonomousTests | Easier to debug and maintain | Atomic -> Concise tests tend to be less complex | Autonomous -> Low coupling ensures tests are “sandboxed” | Easier to parallelize | Atomic + Autonomous -> Independent tests testing independent features
- 31. Why Run in Parallel | High test coverage across multiple key platforms is highly beneficial | …but the resulting number of tests can be high, and execution time can explode | Running tests sequentially is not an option anymore: parallelization is needed
- 32. Agenda Continuous Testing Test Optimization Summary Introduction Test Automation Q & A
- 33. Summary |ContinuousTesting Focuses on Risk |Optimize theTest Process |AutomatedCheckpoints |Automated feedback loops |Integrate testing through the delivery pipeline including deployments at each stage |MatureTest Automation Enables ContinuousTesting |Continued maintenance must be accounted for |Sticking to best practices pays big dividends |Parallelism enables high speed at high coverage
- 34. Agenda Continuous Testing Test Optimization Introduction SummaryTest Automation Q & A
- 36. Thank you
Editor's Notes
- Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
- Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
- Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
- Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
- Regression Test Selection (RTS) is one of the most popular methods for test case suite optimization. This method divides the test suite into reusable test cases, retestable test cases and obsolete test cases. Apart from all these, it also creates new test cases that test the program for areas not covered in current test cases. Another great way to maintain the effectiveness of the regression test suite is to have a good tracking mechanism between the features under development . This should be a consistent activity in order to maintain the test suites effectively as it would help test manager to verify the feature checklist and validate the test coverage for a feature that is being developed in the release. The effectiveness of the regression test suite can be easily maintained by monitoring the changes to the test suite. A clearly outlined process will ensure that only tests that are useful to the entire testing strategy get added to the test suite, which ensures the efficiency and usability of the test harness at a high level. Considering periodic cleanup of old tests is another great approach to maintain effectiveness of a feature rich regression test suite. In this scenario, all the existing tests in the test suite need to be analyzed for their effectiveness in a specific scenario. Also, there will be scenarios where certain features will not be supported because of the different product direction. In such cases, the relevant regression test suites should also be eased out. It will ensure robustness of the regression test suite for a long period of time. You can also measure the effectiveness of regression test suites on a release-to-release basis. It will allow you to know the root cause for reduction in the effectiveness of the test harness if any, and enable you to take appropriate action on the same. Collection of some metrics and their analysis could also be useful when it comes to the effectiveness of the regression test suite. It will help you get good visibility on the effectiveness of the regression test suite. You can consider different metrics such as percentage of defects found by the regression tests suite, their importance, etc.
- Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
- Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation
- Read the Agenda During the webinar, I will focus more on medical device testing while David will elaborate on medical device localisation