It is easy to measure code coverage when running unit tests.
However, very frequently the following questions come up:
- How can we measure API test coverage and e2e / UI test coverage?
- Does e2e / UI test coverage add value?
- If not, what other data can we look at to know if the e2e tests have good coverage?
This session is about understanding the above questions, and finding solutions for the same.
Sriram Angajala gave a presentation on test automation and DevOps. He discussed why learning DevOps is important even if it is not explicitly mentioned in one's job description. He covered topics like continuous integration, environments, collaboration, and monitoring tools. Sriram emphasized that the goal is to use DevOps tools effectively rather than becoming an expert in any single tool. He provided examples of tools like GitHub, Jenkins, Docker, and configuration files that can help automate testing across environments and integrate it with code deployment. Sriram advised attendees to start with basic DevOps skills like version control and scripting before advancing to more complex topics.
Argentesting 2018 - Patrones de diseño en la automatización: ¿PageObject o Sc...Argentesting
Taller ofrecido por Abel Quintana López (de Uruguay) durante la 3ra edición del Argentesting 2018
Los patrones de diseño son soluciones probadas y documentadas a problemas comunes en el desarrollo de software. Cuando implementamos nuestros frameworks de pruebas automatizadas los usamos para evitar problemas ya conocidos como el alto costo de mantenimiento de los test, código duplicado y flaky test.
El principal objetivo de este taller es analizar en profundidad el patrón más usado para las automatizaciones con Selenium (PageObject) y mostrar a través de ejercicios prácticos sus ventajas y desventajas en comparación con otra alternativa, el patrón ScreenPlay, que a diferencia del anterior si cumple con principios básicos de la programación orientada a objeto.
Durante este taller veremos como en la práctica la refactorización del patrón PageObject nos lleva a otro patrón de diseño de igual propósito conocido como ScreenPlay Pattern. La comparación entre ambos patrones nos ayudará a determinar si debemos continuar usando PageObject o usar uno u otro según el contexto. El objetivo es mostrar a la audiencia las debilidades que pueden tener sus pruebas automatizadas cuando usan el patrón PageObject y como pueden mejorar sus frameworks de pruebas automatizadas siendo consciente de estas desventajas.
Ademas, se analizara otra alternativa(ScreenPlay Pattern) que puede sustituir el patrón PageObject cumpliendo con los “SOLID principles”.
Requerimientos
No hay máquinas en el aula. Cada asistente debe llevar su propia notebook con: Sistema operativo Window con Visual Studio instalado.
El software con que vamos a estar trabajando es Visual Studio (preferentemente 2017), en este caso no nos serviría Visual Studio Code.
Abel Quintana López
https://www.linkedin.com/in/abel-quintana-lopez-371a31b9/
API Testing with Open Source Code and CucumberSmartBear
Ole Lensmar discusses various ways and tools for testing web APIs, focusing on using Cucumber. Watch the full presentation: https://www.infoq.com/presentations/api-testing-cucumber
The document discusses QA best practices in an Agile development environment. It describes key aspects of Agile like iterative delivery, self-organizing teams, and rapid feedback. It addresses challenges of fitting QA into short iterations and questions around testing approaches. The document advocates for testing to be collaborative, automated, and continuous throughout development. It provides recommendations for QA roles in activities like planning, stand-ups, retrospectives and acceptance testing. Overall it promotes testing practices in Agile that focus on early feedback, automation, and involvement of QA throughout the development process.
Testing software is important to uncover errors before delivery to customers. There are various techniques for systematically designing test cases, including white box and black box testing. White box testing involves examining the internal logic and paths of a program, while black box testing focuses on inputs and outputs without viewing internal logic. The goal of testing is to find the maximum number of errors with minimum effort.
Testing is a vital part of software development, since it is about finding the software bugs. You can choose from automated and manual testing. Automated testing is good for large projects with many systems users, while manual testing is good for smaller projects or for companies with insufficient budget.
You can also run Continuous testing. The purpose of Continuous testing and continuous integration is to enable continuous delivery, the end goal of which is to accelerate the delivery of high quality software.
Automation is essential for Continuous Testing but Automated Testing is not the same as Continuous Testing.
Try Jenkins Test Automation with codeBeamer ALM. Jenkins is an open source continuous integration tool written in Java. Jenkins provides continuous integration services for software development.
ATDD (Acceptance Test Driven Development) is a practice where the team discusses acceptance criteria and tests needed to ensure a feature meets user expectations. It has four stages: discuss functionality and tests, define test scenarios and data, develop using TDD, and demo tests to stakeholders. A sample shows a calculator user story, criteria, and tests for adding, subtracting, etc. ATDD improves understanding, collaboration, and catches defects earlier.
Building and Managing Scalable Applications on AWS: 1 to 500K usersAmazon Web Services
This presentation session from the Cloud Management, Services and Applications Theatre at Cloud Expo Europe 2014 explores the techniques and AWS services that you can use in order to build high scalability web applications on AWS. It also features a great overview of a high-scalability mobile application built by Myriad Group, and AWS customer, that serves over 41 million users.
This document provides an overview of software testing methods and automation testing. It discusses what software testing is, why it is necessary, and different types of testing like smoke testing, regression testing, and formal testing. It also covers the software development life cycle. The document compares manual and automation testing and discusses advantages and disadvantages of each. It provides details on Selenium, a popular tool for automation testing, including features of Selenium IDE, WebDriver, and Grid. The overall purpose is to introduce various software testing concepts and an automation testing tool.
The document discusses unit testing and provides guidance on how to effectively implement unit testing. It defines unit testing as testing individual units or components of software code to verify they are functioning as intended. The document outlines best practices for unit testing such as writing test cases that cover valid, invalid, and boundary conditions. It also recommends testing frameworks like JUnit that can automate running test cases. Overall, the document advocates for developing a rigorous unit testing strategy and practicing habits like writing tests first and continuously running tests to improve code quality.
The main reason to Software Testing is to find out defects which will cause an error to the users. Suppose any application which we are using in our smartphone which is not working properly, it can lead to many problem.
Continuous and Visible Security Testing with BDD-SecurityStephen de Vries
This presentation makes the case for adapting security requirements and processes to those used by developers. Specifically, it advocates the use of BDD (Given/When/Then) specifications to create self-verifying security requirements.
You've heard of infrastructure as code, with the BDD-Security framework, we can now write security-processes-as-code.
Real case studies of QA management in big teams (60-100 people). How to setup robust QA processes and approaches in them. Main impediments and problems, how to solve them. SAFe.
This document provides an introduction to automation testing. It discusses the need for automation testing to improve speed, reliability and test coverage. The document outlines when tests should be automated such as for regression testing or data-driven testing. It also discusses automation tool options and the process for automating tests. While automation testing provides benefits like time savings, it also has limitations such as the need for programming skills and maintenance of test code. Key challenges of automation testing include unrealistic expectations of tools and dependency on third party integrations.
A brief introduction to test automation covering different automation approaches, when to automate and by whom, commercial vs. open source tools, testability, and so on.
Learn about how Amazon enables its developers to rapidly release and iterate software while maintaining industry-leading standards on security, reliability, and performance. In this talk, we will discuss the culture of two pizza teams and how to maintain a culture of DevOps in a large enterprise.
This document provides guidelines for effective test automation at IBM Global Services. It discusses that automation is viewed as a silver bullet but can also frustrate if not implemented properly. The document recommends starting simple and increasing complexity as skills grow. It provides considerations for automation, such as tests that are long, repetitive, and non-subjective. The document outlines 10 guidelines for automation, including establishing standards, separating what from how, using a six phase process, and defining required skills. It also discusses functional decomposition and keyword-driven methodologies and provides an overview of automation tools.
Hamcrest is a library for creating matchers for usage in unit tests, mocks and UI validation. This talk gives a brief introduction to using and writing Hamcrest matchers.
The topics covered:
* Basic introduction to Hamcrest
* Using Matchers in assertions
* Using Matchers with Mockito
* Writing custom matchers
* Ad-hoc matchers
The document provides an introduction to Selenium WebDriver. It discusses the architecture and components of Selenium 1.0 and 2.0. It describes the Selenium WebDriver API for finding elements, performing actions, and handling windows/frames. It also provides an example of creating a test in Java to validate search results on the Rambler website using Selenium WebDriver.
The Automation Firehose: Be Strategic & Tactical With Your Mobile & Web TestingPerfecto by Perforce
The document discusses strategies for effective test automation. It emphasizes taking a risk-based approach to prioritize what to automate based on factors like frequency of use, complexity of setup, and business impact. The document outlines approaches for test automation frameworks, coding standards, and addressing common challenges like technical debt. It provides examples of metrics to measure the effectiveness of test automation efforts.
Small is Beautiful- Fully Automate your Test Case DesignGeorgina Tilby
This webinar will demonstrate how to fully automate testing and test case design using Grid Tools' Agile Designer software. The software can automatically generate the smallest number of test cases needed to maximize coverage from requirements. It can then push these test cases out as automated tests, removing bottlenecks caused by manually creating, executing, and maintaining tests. Attendees will learn how to automatically generate optimized test cases and scripts from models of their system under test and keep tests up to date as requirements change.
Load testing with Visual Studio and Azure - Andrew SiemerAndrew Siemer
In this presentation we will look at what web performance testing is and the various types of testing that can be performed. We will then dig into Visual Studio 2013 Ultimate to see that the Visual Studio platform is now a real contender in performance testing automation. And we will see how the Visual Studio integration with Visual Studio Online and Azure can take your web performance tests and spin up impressive load tests in a truly useful way.
The Automation Firehose: Be Strategic and Tactical by Thomas HaverQA or the Highway
The document discusses strategies for automating software testing. It emphasizes taking a risk-based approach to determine what to automate based on factors like frequency of use, complexity, and legal risk. The document provides recommendations for test automation best practices like treating automated test code like development code, using frameworks and tools to standardize coding practices, and prioritizing unit and integration testing over UI testing. It also discusses challenges that can arise with test automation like flaky tests, long test execution times, and keeping automation in sync with changing software. Metrics for measuring the effectiveness of test automation are presented, like test coverage, defect findings and trends, and time savings.
Despite the belief that a shared context and collaboration drives quality, too often, software testers and quality professionals struggle to find their place within today's integrated agile teams. This session is a practitioner’s view of testing and testing practices within an iterative/incremental development environment. We will begin with a discussion of some of the challenges of testing within an agile environment and delve into the guiding principles of Agile Testing and key enabling practices. Agile Testing necessitates a change in mindset, and it is as much, if not more, about behavior, as it is about skills and tooling, all of which will be explored.
Slides used in workshop on "Getting started with Appium 2.0" at AppiumConf 2021
https://confengine.com/conferences/appium-conf-2021/proposal/15634/getting-started-with-appium-20
Questions for successful test automation projectsDaniel Ionita
Test automation is not only about coding. Successful test automation involves critical thinking and clarity of objectives before actually beginning development. This material provides guidance in putting some of the right questions and how to think as for having an efficient and effective test automation in the context of your project.
Quality analysis pdf to study For your educationShraddhatadmare1
The document provides an overview of software testing. It defines software, testing, and software testing. Software testing is the process of identifying correctness, completeness, security, and quality of developed software applications. Testing is important to find defects, ensure customer satisfaction, and prove that software is reliable. The roles of testers include understanding requirements, writing test cases, executing tests, and reporting defects. Different models in the software development life cycle like waterfall, spiral, and agile are also discussed.
The document discusses test automation architecture and strategies for building an effective test automation pyramid. It recommends starting with unit tests at the base to test individual classes in isolation using mocks and stubs. Integration and API tests should be the next focus to test application logic through service calls. GUI tests should be limited to a fixed number of end-to-end tests while avoiding fragile tests. Cross-functional involvement from different teams is also important for maintaining tests. The goal is to have a stable, well-designed test automation strategy that finds bugs efficiently without wasting resources on fragile tests.
Test Automation Architecture That Works by Bhupesh DahalQA or the Highway
The document discusses test automation architecture and provides recommendations for building an effective architecture. It recommends prioritizing unit testing and API/service layer testing over GUI testing to create a testing pyramid. Unit tests should be isolated and test small pieces of code, while API tests can test application logic through service calls. GUI tests should be limited in number and used to test broad end-to-end scenarios, not every small scenario. The goal is to have fewer, more stable automated tests rather than many fragile tests. Following best practices like testing different layers, prioritizing types of tests, and continuous refactoring can help create a maintainable and effective test automation architecture.
A lean automation blueprint for testing in continuous deliverySauce Labs
Testing in Continuous Delivery changes test automation. It demands more automation but also requires immediate feedback. Many test teams today suffer from two extremes. Too little or no automation to organizations with hundreds of thousands of tests constantly running all kinds of VMs takes multiple days to execute. Any hope of Continuous Delivery or Pipeline Automation makes these states unsustainable.
This document discusses test automation, including the purpose of test automation, the test automation process, and the test automation pyramid. The key points are:
1. Test automation aims to improve test efficiency, provide wider test coverage, reduce costs, and speed up testing.
2. The test automation process involves defining the test scope, designing tests, coding tests, setting up the test environment, running tests, and maintaining automation over time.
3. The test automation pyramid illustrates that unit tests should form the base, as they are quick to write and run, while user interface tests are at the top as they are more complex and time-consuming.
Alexander Podelko - Context-Driven Performance TestingNeotys_Partner
Since its beginning, the Performance Advisory Council aims to promote engagement between various experts from around the world, to create relevant, value-added content sharing between members. For Neotys, to strengthen our position as a thought leader in load & performance testing. During this event, 12 participants convened in Chamonix (France) exploring several topics on the minds of today’s performance tester such as DevOps, Shift Left/Right, Test Automation, Blockchain and Artificial Intelligence.
Software Quality and Test Strategies for Ruby and Rails ApplicationsBhavin Javia
This document provides an overview of software quality and test strategies for Ruby and Rails applications. It discusses the importance of quality, managing quality through setting goals and measuring metrics. It outlines a test strategy template and covers test types, tools, and approaches for unit, integration, acceptance and other types of tests in Ruby/Rails. It also discusses test data management, defect management, and the Ruby/Rails testing ecosystem including various testing frameworks and quality/metrics tools.
Clean Code in Test Automation Differentiating Between the Good and the BadKnoldus Inc.
This session focuses on the principles of writing clean, maintainable, and efficient code in the context of test automation. The session will highlight the characteristics that distinguish good test automation code from bad, ultimately leading to more reliable and scalable testing frameworks.
This document provides best practices for Apex development at Salesforce, covering principles, processes, development conventions, testing, and more. It recommends tracking requirements, using source control, automating deployments, writing efficient code, handling errors properly, writing tests with assertions and different scenarios, and avoiding governor limits. The goal is to produce code that is well-structured, commented when needed, performs efficiently with queries and loops, has high test coverage, uses test data properly, and can be tested at scale.
Continuous Delivery & Testing Madrid AfterTestPeter Marshall
This document discusses continuous delivery and the testing challenges it presents. It describes a large financial institution's goal of delivering updates to their digital payment platform weekly. Initially they had a 9 month release cycle with many sequential test phases. The author proposed reducing features per release and incorporating more testing into development. This faced resistance from test managers concerned about quality and independent testing. Through education and demonstrations, the approach was modified to have functional testing after development and the culture shifted to prioritize automated testing and quality as everyone's responsibility.
Does your functional automation really add value?Anand Bagmar
We all know that automation is one of the key enablers for those on the CI-CD journey.
Most teams are:
• implementing automation
• talking about its benefits
• up-skilling themselves
• talking about tooling
• etc.
In my experience, unfortunately most of the functional automation that is built is:
• not optimal
• not fit-for-purpose
• does not run fast enough
• gives inconsistent feedback, hence unreliable
Hence, for the amount of effort invested in implementing automation, are you really getting the value from this activity?
In this talk, we will discuss these challenges and why it would lead to poor ROI of automation. More importantly, we will discuss the following techniques to make automation valuable:
• know the objective for the automation framework
• establish criteria for tests to be automated
• design your framework with proper abstraction layers
• develop using appropriate design patterns
When is a project ready for Software Automation_NEWMike Christesen
This document discusses when a project is ready for software automation. It begins with introductions and an agenda that will cover misconceptions about automation, selecting the right tools, and real-world gains from automation. Some key points made include the benefits of automation being quick runs, lower costs, and finding more bugs, while manual testing has difficulties like being repetitive. It suggests automating repetitive, data-intensive, high risk tests and testing early. Selecting the right tools that cover all technologies and maintaining resilient tests is also discussed. Metrics like time savings and reduced defects are presented from an IDC study on automation ROI.
Streamlining End-to-End Testing Automation with Azure DevOps Build & Release Pipelines
Automating end-to-end (e2e) test for Android and iOS native apps, and web apps, within Azure build and release pipelines, poses several challenges. This session dives into the key challenges and the repeatable solutions implemented across multiple teams at a leading Indian telecom disruptor, renowned for its affordable 4G/5G services, digital platforms, and broadband connectivity.
Challenge #1. Ensuring Test Environment Consistency: Establishing a standardized test execution environment across hundreds of Azure DevOps agents is crucial for achieving dependable testing results. This uniformity must seamlessly span from Build pipelines to various stages of the Release pipeline.
Challenge #2. Coordinated Test Execution Across Environments: Executing distinct subsets of tests using the same automation framework across diverse environments, such as the build pipeline and specific stages of the Release Pipeline, demands flexible and cohesive approaches.
Challenge #3. Testing on Linux-based Azure DevOps Agents: Conducting tests, particularly for web and native apps, on Azure DevOps Linux agents lacking browser or device connectivity presents specific challenges in attaining thorough testing coverage.
This session delves into how these challenges were addressed through:
1. Automate the setup of essential dependencies to ensure a consistent testing environment.
2. Create standardized templates for executing API tests, API workflow tests, and end-to-end tests in the Build pipeline, streamlining the testing process.
3. Implement task groups in Release pipeline stages to facilitate the execution of tests, ensuring consistency and efficiency across deployment phases.
4. Deploy browsers within Docker containers for web application testing, enhancing portability and scalability of testing environments.
5. Leverage diverse device farms dedicated to Android, iOS, and browser testing to cover a wide range of platforms and devices.
6. Integrate AI technology, such as Applitools Visual AI and Ultrafast Grid, to automate test execution and validation, improving accuracy and efficiency.
7. Utilize AI/ML-powered central test automation reporting server through platforms like reportportal.io, providing consolidated and real-time insights into test performance and issues.
These solutions not only facilitate comprehensive testing across platforms but also promote the principles of shift-left testing, enabling early feedback, implementing quality gates, and ensuring repeatability. By adopting these techniques, teams can effectively automate and execute tests, accelerating software delivery while upholding high-quality standards across Android, iOS, and web applications.
Automating the real-user scenarios across multi-apps, and multi-devicesAnand Bagmar
This document discusses teswiz, an open source framework for automating real-user scenarios across multiple apps, devices, users, and platforms. It can simulate user actions and behavior to test web, mobile web, Android, iOS, Windows, MacOS and Linux applications. Teswiz uses Cucumber, Appium, Selenium and other tools and supports features like multi-device testing, visual testing with Applitools, and generating reports with ReportPortal. The document provides instructions on getting started with teswiz and links to its GitHub page.
Visual Validation - The missing tip of the automation pyramid @GoT2022Anand Bagmar
The document discusses visual validation testing as a missing piece of the automation testing pyramid. Visual testing is important but challenging as it is typically done manually, which is tedious, error-prone, and difficult to scale. Automating visual testing can help by taking screenshots of expected user interfaces and comparing them to actual screenshots. However, automating also poses challenges around false positives/negatives, maintaining baselines, and accounting for product changes. The document promotes using artificial intelligence in visual test automation to help address these challenges and advocates for including visual testing in an organization's overall quality and automation strategy.
In this talk, we will talk about, and see examples of various types of patterns you can use for:
- Build your Test Automation Framework
- Test Data Management
- Locators / IDs (for finding / interacting with elements in the browser / app)
Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!
Change Tyres In A Moving Car - Make Functional Test Automation Effective KeynoteAnand Bagmar
https://confengine.com/conferences/selenium-conf-2022/proposal/17141/changing-tyres-in-a-moving-car-making-functional-test-automation-effective
In my experience, I have seen teams struggle with more than one, and in many cases, all of the above statements.
On retrospection, I have realized, that most of these challenges are because of a combination of the following issues:
Lack of holistic vision, understanding, skills, and capabilities for taking on this seemingly easy activity of functional test automation
Lack of time for design, implementation, and maintenance of the automated test code
In this session, I will share, with examples, the features & capabilities that are not used right in Test Automation implementation resulting in you answering “NO” to the above statements. These are anti-patterns of functional automation implementation and have a negative impact on the quality of feedback from your automated tests.
More importantly, I will share approaches and solutions of how you can avoid these anti-patterns and evolve to answer “YES” to each of the above statements.
Slides from my talk on "Rewrite Vs Refactor" given at Agile India 2021. https://confengine.com/conferences/agile-india-2021/proposal/15495/rewrite-vs-refactor
In this session, I will share various examples and experiences and as a result of being in such situations, the factors I looked at when enhancing the code-base to decide - should I refactor or rewrite the code-under-consideration to be able to move forward faster, while moving towards the long-term vision.
Though I will focus on various examples of Test Automation, this session is applicable for any role that writes / maintains code of any nature.
Next Generation Functional & Visual Testing powered by AIAnand Bagmar
Slides from hands-on workshop at Agile India 2021.
Machine setup instructions are available in the proposal - https://confengine.com/conferences/agile-india-2021/proposal/15552/next-generation-functional-amp-visual-testing-powered-by-ai
In this workshop, using numerous examples, we will explore:
Why Automated Visual Validation is essential to be part of your Test Strategy
How Visual AI increases the coverage of your functional testing, while reducing the code, and increasing stability of your automated tests
Potential solutions / options for Automated Visual Testing, with pros & cons of each
How an AI-powered tool, Applitools Eyes, can solve this problem
Hands-on look at Applitools Visual AI and how to get started using it
The Best Test Automation Framework is...Anand Bagmar
I had the privilege of delivering the opening keynote at the recent Future Of Testing event focused on Test Automation Frameworks on 30th September 2021. My topic was “The best test automation framework is …”.
Here is the mind map I used in my keynote presentation.
https://applitools.com/future-of-testing-frameworks-north-america-2021/
Slides from my talk on how to Eradicate Flaky Tests from AppiumConf 2021
https://confengine.com/conferences/appium-conf-2021/proposal/15581/eradicate-flaky-tests
Visual Validation - The missing tip of the automation pyramid @AgileIndia2020Anand Bagmar
https://confengine.com/agile-india-2020/proposal/11065/visual-validation-the-missing-tip-of-the-automation-pyramid
The top of the pyramid is our UI / end-2-end functional tests - which simulate end-user behavior and interactions with the product-under test.
While Automation helps validate functionality of your product, aspects of UX validations can only be seen and captured by the human eye and is hence mostly a manual activity. This is an area where AI & ML can truly help.
With everyone wanting to be Agile, make quick releases, the look & feel / UX validation, which is a slow, and error-prone activity, quickly becomes a huge bottleneck. In addition, with any UX related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling and worse - dilute your user-base.
In this session, using numerous examples, we will explore:
Why Automated Visual Validation is essential to be part of your Test Strategy
Potential solutions / options for Automated Visual Testing, with pros & cons of each
How an AI-powered tool, Applitools Eyes, can solve this problem.
This document outlines an agenda for a Selenium workshop. It covers topics like getting started with Selenium, implementing page object models and design patterns, handling test data, using assertions, and responsive web design testing. Exercises are provided to demonstrate concepts like building page objects, running tests across browsers, and using test data. The goal is to help students learn Selenium concepts and address common automation challenges.
When one has fun at work, work becomes fun. However, daily pressures, metrics, KPIs, and what not, have dissolved the fun, and made work drudgery in various ways. This creates stress for individuals, in teams, and across teams, there is mistrust, unnecessary competition, blame, finger-pointing ….What better way to learn, and re-learn the basics of life, work, team-work - than to play a game, have fun, and correlate it with how life and work indeed should be treated as a game, and we should have fun in this journey. Only then can people truly succeed, and so can organisations.Here, we will play a game – “Collaboration - A Taboo!” – where you will –Re-learn collaboration techniques via a game!Learning applicable for individuals & teams, in small or big organisationsRe-live your childhood when playing this gameBe prepared for a twist which will leave you thinking!
Visual Validation - The Missing Tip of the Automation PyramidAnand Bagmar
The Test Automation Pyramid is not a new concept. The top of the pyramid is our UI / end-2-end functional tests - which should cover the breadth of the product.
What the functional tests cannot capture though, is the aspects of UX validations that can only be seen and in some cases, captured by the human eye. This is where the new buzzwords of AI & ML can truly help.
In this session, we will explore why Visual Validation is an important cog in the wheel of Test Automation and also different tools and techniques that can help achieve this. We will also see a demo of Applitools Eyes - and how it can be a good option to close this gap in automation!
What is Agile Testing? How does Automation help?Anand Bagmar
Agile Methodology is not new. Many organisations / teams have already adopted Agile way of Software Development or are in the enablement journey for the same.
What does this mean for Testing? There is no doubt that the Testing approach and mindset also needs to change to be in tune with the Agile Development methodology.
Learn what does it mean to Test on Agile Projects. Also, learn how Test Automation approach needs to change for the team to be successful!
Also, see the skills and capabilities required to make anyone in the Quality / Testing role add tremendous value to the team!
The What, Why and How of Analytics TestingAnand Bagmar
Here are slides from my talk on "What, Why and How of Analytics Testing" at Selenium Conference, Berlin 2017.
This talk focusses on Analytics related to Browser & Mobile Native apps, with impact on / of IoT (Internet of Things) and Big Data.
See my blog for more details - https://essenceoftesting.blogspot.com/2017/10/analytics-forgotten-child.html
Patterns of a “good” test automation frameworkAnand Bagmar
This document discusses patterns for test automation frameworks. It begins by introducing common patterns like page objects, business layer, and factories. It then provides examples of page object and business layer page object patterns. The document also discusses test data patterns and different ways to specify test data and locators. Finally, it outlines advantages of using patterns like reduced complexity, reusability, and maintenance. The key message is that the best pattern depends on the specific test automation context.
Attendance Tracking From Paper To DigitalTask Tracker
If you are having trouble deciding which time tracker tool is best for you, try "Task Tracker" app. It has numerous features, including the ability to check daily attendance sheet, and other that make team management easier.
NBFC Software: Optimize Your Non-Banking Financial CompanyNBFC Softwares
NBFC Software: Optimize Your Non-Banking Financial Company
Enhance Your Financial Services with Comprehensive NBFC Software
NBFC software provides a complete solution for non-banking financial companies, streamlining banking and accounting functions to reduce operational costs. Our software is designed to meet the diverse needs of NBFCs, including investment banks, insurance companies, and hedge funds.
Key Features of NBFC Software:
Centralized Database: Facilitates inter-branch collaboration and smooth operations with a unified platform.
Automation: Simplifies loan lifecycle management and account maintenance, ensuring efficient delivery of financial services.
Customization: Highly customizable to fit specific business needs, offering flexibility in managing various loan types such as home loans, mortgage loans, personal loans, and more.
Security: Ensures safe and secure handling of financial transactions and sensitive data.
User-Friendly Interface: Designed to be intuitive and easy to use, reducing the learning curve for employees.
Cost-Effective: Reduces the need for additional manpower by automating tasks, making it a budget-friendly solution. Benefits of NBFC Software:
Go Paperless: Transition to a fully digital operation, eliminating offline work.
Transparency: Enables managers and executives to monitor various points of the banking process easily.
Defaulter Tracking: Helps track loan defaulters, maintaining a healthy loan management system.
Increased Accessibility: Cutting-edge technology increases the accessibility and usability of NBFC operations. Request a Demo Now!
Are you wondering how to migrate to the Cloud? At the ITB session, we addressed the challenge of managing multiple ColdFusion licenses and AWS EC2 instances. Discover how you can consolidate with just one EC2 instance capable of running over 50 apps using CommandBox ColdFusion. This solution supports both ColdFusion flavors and includes cb-websites, a GoLang binary for managing CommandBox websites.
WhatsApp Tracker - Tracking WhatsApp to Boost Online Safety.pdfonemonitarsoftware
WhatsApp Tracker Software is an effective tool for remotely tracking the target’s WhatsApp activities. It allows users to monitor their loved one’s online behavior to ensure appropriate interactions for responsive device use.
Download this PPTX file and share this information to others.
COMPSAC 2024 D&I Panel: Charting a Course for Equity: Strategies for Overcomi...Hironori Washizaki
Hironori Washizaki, "Charting a Course for Equity: Strategies for Overcoming Challenges and Promoting Inclusion in the Metaverse", IEEE COMPSAC 2024 D&I Panel, 2024.
What is OCR Technology and How to Extract Text from Any Image for FreeTwisterTools
Discover the fascinating world of Optical Character Recognition (OCR) technology with our comprehensive presentation. Learn how OCR converts various types of documents, such as scanned paper documents, PDFs, or images captured by a digital camera, into editable and searchable data. Dive into the history, modern applications, and future trends of OCR technology. Get step-by-step instructions on how to extract text from any image online for free using a simple tool, along with best practices for OCR image preparation. Ideal for professionals, students, and tech enthusiasts looking to harness the power of OCR.
React and Next.js are complementary tools in web development. React, a JavaScript library, specializes in building user interfaces with its component-based architecture and efficient state management. Next.js extends React by providing server-side rendering, routing, and other utilities, making it ideal for building SEO-friendly, high-performance web applications.
Cultural Shifts: Embracing DevOps for Organizational TransformationMindfire Solution
Mindfire Solutions specializes in DevOps services, facilitating digital transformation through streamlined software development and operational efficiency. Their expertise enhances collaboration, accelerates delivery cycles, and ensures scalability using cloud-native technologies. Mindfire Solutions empowers businesses to innovate rapidly and maintain competitive advantage in dynamic market landscapes.
Software development... for all? (keynote at ICSOFT'2024)miso_uam
Our world runs on software. It governs all major aspects of our life. It is an enabler for research and innovation, and is critical for business competitivity. Traditional software engineering techniques have achieved high effectiveness, but still may fall short on delivering software at the accelerated pace and with the increasing quality that future scenarios will require.
To attack this issue, some software paradigms raise the automation of software development via higher levels of abstraction through domain-specific languages (e.g., in model-driven engineering) and empowering non-professional developers with the possibility to build their own software (e.g., in low-code development approaches). In a software-demanding world, this is an attractive possibility, and perhaps -- paraphrasing Andy Warhol -- "in the future, everyone will be a developer for 15 minutes". However, to make this possible, methods are required to tweak languages to their context of use (crucial given the diversity of backgrounds and purposes), and the assistance to developers throughout the development process (especially critical for non-professionals).
In this keynote talk at ICSOFT'2024 I presented enabling techniques for this vision, supporting the creation of families of domain-specific languages, their adaptation to the usage context; and the augmentation of low-code environments with assistants and recommender systems to guide developers (professional or not) in the development process.
CViewSurvey Digitech Pvt Ltd that works on a proven C.A.A.G. model.bhatinidhi2001
CViewSurvey is a SaaS-based Web & Mobile application that provides digital transformation to traditional paper surveys and feedback for customer & employee experience, field & market research that helps you evaluate your customer's as well as employee's loyalty.
With our unique C.A.A.G. Collect, Analysis, Act & Grow approach; business & industry’s can create customized surveys on web, publish on app to collect unlimited response & review AI backed real-time data analytics on mobile & tablets anytime, anywhere. Data collected when offline is securely stored in the device, which syncs to the cloud server when connected to any network.
4. • Tests are “testing” the “right functionality / behavior”!
• Tests give quick feedback
• IF Tests fail, the correct reason of failure is provided
• Tests are optimized & efficient
@BagmarAnand
Criteria of Automated Tests
https://i.pinimg.com/474x/ec/8a/17/ec8a176036490a3338f806172a23e4b3.jpg
6. Unit (xUnit / JavaScript)
Manual / Exploratory
Web Service
UI / e2e
Business-facing
Tests
Technology-facing
Tests
@BagmarAnand
Test Pyramid
Slow
More integration
Expensive
Fast
More isolation
Cheap
7. @BagmarAnand
Test Pyramid – Includes NFRs
Unit (xUnit / JavaScript)
Manual / Exploratory
Web Service
UI / e2e
Performance
Security
Accessibility
Analytics
11. • Gives understanding of current state
• Provides indication of potential problems / issues
• Ability to make educated, data-driven decisions, quickly!
@BagmarAnand
Why are Metrics important?
13. • Abused
• Taken out of context & proportion
• Seem to be set in stone!
@BagmarAnand
(Sad) Reality of Metrics
https://i.pinimg.com/474x/ec/8a/17/ec8a176036490a3338f806172a23e4b3.jpg
14. @BagmarAnand
Way of working has evolved
https://upload.wikimedia.org/wikipedia/commons/1/1c/Evolution-des-wissens.jpg
15. @BagmarAnand
We are still stuck in the past in measuring Quality!
https://previews.123rf.com/images/riedjal/riedjal1403/riedjal140300005/26549722-prehistoric-age-of-beardy-caveman-surprised-to-find-a-laptop.jpg
17. • Test Case Effectiveness = (Number of defects detected / Number of test cases run) x 100
• Test Case Productivity = (Number of Test Cases / Efforts Spent for Test Case Preparation)
• Test Coverage = Number of detected faults/number of predicted defects.
• Test Code Coverage = Produce code coverage by e2e / UI tests
• Requirement Coverage = (Number of requirements covered / Total number of requirements) x 100
• Test Design Coverage = (Total number of requirements mapped to test cases / Total number of requirements) x 100
• Test Execution Coverage = (Total number of executed test cases or scripts / Total number of test cases or scripts
planned to be executed) x 100
• %ge of work completed & yet to be completed
• Time to complete the remaining work
@BagmarAnand
(Traditional) Test Metrics
18. • There is a lot of work required to capture these metrics
• The result of these metrics does not necessarily tell how can I make
the product quality better
• How do these metrics (capturing, measuring and analyzing) work in
the Agile way of working?
Hence, such metrics add very limited value to teams!
@BagmarAnand
Traditional Test Metrics
19. Metrics that add very limited value to teams should be
avoided!
They seem to be more for managers than for teams to
build a high-quality product!
@BagmarAnand
Traditional Test Metrics
20. @BagmarAnand
How to find better ways?
https://previews.123rf.com/images/bosecher/bosecher1509/bosecher150900020/45695557-vector-cartoon-illustration-of-a-stoneage-man-sitting-on-a-stone-wheel.jpg
22. • Ways of working has evolved, but people are still the
same!
• Roles / Titles have changed, but people are still the same!
• We still need to measure quality, so we end up using
techniques what we knew before!
@BagmarAnand
Why is it a challenge?
23. • Metrics should be easy and quick to capture
• Use practices that will allow metrics to be captured
automatically
• Generate meaningful & visual reports to infer Quality
@BagmarAnand
Evolved way of thinking about Metrics
29. Unit (xUnit / JavaScript)
Manual / Exploratory
Web Service
UI / e2e
Business-facing
Tests
Technology-facing
Tests
@BagmarAnand
Test Pyramid
Slow
More integration
Expensive
Fast
More isolation
Cheap
30. Unit (xUnit / JavaScript)
Manual / Exploratory
Web Service
UI / e2e
@BagmarAnand
Quality is NOT some report from QA Team!
Test Execution
report created
by QA Team
31. @BagmarAnand
All aspects of testing combined indicate Quality
of product-under-test
Unit (xUnit / JavaScript)
Manual / Exploratory
Web Service
UI / e2ePerformance
Security
Accessibility
Analytics
Quality of product-under-test
33. • Test Case Effectiveness = (Number of defects detected / Number of test cases run) x 100
• Test Case Productivity = (Number of Test Cases / Efforts Spent for Test Case Preparation)
• Test Coverage = Number of detected faults/number of predicted defects.
• Test Code Coverage = Produce code coverage by e2e / UI tests
• Requirement Coverage = (Number of requirements covered / Total number of requirements) x 100
• Test Design Coverage = (Total number of requirements mapped to test cases / Total number of requirements) x 100
• Test Execution Coverage = (Total number of executed test cases or scripts / Total number of test cases or scripts
planned to be executed) x 100
• %ge of work completed & yet to be completed
• Time to complete the remaining work
@BagmarAnand
(Traditional) Test Metrics
34. I often get asked ….
@BagmarAnand
https://lh3.googleusercontent.com/proxy/3JFWJGTlmTym_g9xCZpHIFNLN3rQZu3js8f7iJDHFc6CCeTDb4hpLJaaQm1YCe2BfxyuA_7Lv3B4Clt7MpICbe_Ay09TY8k
35. How can you capture code
coverage from e2e / UI tests?
@BagmarAnand
https://lh3.googleusercontent.com/proxy/3JFWJGTlmTym_g9xCZpHIFNLN3rQZu3js8f7iJDHFc6CCeTDb4hpLJaaQm1YCe2BfxyuA_7Lv3B4Clt7MpICbe_Ay09TY8k
36. Let us first understand
Code Coverage?
@BagmarAnand
40. But, does it really make sense?
@BagmarAnand
https://lh3.googleusercontent.com/proxy/3JFWJGTlmTym_g9xCZpHIFNLN3rQZu3js8f7iJDHFc6CCeTDb4hpLJaaQm1YCe2BfxyuA_7Lv3B4Clt7MpICbe_Ay09TY8k
43. When tests are running:
• Instrumented product-under-test to allow tracing and
capturing data / metrics
• Isolated environment – i.e. No one else should be using
the same environment
@BagmarAnand
Criteria for Code Coverage & Analysis
44. @BagmarAnand
Feasibility of Code Coverage & Analysis
How the tests run?
Code Coverage
Analysis Possible?
Challenges
Unit Tests
• Mocks / Stubs
• Isolated environment
Yes
Minimal
• Standard mocking & stubbing
practices should suffice
45. @BagmarAnand
Feasibility of Code Coverage & Analysis
How the tests run?
Code Coverage
Analysis Possible?
Challenges
Unit Tests
• Mocks / Stubs
• Isolated environment
Yes
Minimal
• Standard mocking & stubbing
practices should suffice
API / WebService
Tests
• Against deployed Service(s)
• Dependent services may be
stubbed / mocked for
better orchestration
Yes, but at service level
Medium
• From a setup / mocking
dependencies perspective
46. @BagmarAnand
Feasibility of Code Coverage & Analysis
How the tests run?
Code Coverage
Analysis Possible?
Challenges
Unit Tests
• Mocks / Stubs
• Isolated environment
Yes
Minimal
• Standard mocking & stubbing
practices should suffice
API / WebService
Tests
• Against deployed Service(s)
• Dependent services may be
stubbed / mocked for
better orchestration
Yes, but at service level
Medium
• From a setup / mocking
dependencies perspective
e2e / UI Tests
• Against deployed
environment
• All integrations in place
• Data / config setup may be
necessary
• Some systems may be
stubbed
Yes, but may not be a
good strategy (cost Vs
value)
High
• Need full environment setup
• Keeping environment “pure” and
“isolated” from use by tests or
humans for tracing to work
• e2e tests may trigger only a small
set of code paths compared to
Unit & API tests
47. Unit (xUnit / JavaScript)
Manual / Exploratory
Web Service
UI / e2e
Business-facing
Tests
Technology-facing
Tests
@BagmarAnand
Test Pyramid
Slow
More integration
Expensive
Fast
More isolation
Cheap
48. @BagmarAnand
Feasibility of Code Coverage & Analysis
How the tests run?
Code Coverage
Analysis Possible?
Challenges
Unit Tests
• Mocks / Stubs
• Isolated environment
Yes
Minimal
• Standard mocking & stubbing
practices should suffice
API / WebService
Tests
• Against deployed Service(s)
• Dependent services may be
stubbed / mocked for
better orchestration
Yes, but at service level
Medium
• From a setup / mocking
dependencies perspective
e2e / UI Tests
• Against deployed
environment
• All integrations in place
• Data / config setup may be
necessary
• Some systems may be
stubbed
Yes, but may not be a
good strategy (cost Vs
value)
High
• Need full environment setup
• Keeping environment “pure” and
“isolated” from use by tests or
humans for tracing to work
• e2e tests may trigger only a small
set of code paths compared to
Unit & API tests
Value
Easy
Feasibility
Difficult
50. Unit (xUnit / JavaScript)
Manual / Exploratory
Web Service
UI / e2e
@BagmarAnand
Test Pyramid
51. Unit (xUnit / JavaScript)
Manual / Exploratory
Web Service
UI / e2e
@BagmarAnand
Measuring Quality from each Test Type
Feature Coverage
User Experience / Usability
Accessibility
Feature Coverage
Visual Coverage
Accessibility
Product-deployment & setup time
Test Execution time
Code Coverage
Contract Tests
Load / Performance / Security
API / Service Deployment & Setup time
Execution time
Code Coverage
Static Code Analysis
Execution time
52. In the rest of this session
we will focus on Feature
Coverage for e2e / UI Tests
@BagmarAnand
54. • e2e / UI tests are typically user scenarios that cut across
many features of the product
• High-impact / High-risk features should be tested more
than low-impact / low-risk features
* Risk / Impact is relative to business goals / user functionality
@BagmarAnand
Why should we capture Feature Coverage?
55. The ability to know which features are
covered, and how many times, as part of
execution of the e2e / UI Tests!
@BagmarAnand
Feature Coverage - Objective
56. Visualizing the feature coverage can give –
• confidence of the automated e2e tests,
• provide data where to focus next
@BagmarAnand
Feature Coverage – how will it help?
57. • Add appropriate ‘tags’ to automated e2e / UI tests
• Gherkin-based tools (ex: Cucumber, Karate, etc.) make it easy to add tags
• Test runners have different ways to provide custom Annotations
• Ex: TestNG, Junit, etc.
@BagmarAnand
Capturing Feature Coverage
58. • Approach #1: Update your reports to include ‘tags’ for the tests
• Create visualization based on the ’tags’ added in test reports
• Ex: Cucumber-reporting gives you ‘tag’ analysis out-of-the-box
• Uses json report files as input to create report
• Approach #2: Benchmark tag data (if #1 is not feasible)
• Update test framework to “log” all tags & and its test name to a common csv
file
• Post test execution, open csv file in a spreadsheet, and create chart to
visualize Feature Coverage
@BagmarAnand
Visualizing Feature Coverage
62. • Identify the correct set of tags to be used, for each type of test
• Tags can be based on features, components of your product
• Watch out for “tag-hell”
@BagmarAnand
Feature Coverage – Tips
63. Tracking feature coverage from your api / functional UI (e2e) tests
https://essenceoftesting.blogspot.com/2020/03/tracking-functional-coverage.html
Sample repo using cucumber-reports with karate:
https://github.com/anandbagmar/karate-sample
Cucumber-reporting
https://github.com/damianszczepanik/cucumber-reporting
@BagmarAnand
References