Eradicate Flaky Tests - AppiumConf 2021

Slides from my talk on Measuring Consumer Quality - The Missing Feedback Loop - delivered at PSTC 2018

This document discusses patterns for test automation frameworks. It begins by introducing common patterns like page objects, business layer, and factories. It then provides examples of page object and business layer page object patterns. The document also discusses test data patterns and different ways to specify test data and locators. Finally, it outlines advantages of using patterns like reduced complexity, reusability, and maintenance. The key message is that the best pattern depends on the specific test automation context.

Here are the slides from my recent talks on "To Deploy or Not-To-Deploy - decide using TTA's Trend and Failure Analysis" More information available from my blog: http://essenceoftesting.blogspot.com/search/label/tta

Slides from my talk in Selenium Conference 2016, Bangalore Blog post: http://essenceoftesting.blogspot.com/2016/06/test-data-food-for-test-automation.html Abstract: Building a Test Automation Framework is easy - there are so many resources / guides / blogs / etc. available to help you get started and help solve the issues you get along the journey. Teams already building 1000s of tests of various types - UI, web service-based, integration, unit, etc. is a proof of that. However, building a "good" Test Automation Framework is not very easy. There are a lot of principles and practices you need to use, in the right context, with a good set of skills required to make the Test Automation Framework maintainable, scalable and reusable. In this talk, we will focus on one of the critical aspects and patterns in building the Test Automation framework - Test Data! We will look at different data patterns as options and techniques how to create, manage, use, reuse Test Data in a way to keep the tests running in an reliable and deterministic way. We will also discuss what questions to ask, what things to think about in selecting your approach for Test Data! This discussion will be applicable for any type of Test Automation (web / mobile / desktop), but, we will focus primarily on UI automation frameworks, ex. using Selenium. Video & blog post will be linked soon!

The Test Automation Pyramid is not a new concept. The top of the pyramid is our UI / end-2-end functional tests - which should cover the breadth of the product. What the functional tests cannot capture though, is the aspects of UX validations that can only be seen and in some cases, captured by the human eye. This is where the new buzzwords of AI & ML can truly help. In this session, we will explore why Visual Validation is an important cog in the wheel of Test Automation and also different tools and techniques that can help achieve this. We will also see a demo of Applitools Eyes - and how it can be a good option to close this gap in automation!

Slides explaining the Agile process, and how Test Automation plays a key role in enabling teams to be successful in this fast moving process.

Slides from my talk in Agile India 2015, Discuss Agile 2015 on "Enabling Continuous Delivery (CD) in Enterprises with Testing"

This talk is a brief overview of how we deal with UI test architecture on our project. Complex backend, SPA as a frontend, 20+ different brands and 100+ features for each, more and more code. We will discuss: – fakes and test doubles, approaches how to build and maintain them; – development patterns, which help you to make your architecture more simple, stable and readable; – tips and tricks – how to make your life with UI tests easier.

Creating automated functional tests of your web application with Selenium IDE is quick and easy, but leads to massive duplication in your test scripts. See why this duplication is a problem, and how to address it using a simple design pattern called the Page Object.

Slides from workshop conducted at ThoughtWorks, Pune in vodQA, on Sat, 27th August, 2016. Workshop Facilitators - Anand Bagmar, Smriti Tuteja, Pallipuspa Samal, Rohit Singhal, S Ramalingam, Shilpa G More information about vodQA, and this workshop can be found from the following links - https://essenceoftesting.blogspot.com/2016/08/vodqa-pune-less-talk-only-action-agenda.html https://essenceoftesting.blogspot.com/search/label/vodQA https://essenceoftesting.blogspot.com/search/label/waat

The document discusses using Applitools for visual AI testing. It provides an overview of Applitools, which allows testing user interfaces through visual assertions rather than traditional code-based assertions. This avoids issues like managing locators and adjusting assertions. The document then demonstrates how to set up and implement visual AI testing using Applitools in Java, including configuring Eyes, setting up tests, and executing visual checks on elements and full pages. It concludes that while visual AI testing has advantages, a hybrid approach combining it with traditional testing may be best to avoid false positives.

Slides from the workshop I conducted on "Client-side Performance Testing". Abstract of the workshop: In this workshop, we will see the different dimensions of Performance Testing and Performance Engineering, and focus on Client-side Performance Testing. Before we get to doing some Client-side Performance Testing activities, we will first understand how to look at client-side performance, and putting that in the context of the product under test. We will see, using a case study, the impact of caching on performance, the good & the bad! We will then experiment with some tools like WebPageTest and Page Speed to understand how to measure client-side performance. Lastly - just understanding the performance of the product is not sufficient. We will look at how to automate the testing for this activity - using WebPageTest (private instance setup), and experiment with yslow - as a low-cost, programatic alternative to WebPageTest.

Agile Methodology is not new. Many organisations / teams have already adopted Agile way of Software Development or are in the enablement journey for the same. What does this mean for Testing? There is no doubt that the Testing approach and mindset also needs to change to be in tune with the Agile Development methodology. Learn what does it mean to Test on Agile Projects. Also, learn how Test Automation approach needs to change for the team to be successful! Also, see the skills and capabilities required to make anyone in the Quality / Testing role add tremendous value to the team!

The key objectives of Organizations is to provide / derive value from the products / services they offer. To achieve this, they need to be able to deliver their offerings in the quickest time possible, and of good quality! In such a fast moving environment, CI (Continuous Integration) and CD (Continuous Delivery) are now a necessity and not a luxury! There are various practices that Organizations and Enterprises need to implement to enable CD. Testing (automation) is one of the important practices that needs to be setup correctly for CD to be successful. Testing in Organizations on the CD journey is tricky and requires a lot of discipline, rigor and hard work. In Enterprises, the Testing complexity and challenges increase exponentially. I am sharing my vision of the Test Vision and Strategy required to make successful the journey of an Enterprise on the path of implementing CD. Learn via a case study of an Enterprise, what does CD really means, and how and what Testing needs to be done in order to enable CD.

The document discusses test automation frameworks, applications, and challenges. It introduces topics around revisiting test automation approaches, evaluating the need for AI/ML in testing, ensuring a precise testing approach, and comparing licensed versus open source test automation solutions. The presentation aims to have an open discussion around improving test automation through meticulous planning.

Design patters exist for years in software development. Some developers love them, some think they are useless. But design patters has very clear goals: describe common solutions for common problems, create shared language for community, improve understanding and reuse of existing approaches. Test automation has its own set of problems, so there is a set of helpful design patterns for this area. In this talk I will run through all known patterns and describe them in details with several practical samples.

A session where we learn about the clues to keep in mind to not fall into a maintenance overhead autoamtion

Slides from my talk on "Techniques to Eradicate Flaky Tests" at Testing Talks - The Reunion, Melbourne, Australia on 20th Oct 2022

Streamlining End-to-End Testing Automation with Azure DevOps Build & Release Pipelines Automating end-to-end (e2e) test for Android and iOS native apps, and web apps, within Azure build and release pipelines, poses several challenges. This session dives into the key challenges and the repeatable solutions implemented across multiple teams at a leading Indian telecom disruptor, renowned for its affordable 4G/5G services, digital platforms, and broadband connectivity. Challenge #1. Ensuring Test Environment Consistency: Establishing a standardized test execution environment across hundreds of Azure DevOps agents is crucial for achieving dependable testing results. This uniformity must seamlessly span from Build pipelines to various stages of the Release pipeline. Challenge #2. Coordinated Test Execution Across Environments: Executing distinct subsets of tests using the same automation framework across diverse environments, such as the build pipeline and specific stages of the Release Pipeline, demands flexible and cohesive approaches. Challenge #3. Testing on Linux-based Azure DevOps Agents: Conducting tests, particularly for web and native apps, on Azure DevOps Linux agents lacking browser or device connectivity presents specific challenges in attaining thorough testing coverage. This session delves into how these challenges were addressed through: 1. Automate the setup of essential dependencies to ensure a consistent testing environment. 2. Create standardized templates for executing API tests, API workflow tests, and end-to-end tests in the Build pipeline, streamlining the testing process. 3. Implement task groups in Release pipeline stages to facilitate the execution of tests, ensuring consistency and efficiency across deployment phases. 4. Deploy browsers within Docker containers for web application testing, enhancing portability and scalability of testing environments. 5. Leverage diverse device farms dedicated to Android, iOS, and browser testing to cover a wide range of platforms and devices. 6. Integrate AI technology, such as Applitools Visual AI and Ultrafast Grid, to automate test execution and validation, improving accuracy and efficiency. 7. Utilize AI/ML-powered central test automation reporting server through platforms like reportportal.io, providing consolidated and real-time insights into test performance and issues. These solutions not only facilitate comprehensive testing across platforms but also promote the principles of shift-left testing, enabling early feedback, implementing quality gates, and ensuring repeatability. By adopting these techniques, teams can effectively automate and execute tests, accelerating software delivery while upholding high-quality standards across Android, iOS, and web applications.

https://confengine.com/conferences/selenium-conf-2022/proposal/17141/changing-tyres-in-a-moving-car-making-functional-test-automation-effective In my experience, I have seen teams struggle with more than one, and in many cases, all of the above statements. On retrospection, I have realized, that most of these challenges are because of a combination of the following issues: Lack of holistic vision, understanding, skills, and capabilities for taking on this seemingly easy activity of functional test automation Lack of time for design, implementation, and maintenance of the automated test code In this session, I will share, with examples, the features & capabilities that are not used right in Test Automation implementation resulting in you answering “NO” to the above statements. These are anti-patterns of functional automation implementation and have a negative impact on the quality of feedback from your automated tests. More importantly, I will share approaches and solutions of how you can avoid these anti-patterns and evolve to answer “YES” to each of the above statements.

Believe it or not - 85% of mobile apps are removed after first usage! In this presentation - given at the APM Meetup in Singapore in April 2015 - I talked about the challenges, best practices and especially metrics to avoid this situation. Key Points of the Presentation The two key trends "Internet of Things" and "DevOps" play a big role in our life when we talk about User Experience and especially mobile user experience. In this presentation I tell you what metrics to use to make sure you deliver your ideas faster to your mobile end users but also ensuring the right quality and user experience so that your users stay loyal and dont delete the mobile app after first usage.

The document discusses strategies for making mobile tests less flaky in 6 steps: 1. Use explicit waits instead of hardcoded waits 2. Use reliable locator strategies like accessibility IDs instead of XPath 3. Make tests faster by launching specific app screens or using deep links instead of full user flows 4. Mock API calls to avoid failures from changing backend services 5. Debug test failures by examining logs and error messages 6. Follow best practices like storing results and getting involved in app development to ensure testability

This presentation shares expertise and insight on Automated Low Level Requirements Testing for DO-178C: • DO-178C SW Verification Process • Software Testing Activities • Software Testing Stages • Test Coverage Analysis • Software Testing Activities • Structural Coverage Analysis • Requirements Based Test Selection • Manual Test Generation • So…How to Automate? • Generation from Requirements • Generation from Code • AutoTest & Trace for DO-178C • AutoTest Generation • AutoTest Process • AutoTest DO-178C Use Cases For more information, please refer to: https://www.qa-systems.com/

The document discusses AI tools for software testing such as ChatGPT, Github Copilot, and Applitools Visual AI. It provides an overview of each tool and how they can help with testing tasks like test automation, debugging, and handling dynamic content. The document also covers potential challenges with AI like data privacy issues and tools having superficial knowledge. It emphasizes that AI should be used as an assistance to humans rather than replacing them and that finding the right balance and application of tools is important.

Heard about Automated Acceptance Testing but not sure what to make of it. Check out this brief intro and demo of an automated acceptance test using Selenium Webdriver, Jasmine and Protractor.

Learn how to accelerate your test execution pipeline with test frameworks, automation and parallel testing from our very own Bria Grangard, Product Marketing Manager.

Vskills certification for The Grinder Testing Professional assesses the candidate as per the company’s need for load testing web applications. The certification tests the candidates on various areas in agents, workers, properties file, logging, console, TCPProxy, scripts, Jython, Clojure, instrumentation, script gallery, plug-ins, statistics, SSL and garbage collection.

The promise of DevOps is that we can push new ideas out to market faster while avoiding delivering serious defects into production. Andreas Grabner explains that testers are no longer measured by the number of defect reports they enter, nor are developers measured by the lines of code they write. As a team, you are measured by how fast you can deploy high quality functionality to the end user. Achieving this goal requires testers to increase their skills. It’s all about finding solutions—not just problems. Testers must transition from reporting “app crashes” to providing details such as “memory leak caused by bad cache implementation.” Instead of reporting “it’s slow,” testers must discover “wrong hibernate configuration causes too much traffic from the database.” Using three real-life examples, Andreas illustrates what it takes for testing teams to become part of the DevOps transformation—bringing more value to the entire organization.

Everyone wants to make quick releases, but the look-and-feel UX validation is a manual, slow, and error-prone activity. Any UX-related issues propping up cause huge brand-value and revenue loss, may lead to social-trolling, and, even worse, dilute your user base. This is an area where AI & ML can help. In this hands-on workshop, using examples, we will explore: the importance of automated visual validation how an AI-powered tool, Applitools Visual AI, can solve this problem. Integrate Applitools Visual AI in your Selenium-Java automation and learn by practice: The different AI algorithms various Applitools capabilities and features scale your automation using the Applitools Ultrafast Grid.

This document explains Property Based Testing – a method beyond unit testing to help detect defects in codes during the development phase.

(Mike Graham + Dan Carroll, Comcast) Kafka Summit SF 2018 Comcast manages over 2 million miles of fiber and coax, and over 40 million in home devices. This “outside plant” is subject to adverse conditions from severe weather to power grid outages to construction-related disruptions. Maintaining the health of this large and important infrastructure requires a distributed, scalable, reliable and fast information system capable of real-time processing and rapid analysis and response. Using Apache Kafka and the Kafka Streams Processor API, Comcast built an innovative new system for monitoring, problem analysis, metrics reporting and action response for the outside plant. In this talk, you’ll learn how topic partitions, state stores, key mapping, source and sink topics and processors from the Kafka Streams Processor API work together to build a powerful dynamic system. We will dive into the details about the inner workings of the state store—how it is backed by a Kafka “changelog” topic, how it is scaled horizontally by partition and how the instances are rebuilt on startup or on processor failure. We will discuss how these state stores essentially become like materialized views in a SQL database but are updated incrementally as data flows through the system, and how this allows the developers to maintain the data in the optimal structures for performing the processing. The best part is that the data is readily available when needed by the processors. You will see how a REST API using Kafka Streams “interactive queries” can be used to retrieve the data in the state stores. We will explore the deployment and monitoring mechanisms used to deliver this system as a set of independently deployed components.

You know it’s the right thing to automate testing, workflows, and tedious processes. Automation saves you time, reduces errors, and allows your team to work more quickly. But where do you get started? This talk covers five ways to automate API testing, which can then be applied to any API workflows. Learn about testing as part of your CI/CD pipeline, uptime monitoring, and other APIOps practices.

The document discusses various options for creating plug-ins or extensions for the Aware IM software system. It describes five main options: 1. Database extensions to improve efficiency of CRUD operations by calling stored procedures. 2. Combining Aware IM with Java and a database, calling stored procedures from Java and implementing additional processing rules. 3. Implementing custom business logic in Java by passing data between Aware IM and a Java plugin. 4. Consuming REST APIs from within Aware IM by calling a Java plugin that builds requests, processes responses, and returns objects to Aware IM. 5. Enhancing Aware IM features using JavaScript APIs through the Advanced Script configuration.

As release velocity increases, teams are finding innovative ways to detect and resolve performance issues earlier in the development cycle. This session will explore how to integrate performance testing into a development lifecycle and why the practice is gaining popularity. Learn how to create a balanced test strategy that matches test type, coverage, environment, and defect target. Scale to deliver high-performing applications. Through continuous integration platforms, performance testing tools, and AppDynamics, see how automated performance testing can reduce time-to-market while increasing overall quality. Key takeaways: o How to get started with performance test automation o How to detect and resolve performance issues earlier in the development lifecycle with AppDynamics o How to maximize the quality and value of your performance test strategy Brad Stoner Senior Sales Engineer, AppDynamics Resources: KonaKart - http://www.konakart.com/ Adminer - https://www.adminer.org/ Jenkins - https://jenkins.io/ NeoLoad - http://www.neotys.com/neoload/overview AppDynamics - https://www.appdynamics.com/free-trial/ GitHub - https://github.com/ Docker Hub - https://hub.docker.com/

Strong software testing process is a key to project success. This presentation helps to understand better how to improve the process by automating routine API endpoints testing.

Designing a perfect, failproof software delivery system is impossible. Failures will happen. What's more important is the speed and reliability of your recovery. Shipping with feature flags helps you limit your risk in the first place and recover faster when the unexpected happens. Today, with Optimizely Agent, companies that build their apps using service-oriented architectures can achieve production-scale faster with their feature delivery and experimentation platform.

This document provides an agenda and overview for a workshop on testing, automation, and reporting with Postman. The workshop will cover testing concepts and practical exercises, advanced testing techniques like dynamic variables and Chai assertions, automation with the Collection Runner and Newman, and creating reports. Speakers Trent McCann and Danny Dainton will present on prerequisites, testing modules, breaks, automation, and workshop wrap-up.

This document discusses teswiz, an open source framework for automating real-user scenarios across multiple apps, devices, users, and platforms. It can simulate user actions and behavior to test web, mobile web, Android, iOS, Windows, MacOS and Linux applications. Teswiz uses Cucumber, Appium, Selenium and other tools and supports features like multi-device testing, visual testing with Applitools, and generating reports with ReportPortal. The document provides instructions on getting started with teswiz and links to its GitHub page.

The document discusses visual validation testing as a missing piece of the automation testing pyramid. Visual testing is important but challenging as it is typically done manually, which is tedious, error-prone, and difficult to scale. Automating visual testing can help by taking screenshots of expected user interfaces and comparing them to actual screenshots. However, automating also poses challenges around false positives/negatives, maintaining baselines, and accounting for product changes. The document promotes using artificial intelligence in visual test automation to help address these challenges and advocates for including visual testing in an organization's overall quality and automation strategy.

In this talk, we will talk about, and see examples of various types of patterns you can use for: - Build your Test Automation Framework - Test Data Management - Locators / IDs (for finding / interacting with elements in the browser / app) Using these patterns you will be able to build a good framework, that will help keep your tests running fast, and reliably in your CI / CD setup!

I had the privilege of delivering the opening keynote at the recent Future Of Testing event focused on Test Automation Frameworks on 30th September 2021. My topic was “The best test automation framework is …”. Here is the mind map I used in my keynote presentation. https://applitools.com/future-of-testing-frameworks-north-america-2021/

Slides from my talk on "Visual Validation - The missing tip of the automation pyramid", given at QA Symposium, Feb 2020

When one has fun at work, work becomes fun. However, daily pressures, metrics, KPIs, and what not, have dissolved the fun, and made work drudgery in various ways. This creates stress for individuals, in teams, and across teams, there is mistrust, unnecessary competition, blame, finger-pointing ….What better way to learn, and re-learn the basics of life, work, team-work - than to play a game, have fun, and correlate it with how life and work indeed should be treated as a game, and we should have fun in this journey. Only then can people truly succeed, and so can organisations.Here, we will play a game – “Collaboration - A Taboo!” – where you will –Re-learn collaboration techniques via a game!Learning applicable for individuals & teams, in small or big organisationsRe-live your childhood when playing this gameBe prepared for a twist which will leave you thinking!

Here are slides from my talk on "What, Why and How of Analytics Testing" at Selenium Conference, Berlin 2017. This talk focusses on Analytics related to Browser & Mobile Native apps, with impact on / of IoT (Internet of Things) and Big Data. See my blog for more details - https://essenceoftesting.blogspot.com/2017/10/analytics-forgotten-child.html

Slides from my 4-hour workshop on Client-Side Performance Testing conducted at Phoenix, AZ in STPCon 2017 (March). Workshop Takeaways: Understand difference between is Performance Testing and Performance Engineering. Hand’s on experience of some open-source tools to monitor, measure and automate Client-side Performance Testing. Examples / code walk-through of some ways to automate Client-side Performance Testing. See blog for more details - https://essenceoftesting.blogspot.com/2017/03/workshop-client-side-performance.html

Slides from my talk in Selenium Conference London 2016 about "Sharing (Less) Pain with Protractor & Selenium WebDriver" See blog for more information - http://essenceoftesting.blogspot.com/2016/11/shared-relatively-less-pain-of-using.html My blog: https://essenceoftesting.blogspot.com

Slides from my talk in Selenium Conference 2016 about "Sharing the pain with Protractor & Selenium WebDriver" See blog for more information - http://essenceoftesting.blogspot.com/2016/06/sharing-pain-using-protractor.html My blog: http://essenceoftesting.blogspot.com

This document discusses principles and practices for test automation. It covers topics like different levels of testing from unit to integration to UI, the importance of testing non-functional requirements, using a test pyramid approach with more unit and component tests than UI tests. It also discusses practices like continuous integration, test automation framework design considerations, sample framework architectures, and page object patterns. The overall goal is to provide guidance on building quality into software through effective test automation practices.