Introduction to automated visual testing
•Download as PPTX, PDF•
3 likes•1,641 views
This document discusses automating visual software testing. It begins by explaining that visual testing is used to verify a graphical user interface appears correctly. It then discusses why visual testing should be automated, such as the large test matrix of browsers, devices, resolutions etc. that is too big to manually test. It also notes that mobile app quality is critical given limitations on rolling back changes. The document outlines how automated visual testing works by taking screenshots, comparing them to baselines, and reporting differences. It discusses various tools and techniques used for image comparison, visual testing frameworks, and integrating tests into development and deployment pipelines. In general, it promotes automating visual testing for regression testing, localization, accessibility, and more throughout the development and deployment process
Report
Share
Introduction to automated visual testing
- 1. Automating visual software testing Adam Carmi Co-Founder and CTO Applitools INTRODUCTION TO AUTOMATED VISUAL TESTING
- 2. Automating visual software testing YOU CAN AND SHOULD AUTOMATE YOUR VISUAL TESTS!
- 3. Automating visual software testing
- 4. Automating visual software testing AGENDA • Why automated visual testing? • Tools & Technology • Where does it fit? • Q & A
- 5. Automating visual software testing WHAT IS VISUAL TESTING? A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
- 6. Automating visual software testing
- 7. Automating visual software testing
- 8. Automating visual software testing
- 9. Automating visual software testing WHY AUTOMATE? The test matrix is too big to cover manually • Web browsers • Devices • Operating systems • Screen resolutions • Responsive design • L10n • 3rd Party upgrades
- 10. Automating visual software testing WHY AUTOMATE? Mobile apps quality is critical • Harder to roll back changes • Can’t push daily • Updates take battery and data • Updates are not mandatory • Higher quality bar
- 11. Automating visual software testing WHY AUTOMATE? Release cycles keep getting shorter
- 12. Automating visual software testing AGENDA • Why automated visual testing? • Tools & Technology • Where does it fit? • Q & A
- 13. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
- 14. Automating visual software testing DEMOHELLO WORLD
- 15. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
- 16. Automating visual software testing
- 17. Automating visual software testing
- 18. Automating visual software testing
- 19. Automating visual software testing
- 20. Automating visual software testing
- 21. Automating visual software testing AND MORE… • 1 pixel offsets in element positioning • Dynamic content • Moving elements • Images of different size • Performance
- 22. Automating visual software testing
- 23. Automating visual software testing IMAGE COMPARISON ENGINES
- 24. Automating visual software testing IMAGE MAGICK • A command line tool for bitmap processing • Fuzzing is used to eliminate slight color differences • An error ratio is usually used to determine a match $ compare –metric AE –fuzz 5% img1.png img2.png diff.png 2246
- 25. Automating visual software testing THE JAVASCRIPT ENGINES • Resemble.js Pixel by pixel + error ratio + anti-aliasing http://huddle.github.io/Resemble.js • Blink-Diff Pixel by pixel + error ratio + anti-aliasing + perceptual color distance computation https://github.com/yahoo/blink-diff • Looks-Same Pixel by pixel + perceptual color distance computation + ignore Caret https://github.com/gemini-testing/looks-same
- 26. Automating visual software testing APPLITOOLS EYES • Handles anti-aliasing, pixel offsets, color similarity, and image scaling • Dynamic and moving content • Compare images of different sizes • No error ratio configuration required • Validates full UI pages • Fast! • Supports layout / structure matching
- 27. Automating visual software testing DEMOVISUAL LAYOUT MATCHING
- 28. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
- 29. Automating visual software testing QUICK FEEDBACK TOOLS MOTIVATION • Get fast feedback on code changes • Run tests locally in the background CONS • Partial coverage • Chrome, IE? • Limited navigation • Stale browser versions SETUP • Render screenshots with a headless browser • PhatomJS, SlimerJS • Configuration file driven tests
- 30. Automating visual software testing VISUAL COVERAGE TOOLS MOTIVATION • Verify that the app / site looks right in all real execution environments • Test as many UI states as possible • Execute many tests in parallel CONS • Requires test infrastructure • Implement and maintain test code SETUP • Render screenshots using real browsers on a variety of operating systems and devices in parallel • WebDriver, Grid • Code driven tests • WebDriver, DSL
- 31. Automating visual software testing SAMPLE TOOLS CODE / SCRIPT ● Needle (Python WD) ● Gemini (JS DSL) ● Selenium Visual Diff (Java WD) ● VisualCeption (PHP CodeCeption) ● Shoov (JS WebDriverIO) ● Vizregress (.NET WD) ● Rspec Page Regression (Capibara) ● Applitools Eyes (Appium, XCUI, Espresso) ● PhantomCSS (JS CasperJS) ● Specter (JS DSL) ● FBSnapshotTestCase (XCTest) CONFIGURATION ● Viff ● CSS Visual Test ● Wraith ● Dpdxt ● Grunt PhotoBox ● Snap And Compare ● BackstopJS ● CSSCritic ● Kobold ● DiffCop ● SUCCSS
- 32. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
- 33. Automating visual software testing REPORT DIFFERENCES As files on the file system
- 34. Automating visual software testing REPORT DIFFERENCES As a gallery
- 35. Automating visual software testing REPORT DIFFERENCES In your CI build report
- 36. Automating visual software testing HOW DOES IT WORK? Drive the AUT and take screenshots Compare screenshots with baseline images Report differences Update the baseline
- 37. Automating visual software testing UPDATE THE BASELINE Gemini
- 38. Automating visual software testing UI VERSION CONTROL • Keep CI tests green • Multiple app versions • Revert to previous baselines • A visual change log (documentation) 1 1 1 2 1 2 3 3 2 4 4
- 39. Automating visual software testing
- 40. Automating visual software testing DEMOAUTOMATED MAINTENANCE
- 41. Automating visual software testing AGENDA • Why automated visual testing? • Tools & Technology • Where does it fit? • Q & A
- 42. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Before committing / merging code: • See how UI changes appear in other execution environments • Make sure there are no unintended changes
- 43. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Visual testing of frontend components by frontend developers
- 44. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Regression, Localization, Accessibility, Cross browser/device, responsive design testing and more… All team members can review and maintain A collaboration tool
- 45. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION Validate your staging deployment using your production deployment as a baseline
- 46. Automating visual software testing WHERE DOES IT FIT? CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION No missing resources in production No corruption due to 3rd party data, upgrades, hackers, etc.
- 47. Automating visual software testing Adam Carmi (@carmiadam) Co-Founder and CTO at Applitools QUESTIONS?
Editor's Notes
- Components, Code review, Developers, Designers
- Components, Code review, Developers, Designers