iOS Parallel Automation: run faster than fast — Viktar Karanevich — SeleniumCamp 2017
•
0 likes•1,186 views
iOS Automation rapidly changed due to the evolution of Apple automation frameworks and Calabash community support. One year ago, we parallelised our iOS testing with multiple desktops – a large improvement in performance. Here is an account of the subsequent changes, including the removal of multiple desktops and the adoption of Apple’s new XCTest framework. The automation community (Appium, Calabash etc.) faced a new challenge: how could we create iOS Automation Infrastructure with the new XCTest Framework? I adopted Facebook solution (FBSimCtl and WebAgentDriver) into the multi-simulators approach. In this talk, I will discuss the full evolution path to Facebash Approach based on multi-simulators, Calabash server and Facebook Web Driver Agent.
Report
Share
iOS Parallel Automation: run faster than fast — Viktar Karanevich — SeleniumCamp 2017
- 1. iOS Parallel Automation: Run faster than fast! Viktar Karanevich
- 2. Content: I. History of iOS Automation Frameworks (5 min) II. Path to Current Approach at Badoo (10 min) III. Summary (5 min) IV. Movie: Test Run (3 min)
- 3. self.introduce => { name: "Viktar Karanevich", title: ["SSTAE", "SSDET", "SQAAE"], languages: ["Java", "Ruby"], experience: "iOS Automation 7+ years" }
- 4. The original, largest and leading dating network: 330M+ registered users 800+ automated tests on iOS platform 50M+ downloads on iOS
- 7. Jun 2010 UI Automation Sep 2014 UI Automation xcrun simctl Apple "xcrun simctl" is introduced in xCode 6 It is command line utility to control the Simulator (create, delete, erase, install, uninstall, ...).
- 8. Jun 2010 UI Automation Sep 2015 UI Automation xcrun simctl UI Testing Apple UI Testing is introduced as replacement for UIAutomation in xCode 7 Sep 2014 UI Automation xcrun simctl
- 9. Jun 2010 UI Automation Sep 2016 UI Testing xcrun simctl Apple Sep 2015 UI Automation xcrun simctl UI Testing Sep 2014 UI Automation xcrun simctl UI Automation removed in xCode 8
- 11. Jun 2010 UI Automation Sep 2014 xcrun simctl Sep 2016 UI Testing Sep 2016 UI Testing Apple Calabash Nov 2009 UI Spec
- 12. Calabash Nov 2009 UI Spec Feb 2011 Frank
- 13. Calabash Nov 2009 UI Spec Feb 2011 Frank Jan 2012 Calabash Step Definitions Ruby Client Library Features Developer host Calabash server Simulator Your app UIQuery Touch Actions
- 14. Calabash Apr 2013 Calabash + UIAutomation Link: https://github.com/calabash/run_loop Step Definitions Ruby Client Library Features Developer host Calabash server Simulator Your app UIQuery Run loop: UIAutomation
- 15. Calabash Sep 2016 Calabash + DeviceAgent Link: https://github.com/calabash/run_loop/ tree/develop/lib/run_loop/device_agent Developer host Simulator Step Definitions Ruby Client Library XCTest Runner Features Calabash server Your app UIQuery Device Agent
- 16. Calabash Nov 2009 UI Spec Feb 2011 Frank Jan 2012 Calabash iOSAutomation Frameworks Apr 2013 Calabash + UIAutomation Sep 2016 Calabash + DeviceAgent Jun 2010 UI Automation Sep 2016 UI Testing xcrun simctl Sep 2015 UI Automation xcrun simctl UI Testing Sep 2014 UI Automation xcrun simctl Apple 3 years 1 year
- 17. Calabash Nov 2009 UI Spec Feb 2011 Frank Jan 2012 Calabash iOSAutomation Frameworks Apr 2013 Calabash + UIAutomation Sep 2016 Calabash + DeviceAgent Jun 2010 UI Automation Sep 2016 UI Testing xcrun simctl Sep 2015 UI Automation xcrun simctl UI Testing Sep 2014 UI Automation xcrun simctl Apple 3 years 1 year
- 18. II. iOS Automation at Badoo Approach ~80 flaky tests 80 data fail tests
- 19. II. Before Parallel Run Test Suite Run Nightly Cucumber Report Nightmare! 15h total time 1 simulator 80 data ~500 tests
- 20. II. Before Parallel Run Test Suite Run Nightly Cucumber Report Nightmare! 80 data fail tests 1simulator Key Issue: How to speed up Test Run? 15h total time ~500 tests 1 simulator
- 21. Let's Parallel!
- 22. Let's Parallel! Blocker Issue: - Calabash does not support Parallel run on 1 desktop
- 24. https://techblog.badoo.com Tim's talk at Badoo: "Parallel Calabash Testing on iOS"
- 25. II. Test Result: multi-desktops ~4 hours regression suite run time 5 Desktops
- 26. II. Test Result: multi-desktops Mac PRO (24 cores) ~2 hours regression suite run time 10 Desktops
- 28. X Desktops: host limit reached
- 30. II. Clustering Hosts - Parallel run on multi (virtual) hosts - Mac Pro and hosts: test suite - ~1h 30 min
- 31. II. Clustering Hosts Issues: - multi-desktops start up is Flaky - simulators can get stuck
- 32. II. Clustering Hosts Is it possible to automate multi-simulators on one desktop?
- 33. III. Facebook: Web Driver Agent
- 34. Developer host Simulator Selenium Driver WDA Runner Your app FB UIA WDA Key Change: Reimplemented UIAutomation dependency Link: https://github.com/facebook/WebDriverAgent
- 35. Profit: - 7 simulators on single desktop - More stable! more efficient!
- 36. III. Full Migration to FB UIA WDA Issues: - all tests have calabash code
- 37. III. Full Migration to FB UIA WDA Issues: - all tests have calabash code - custom app backdoors
- 38. III. Full Migration to FB UIA WDA Issues: - all tests have calabash code - custom app backdoors - workarounds and other
- 39. III. Full Migration to FB UIA WDA Full Migration Costs: ~x hours Is it possible to do Fast Migration?
- 40. III. Calabash + Facebook Fast migration: Facebash! Step Definitions Ruby Client Library Features Developer host Calabash server Simulator Your app UIQuery Run loop: UIAutomation
- 41. III. Calabash + Facebook Fast migration: Facebash! Step Definitions Ruby Client Library Features Developer host Calabash server Simulator Your app UIQuery Run loop: UIAutomation Developer host Simulator Step Definitions Ruby Client Library WDA Runner Features Calabash server Your app UIQuery FB UIA WDA
- 42. IV. iOS 10 Update: from UIAutomation WDAto XCTest WDA Link: https://github.com/facebook/WebDriverAgent Developer host Simulator Selenium Driver Your app FB UIA WDA WDA Runner
- 43. IV. iOS 10 Update: from UIAutomation WDAto XCTest WDA Link: https://github.com/facebook/WebDriverAgent Developer host Simulator Selenium Driver Your app FB UIA WDA Developer host Simulator Your app FB XCTest WDA WDA Runner Selenium Driver FB simctl
- 44. IV. iOS Hybrid 2.0 iOS 10 update Developer host Simulator Step Definitions Ruby Client Library WDA Runner Features Calabash server Your app UIQuery FB UIA WDA
- 45. IV. iOS Hybrid 2.0 iOS 10 update Developer host Simulator Step Definitions Ruby Client Library WDA Runner Features Calabash server Your app UIQuery FB UIA WDA Developer host Simulator Step Definitions Ruby Client Library FB simctl Features Calabash server Your app UIQuery FB XCTest WDA
- 46. V. Summary: multi-desktops vs multi-simulators Test Suite: 500 tests Nightly Cucumber Report Happy morning! 80 data Mac PRO Test Suite Run Time 0 min 225 min 450 min 675 min 900 min 1 sim 10 sims 16 sims 75 min 120 min 900 min Multi Desktops Multi Simulators
- 47. Developer host Simulator Step Definitions Ruby Client Library FB simctl Features Calabash server Your app UIQuery FB XCTest WDA "Face"book + Cala"bash" = Facebash V. Summary: current approach
- 48. - easy to set up Parallel Run on 1 Desktop - keep using of backdoors - full access to internal AUT state - cross iOS app automation - iOS simulator control with FBSimCtl - FB WDA supports Selenium protocol - FB WDA independent of Calabash/Appium Facebook Calabash Calabash Facebook Facebook Facebook Facebook V. Summary: PROs
- 49. - support framework parts: calabash server, FB WDA, fbsimctl, ... - "poor" documentation - using of mixed locator strategies V. Summary: PROs and CONs
- 51. - Parallel Run in Headless Mode - Simulator Start/Shutdown/Reset optimization - Cold/Hot start iOS app with relogin - record video per the each test in Parallel Run V. TODO