Our Snapshot Test Journey
What Is a Snapshot Test?
A snapshot test is a type of test that compares the screen capture obtained after running a test with a reference screen capture to check for any changes and provides these changes to us in a diff image.
We write snapshot tests using mock data, thus eliminating the problem of data corruption. We can capture even the slightest pixel changes in the UI.
We run these tests in CI (Continuous Integration). Snapshot tests have Slack integration, and they run daily every morning and for every merge request (MR), displaying the culprits in the Slack channel. This is an important mechanism for quick feedback and early detection of errors.
In this article, I will explain how we have improved our snapshot test processes. For detailed technical information about snapshot tests, you can review the article by Atakan Karslı and Beyza İnce. (mediumlink-1, mediumlink-2)
How Do We Capture Error Images In Snapshot Tests?
As shown in the screenshot below, we have a reference image. Afterwards, our test runs and produces a result different from this reference image, creating a failure image. By comparing these results, we generate a diff image that allows us to see our errors more clearly.
If the screen capture obtained when the test runs differs from the reference image, then our test fails.
If the screen capture obtained when the test runs matches the reference image, then our test succeeds.
https://medium.com/trendyol-tech/automated-visual-testing-with-snapshots-part-1-ee9c5cf58cca
How Do We Update The Reference Images?
Our fails are dropped into the Slack channel, and if we want to update the fails, we can do so by using the update button in Slack, thereby updating the reference value.
Why Did We Decide To Write Snapshot Tests?
Problem: It was very difficult to find live data in the environment and running manual tests was very costly.
Solution: We automated the tests using snapshot tests and mock data, saving us from the manual process.
We decided to write snapshot tests during the Black Friday period. This decision was prompted by a UI glitch at the bottom of the product detail page during Black Friday, specifically in the price area. Even though we were already checking this area with regression tests, the problem wasn’t about missing parameters; it was about a UI glitch that occurred even when the parameters were present. You can see the bug in the screenshot below. This area is crucial as it represents the most important aspect of the product detail page — ensuring the price is readable. At that time, there were 48 cases in this small area, and manually checking each case with separate data was a very difficult process. So, we decided to utilize snapshot tests for these area checks. With snapshot tests, we write with mock data, so we wouldn’t constantly need to find new data, and even if we did, we wouldn’t have to constantly check if the data had changed.
Component-Based Snapshot Test
Problem: The inconsistency we experienced in swipe tests.
Due to the necessity for extensive downward swiping on the product detail page (PDP), even a slight deviation in snapshots could lead to significant differences. With animated scrolling, we were able to scroll to different locations. As illustrated in the example image below;
Solution: Component Based Snapshot Test
We thought about how to overcome the inconsistency we experienced in swipe tests and decided to write tests that would compare components independently of the swipe action. Our goal is to capture only the title, footer, and the screen capture of the component itself, regardless of the page. This way, we can make accurate comparisons regardless of the page’s position.
In the example below, we noticed a 1-pixel deviation in the description area of the product cards, and our component-based snapshot tests successfully detected this deviation. By eliminating the inconsistency caused by animated scrolling, we were able to identify real UI errors in our tests.
Rather than taking screenshots of the entire page for component-based snapshot testing, we define specific elements as XCUIElements (like headers, cells, and footers or like header and cell) and capture their screenshots individually. We then stack these screenshots vertically. For example, if a component includes a header, cell, and footer, we capture separate screenshots of each and arrange them one below the other. This approach lets us focus on capturing screenshots of only the elements we’re interested in, without including the entire page.
func combineScreenshots(screenshots: [UIImage]) -> UIImage? {
guard let firstScreenshot = screenshots.first else { return nil }
let width = firstScreenshot.size.width
let totalHeight = screenshots.reduce(0.0, { $0 + $1.size.height })
let combinedSize = CGSize(width: width, height: totalHeight)
UIGraphicsBeginImageContext(combinedSize)
var yPosition: CGFloat = 0
screenshots.forEach {
$0.draw(at: CGPoint(x: 0, y: yPosition))
yPosition += $0.size.height
}Disadvantages Of Component-Based Snapshot Tests
We encountered some disadvantages with component-based snapshot tests. These include:
1. Inability to test two different components together since we only compare the screen captures of components.
2. Inability to control dividers and spaces between components.
While component-based snapshot tests addressed the issues we faced with animated scrolling, we still needed to run snapshot tests for the entire page due to the above two reasons. We discussed this issue with our developers and aimed to find a solution.
Scroll Helper
Problem: The scrolling taking 50 seconds for a single scroll, all tests running for over an hour, and tests failing due to scrolling to the wrong place.
Solution: Scroll helper.
When the UI test app runner launched the application, it caused errors in snapshot tests because it scrolled manually like an end user. We developed a solution to ensure scrolling stops at the same point each time. For this purpose, we created a structure called “scroll helper”. Using the scroll helper, within the test function, we define a model specifying the component to scroll, along with its accessibility identifier, and whether to scroll on the x or y axis, and optionally, the padding amount.
At the start of the test, an “automation” file is created in the simulator, and the UI test app writes this information to the file. Both the UI test app and our own application monitor this file. Our application decodes the defined model and scrolls through the code until it finds the relevant component within the collection view based on the specified attributes. Once scrolling is complete, it writes the result model back to the simulator. The UI test app decodes this result and either completes the test if successful or provides an error message if not.
This method involves writing two types of models to the created file in the simulator, and both applications perform necessary actions based on their ability to decode these models. Thus, instead of manually scrolling in the UI test app, we efficiently scroll to the same point each time by adjusting the collection view content offset.
- With the scroll helper, we’ve eliminated the inconsistency in scrolling.
- The scroll helper has provided us with an incredible 45% time savings.
Throughout our journey of resolving issues in our snapshot tests, we’ve made progress by addressing challenges as a team and reaching our desired destination. We aim to continue this journey by further enhancing and developing this process.
I would like to express my thanks to Göktuğ Başaran and Mine ÖZ for their contributions to my Medium article and to İnanç Er for their support throughout the writing process. I want to thank my dear buddy Göktuğ Başaran once again, who never knew how to give up throughout the process and with whom I faced all the challenges together.
It was a great teamwork.
Join Us
Be a part of something great! Trendyol is currently hiring. Visit the pages below for more information and to apply.
