Android Compose Screenshot testing failing on pipeline with no visible differences

Question

I'm trying out the new Compose Screenshot Testing https://developer.android.com/studio/preview/compose-screenshot-testing.

When I create reference images locally, then run the tests to validate the UI against these images, it passes.

However, when I edit my Github actions and include screenshot testing in my workflow, then it fails.

Looking at the artifact that is produced, I can't see any visible differences despite the report highlighting what the differences are.

I made sure the composable that is being tested via screenshot testing was not changed at the time of generating the reference image + at the time my workflow is run.

I'm not sure what's going on.

It's possible that the reference image created locally doesn't match the image created in the pipeline for testing due to differences in the environment. But I'm not sure if this is true or not. Given these tests aren't run on a device that rules out any differences on an emulator/device.

Answer 1

I have run into this myself as well!

There's an open issue here: https://issuetracker.google.com/issues/348590914

There appear to be platform specific differences in anti-aliasing, which results in issues around fonts, shadows, and partially transparent layers - the differences can't really be seen unless you blow it up and compare pixel by pixel on a well-tuned monitor.

This shows up as diffs with subtle differences in font rendering of partially transparent colors.

So until there's a workaround, you have to use the same OS to both generate and validate your images.