Responsive Imaging and Dynamic Media Done Right — Part I
What is responsive imaging, and why do we need it?
Today’s websites are experienced on a multitude of different devices.
Not only do screens vary in sizes, but they also vary in resolution, orientation, and aspect ratio.
Having different devices requires us to provide images in different resolutions and formats.
HTML5 gives some clues how this can be achieved. But the standard is quite complex and it’s easy to make errors. Also, HTML5 is a bit static, and it does not fully exploit the advanced rendering features you can use when you have AEM Dynamic Media.
Which is why we’re going to talk about responsive imaging and Dynamic Media, and how to make them work for you.
This is a four-part series:
- Part I: What is responsive imaging? Why we need it and what use cases it has. (This article)
- Part II: How can we use AEM Dynamic Media, to increase page loading performance?
- Part III: How can we implement Dynamic Media with HTML 5?
- Part IV: Beyond HTML 5: Shrinking the page load time with a custom image loader.
Responsive Layout and Responsive Images
Now we all know — at least to some extent — that websites adapt to various devices with responsive web design.
But what does “adapt” mean? Adapt to what? And how?
For this article, I want to divide “responsive adapting” in two parts:
- Responsive layout adapts … well …the layout of the page. What HTML elements to place where and how large to render each.
- Responsive imaging adapts the size and format of images used on a page.
Don’t trust “best practices” in over-simplified tutorials. There are a lot of misconceptions out there — and a fair amount of superstition, too.
While techniques for responsive layouts are well documented, my impression is that responsive imaging is still somewhat uncharted territory. I found dozens of tutorials, but on my personal journey to implement some of the “best practices”, I found a lot of misconceptions and a fair amount of superstition, too.
The field indeed is quite confusing. And different browsers behave differently. Even the same browser can show a slightly different behavior from one version to the next.
So, I had to do a good amount of experimentation to reach my personal conclusions.
Join me in my journey into the world of adaptive imaging. We’ll explore the “why adapt”, the “what to adapt” and the “how to adapt” in detail. I implemented adaptive imaging with AEM Dynamic Media. But most of the techniques described here can be implemented with other systems, too.
This article will be split in three parts:
Part I — Why and What: Use cases for responsive imaging.
Part II — How to render responsive images.
Part III — How to use and choose the proper rendition in a responsive layout.
You can find the demo code for my experiments on GitHub at https://github.com/ackoch/image-zoo.
Fork if you want to play around. But you can also passively follow this tutorial.
Responsive Web Design from a 100 ft view
As I said in the introduction, responsive web design consists of two parts:
- Responsive layout
- Responsive imaging
The goal of responsive imaging, which I will be focusing on here, is to deliver images that provide a balance between the resolution of an image and the time spent to transmit the image data over the network.
On a 1280 pixel-wide screen, a 768 pixel-wide image displayed at full width will look blurred. On the other hand, transmitting a 1280 pixel-wide image to a mobile phone that only has a 768 pixel-wide screen is a waste of bandwidth and the website will render slower.
Yes — this is a simplified example. Don’t worry. We’ll dig deeper later.
Even if your images are too large, the speed at which your page will render will depend on the connection (Cable, DSL, WIFI, LTE, G5, G4, ...) and the quality of the signal.
Therefore, when designing a web site, we must consider the primary audience, and where they are mostly accessing it.
For example, a B2B website is likely consumed using a desktop computer in an office, whereas a tourist guide for the old town of Lisbon is more likely to be experienced on a mobile device.
Page rendering speed is important for ranking in search engines, and each additional second a page renders increases the risk that a user abandons the site and does not convert.
However, in the first case, we can be a little bit more lenient with optimizations, as we expect our audience here to have a solid connection and a decent-sized screen.
In the second case, it is more important to deliver the page with as few resources as possible. An iPhone 12 has a resolution of 1080 x 2340 pixels. But you should ask yourself if you really want to max that out or if you want to trade image quality for a shorter transfer time.
I have conducted a set of experiments to figure out what impact the image resolution has on its size.
Two times the width and two times the height means four times the total number of pixels.
In theory, doubling the resolution of an image quadruples the number of pixels in the image (width x height = pixels). Two times the width and two times the height means four times the total number of pixels.
Now, the server does not transfer bare pixels. Instead, it compresses the image. So, there is no exact square growth in image size.
The graph above summarizes a few experiments I did:
- Varying the width of an image (and the height proportionally)
- Varying the compression rate on the square image
All experiments were conducted in a JPEG format. More modern WebP or AVIF images will be a tad smaller, but the growth in size is non-linear as well.
The key observations were:
- In default compression mode, doubling the resolution resulted in three times the image size.
- On higher resolutions the compression artifacts became less obvious, therefore on higher resolutions you might want to compress more aggressively.
Conclusion: It is important to reduce the file size of images as much as acceptable. Even minor changes can have a large effect on the page loading time.
We will now explore how to squeeze the maximum performance out of your site — at least with regards to image loading performance.
Use Cases: Not only size matters
By now I should have convinced you that we need to adapt the resolution and compression rate for each device the site could be viewed on. Unfortunately, the number of different devices and screens a website can be experienced on is growing each year.
The good news is, Dynamic Media is the perfect tool to … well … dynamically accommodate all types of screens. We’ll explore its capabilities (and a few limitations) later.
Resizing images is only one of many techniques used in responsive imaging. Don’t worry — you won’t have to implement all. But it’s worthwhile knowing some edge cases anyway.
What is Art Direction?
Note, in the following, I am deviating a bit from the standard terminology used in responsive imaging. A very good article on MDN uses the term “art direction” as an umbrella for all kinds of adaptive behaviour in responsive imaging.
I feel this requires some more differentiation, so let’s make it more tangible.
Art direction can be:
- Proportional scaling
- Cropping (manual, server-side, client-side, smart, dynamic, directed)
- Detailing / Zooming
Or a combination of any of the techniques above.
Proportional scaling
The most common use case: You resize the image proportionally to fit on smaller screens. This is also called down-sampling.
Cropping
You want to display an image with an aspect ratio (width/height) of 1:1 on a desktop screen but with a 2:1 ratio on a mobile device. Here you will have to cut away the extra pixels that do not fit in the new format. This can be achieved in several ways:
Manual cropping
Provide different images for different aspect ratios. These images will be produced up-front: for example, using Photoshop. Obviously, this results in a fair amount of manual work and more hassle to keep the independent assets organized in the DAM.
Client-side cropping
You transmit the full image and let the browser cut off the pixels that don’t fit into the format. This is easy to implement with the object-fit: cover attribute [1]. But it has two major disadvantages:
a) You transmit surplus pixels that you’ll discard in the browser. This unnecessarily increases the transfer-volume.
b) You don’t have control over what is cut-away. The browser will cut away the same number of pixels left and right (or top and bottom), effectively performing a “center crop”. In these cases, you might want select images that have the main motif in the center so it’s safe to crop the non-important pieces at the edges.
Smart cropping
Use AEM Dynamic Media’s Smart Cropping to automatically generate crops for you in any aspect ratio. This works remarkably well. DM’s Artificial Intelligence does an excellent job of finding what is interesting in a picture and what is not. If you do not agree with the results — you can adjust manually.
Smart cropping is non-destructive. You can always re-adjust a cropping later.
Smart cropping also does not generate new assets but additional “virtual” renditions. This makes the handling much easier. Also, you do not require desktop tools like Photoshop for cropping.
By “virtual” I mean, that Dynamic Media calculates cropping coordinates, only. These coordinates are applied when the image is requested. Dynamic Media creates and stores physical images.
Dynamic cropping
Static cropping means that we know in advance — when designing the page — which format to display at what screen size. Thus, we can pre-define a fixed number of aspect ratios and re-use them in our components. E.g., 1:1, 3:1, 2:1, 1:2, etc.
There are cases however, when static cropping not possible. Imagine a layout for a text-image component that requires the text to be 2/3 of the screen and the image 1/3 but it also requires the image to be exactly as high as the rendered text. Such cases result in an infinite number of aspect ratios.
Even Dynamic Media Smart Cropping cannot fully address this issue. Dynamic Media Smart Cropping requires pre-defined formats*. In this case we could use a combination of smart cropping to approximate the range of the target aspect ratios and rely on client-side cropping to make it pixel perfect.
This works well to some extent. Note though, that if the amount of text defines the height of component, you should limit the text length to a reasonable size.
Directed Dynamic Cropping
Dynamic cropping can be further improved by letting page editors define where the “center of gravity” of an image is. The browser (or Dynamic Media) could then use this information to crop from the right side. An uncomplicated way to express the center of gravity is by using a 9-field matrix from which the content editor can select.
Magnifying / Detailing
The aspect ratio on the desktop and mobile screen are the same. No cropping is required here. However, you may find that, when downscaling the image to fit on a mobile device, key details are lost.
You might want to highlight the most important content by zooming in discarding the less-important surrounding context. You can think of detailing as all-around-cropping.
Custom imaging
There might be edge cases, where cropping or detailing might not be sufficient for a mobile device, and you want to provide a completely different image. Consider carefully if you want to support such cases on your platform. It increases the effort required to maintain of your codebase, the asset library, the components, and the tracking of the images (ie. how often an image was displayed on a page).
Reminder: Cropping, detailing, and custom imaging are the terms that usually are simply subsumed under art direction, as they require a similar HTML markup [1]. We needed to differentiate further here because they require different implementation techniques in Dynamic Media.
Note: The most common use case is combining transformations: You will likely want to change the aspect ratio, optionally magnify and down-scale an image.
This is the end of Part I on use cases for responsive imaging.
Get Ready for Part II
Acknowledgements
Thanks to my dear colleagues Eryk Lagun and Rob Freeman for inspiration, proof reading and fact-checking.
References
[1] https://developer.mozilla.org/en-US/docs/Learn/HTML/Multimedia_and_embedding/Responsive_images
[5] https://github.com/ackoch/image-zoo/blob/master/README.md
[6] https://developer.mozilla.org/en-US/docs/Web/API/HTMLImageElement/srcset
