[css-view-transitions-1] Capturing fragmented elements

[vmpstr]

View transitions specify that the image should be captured at the same size as in the document

Render element and its descendants, at the same size it appears in its node document, over an infinite transparent canvas, following the capture rendering characteristics.

It's unclear what to do if the element is fragmented. Is this supposed to capture the union of the fragment sizes/positions or something else?

I can't think of any good example where a developer would want to transition a fragmented element, so my proposal is to treat an element that is fragmented as invalid element to share in a view transition and skip the transition.

[flackr]

As a developer I would expect the union of the fragments to be captured. FYI this is also done by the element() function: https://jsbin.com/wukobal/edit?html,css,output

I think not doing this is just making users add or use a wrapping element which they wouldn't otherwise have had to use to e.g. transition from article content in columns.

[khushalsagar]

The definition for how to get the bounding box for a fragmented element is here, which does take a union of all fragments. This spec uses the border image areas of each fragment (since snapshots are clipped to the element's border box). In VT's case, that would be the ink overflow areas of each fragment.

[vmpstr]

I still don't think there are good cases where transitioning a fragmented element makes for a good visual transition. I understand that there's nothing to stop developers from trying, I just don't think that this is a use-case that is important.

There are also complexities with things like transforms where each fragment is transformed independently, which doesn't align with our model of "capture contents without transform, and then apply the transform to the corresponding group pseudo". Specifically, performing the steps in our model will not result in the same visual look as original fragments. There are also complexities with overflow and object-view-box calculations in these cases.

So in general, I'd still like us to say that you cannot transition a fragmented element. I don't have a strong preference between skipping the whole transition or omitting the fragmented element from participating. We can also consider a new break-inside value to ensure the element cannot fragment (TIL that break-inside: avoid still sometimes fragments)

[flackr]

For transformed fragments one way this could work is to bake the transform into the captured pixels and only apply any transforms from outside the fragmentation. For example, consider how this is captured with the element() function in Firefox:

https://jsbin.com/pozumog/edit?html,css,output

We could do the same except include in the ink overflow the content which flowed outside of the bounding box of the fragments.

[khushalsagar]

The Firefox implementation doesn't seem to align to the spec, it explicitly states that transforms on the element should be ignored: "If the referenced element has a transform applied to it or an ancestor, the transform must be ignored when rendering the element as an image."

View Transition spec handles transforms the same way, they are ignored when rendering the element's image: "If the referenced element has a transform applied to it (or its ancestors), then the transform is ignored".

This model makes sense for non-fragmented content since the element's screen space transform, which includes layout position, transform from ancestors and its local transform; is applied to the pseudo-element displaying it. We'll have to carve out many special cases for fragmented elements:

• Figure out what the border box size should be to decide what to size its ::view-transition-group. We've talked about unioning the rects of each fragment, but I'm not sure what these rects are relative to.
• Render the image with the local transform, as opposed to without it in the default case.
• Figure out what the image size should be. This is different from the non-fragmented case where we take the ink overflow rectangle. This size will be the union of transformed rects for each fragment with their overflow.
• Figure out what the object-view-box should be. In the default case its the value which places the border-box origin in the image at ::view-transition-group's pseudo's origin.

I'm not saying the above is not doable, but its a lot of complexity for an esoteric case that I don't think is worth doing. Though would be good to get another implementor's take on this: @emilio @dholbert.

Also @vmpstr, @bfgeek and @chrishtr who probably know a lot more complications with this case than me; and have expressed disallowing transitions on a fragmented element as the better option. Please clarify if I misunderstood.

[jakearchibald]

The element() spec does have specific things to say about fragmentation:

If the referenced element is broken across pages, the element is displayed as if the page content areas were joined flush in the pagination direction, with pages' edges corresponding to the initial containing block’s start edge aligned. Elements broken across lines or columns are just rendered with their decorated bounding box.

I'm not 100% sure what this means, particularly in the multi-column layouts where the column widths are not equal (which seems to be possible using negative margins, although that might be an implementation bug).

[jakearchibald]

Summary for CSSWG:

When an element is captured as part of a view transition, we ignore all ancestor & element transforms when capturing the image, and apply a transform to the group that positions the element in the correct place in relation to the snapshot root. So, if the difference between the old & new view is that the element has been rotated 90 degrees, by itself or a parent, you'll get a rotating transition, rather than just a cross-fade.

Elements can become fragmented, eg across columns. When transforms are applied to fragmented elements, the transform is applied to each fragment individually. This isn't compatible with the view transition model that wants to capture the element as a single image + a single transform.

It's unclear whether view transitions are desirable on elements that fragment. Particularly if the amount of fragmentation is different between the old and new views. Eg, going from fragmented across 2 columns, to fragmented across 3.

Prior art: The element() spec says:

If the referenced element is broken across pages, the element is displayed as if the page content areas were joined flush in the pagination direction, with pages' edges corresponding to the initial containing block’s start edge aligned. Elements broken across lines or columns are just rendered with their decorated bounding box.

I'm not 100% what that means. Firefox's implementation seems to capture the union of all fragment rects before transforms are applied, but captures the pixels with transforms applied (demo).

Option 1: Fragmentation breaks view transition constraints

Because it's complicated, and seemingly not desirable, make no-fragmentation a constraint.

When constraints are broken, transitions skip to the end (previous resolution).

Sub-option: Is the constraint broken if an element fragments, or if an element can fragment?

Previous resolution: A view-transition-name value that isn't none gives an element stacking context, grouping element, backdrop root, similar to non-1 opacity, as view transitions need these constraints.

Sub-option: Should a non-none view-transition-name prevent an element being fragmented? This would be in line with a previous resolution.

Option 2: The view transition only applies to the first fragment

This doesn't seem useful, but it's not-failing. It's also an easily-definable rule. You could argue that how transforms apply to fragments is similarly not-useful, but also not-failing.

Option 3: Capture union of fragments with transforms baked-in

This is taking inspiration from what Firefox does with element(), which may not be in line with the spec.

We capture the transform for the transition group, for fragmented elements, by:

1. Remove any transform from the element.
2. The quad is a union of fragment quads.
3. Figure out the transform needed to place the quad in the snapshot root, equivalent to its current position.
4. Reapply the element's transform.
5. Capture the image. Anything overflowing the quad is ink overflow.

Again, it isn't clear if this is useful, but it's not-failing. It's much more complex than option 2, but involves all fragments.

[css-meeting-bot]

The CSS Working Group just discussed [css-view-transitions-1] Capturing fragmented elements, and agreed to the following:

• RESOLVED: if an element if fragmented into more than one element, it does not participate in View Transitions (plus note that we want to do such things in the future)

The full IRC log of that discussion

<emeyer> Topic: [css-view-transitions-1] Capturing fragmented elements
<emeyer> github: https://github.com//issues/8339
<emeyer> JakeA: When an element is captured as part of a transition, we ignore all the transforms and then figure out what transforms we need to apply to place it correctly
<emeyer> …so if the transition is a 90 deg rotation, you get a rotating transition
<emeyer> …The columns case is particularly interesting because you can apply transitions across fragments, which transitions each fragment individually
<emeyer> …That’s not compatible with the View Transitions model, which we want to be a single image and single transform
<emeyer> …It’s unclear to us if View Transitions are desirable in fragmented cases
<emeyer> …Prior art indicated fragmented elements should be treated as if joined flush
<emeyer> …We looked at Firefox’s implementation; it seems to capture the union of the fragments before a transition is applied
<emeyer> TabAtkins: That’s per-spec
<emeyer> JakeA: Good to know Firefox is doing the right thing
<emeyer> khush: I was confused bewtween what old specs said and what Firefox does
<emeyer> TabAtkins: The capturing of dimensions is spec-compliant; other things may not be
<emeyer> JakeA: Trying to figure out View Transitions in this case
<emeyer> …Option 1: You Are Not Allowed; the captured element must not fragment, which means you skip to the end of the transition
<emeyer> …Is a constraint broken when an element fragment, or if an element ever can fragment
<emeyer> …Previous resolutions have been that if a constraint can be broken, then the treatment is as if it has been broken
<emeyer> …If you’ve given a name a none-none value, should it prevent an element from being fragmented?
<emeyer> …Option 2: View Transition only applies to the first fragment; doesn’t seem useful, but could be done
<emeyer> …Option 3: Take a bit from the element() function and remove any transform from the elemenbt, figure out union of fragment quads, reapply transform to each fragment and capture image with that transform baked in
<emeyer> …Not sure if that’s useful, but it’s a non-failing treatment
<vmpstr> q+
<emeyer> …We’re stuck here.
<astearns> ack vmpstr
<TabAtkins> q+
<emeyer> vmpstr: One of the constraints we have is that the element has a box, so we don’t check if the element can lose its box; we only care whether it has a box
<emeyer> …I prefer option 1, because as you said this adds a lot of complexity to both model and implementation
<emeyer> …I don]t think there are compelling use cases to transition a fragmented element
<emeyer> …You could work around by putting a box around the fragmented element
<khush> +1 to Vlad's comment.
<astearns> ack TabAtkins
<emeyer> TabAtkins: I also think Option 1 is probably the right idea
<dbaron> +1 to not doing something complex if there aren't actual use cases for it. (I think that most strongly says "not Option 3".)
<emeyer> …none of the options are great, but I don’t think there is a good way to do it
<emeyer> …I think making thing with view transitions not-fragmentable makes sense
<florian> q+ to disagree with making non fragmentable
<astearns> also disagree with making things non-fragmentable
<emeyer> fantasai: The problem is a people will apply this to a lot of elements and if you force things to be not-fragmentable they’ll break in print
<emeyer> …I would just not go down this path; the problem here is the combination o f fragmentation and transforms
<emeyer> …if you capture the bounding box and say this is a the view transition snapshot, that’s okay
<astearns> ack dbaron
<Zakim> dbaron, you wanted to react to TabAtkins
<emeyer> dbaron: We seem to be focused on Tab’s variant of Jake’s option 1, not the original option 1
<emeyer> astearns: I think we’re agree Tab’s variant is Right Out
<florian> q-
<astearns> ack fantasai
<khush> I'd be ok with that. Ignore the element if it fragments.
<TabAtkins> It feels a little unpredictable, but I think I'm okay with the "just don't VT if it's fragmenting"
<astearns> q+
<florian> q+
<TabAtkins> q+
<emeyer> fantasai: We could not transition when fragmented, or take the bounding box and transition the whole thing, which will interact weirdly with transforms
<emeyer> astearns: I’m fine with not transitioning view-fragmented things, but I’m concerned this is like other things we’ve punted for being too complicated and then never got back to
<fantasai> fantasai^: but would let you do simple transitions like translations or fades
<emeyer> …We already have the optoin to apply a view transition to multiple elements, so we could treat fragmented elements as if they created separate view transitions
<emeyer> JakeA: The difficulty there is how you’d address each fragment with CSS, and what do you do if the number of fragments changes
<emeyer> astearns: I wouldn’t expect being ablke to address each fragment directly
<fantasai> s/directly/directly, until we have fragment pseudos (which we have talked about)/
<emeyer> …As in regular view transitions, if you have an element that appears in one state but not the other, it doesn’t participate
<astearns> ack astearns
<astearns> ACK florian
<emeyer> florian: I agree with dbaron and fantasi
<emeyer> …Another concern: a user agent that’s interactive but paginates is not common today, but there are things like e-readers that could do that, and we don’t want a model incompatible with that
<jensimmons> q+
<emeyer> …It’s hard to make a union of things that don’t share the same coordinate space, like columns
<astearns> ack TabAtkins
<emeyer> TabAtkins: That union of non-oriented coordinate spaces is why element() stitches them together
<emeyer> …line-box-clamp could intersect with this; it’s at least fragmentation-adjacent
<emeyer> …I think option 2, transitioning the first fragment, is the best choice
<khush> q+
<emeyer> …I think it’s better to get some of the element rather than none of the element during a transition
<emeyer> astearns: If we fragment the first box, we’ll find out if people want other fragments to transition
<emeyer> JakeA: Different transitions names for different fragments would be a nice way to address that
<astearns> ack jensimmons
<emeyer> jensimmons: +1 to actually solving this and taking it seriously
<emeyer> …I can see a future where simple and elegant transitions make it so web sites use a lot of transitions to go between pages and even sites
<chrishtr> +1 to solving both
<fantasai> "using overflow fragments to solve the regions use case"
<emeyer> …We also need to figure out overflow fragments to solve the Regions use case, and make ihs and view transitions work together
<emeyer> JakeA: I would worry that because we don]t know how the regions thing will work, we might pick a solution today that clashes
<astearns> ack khush
<emeyer> khush: As an implementor, if we ignore a fragmented element, the fallback is you get a crossfade rather than a nice animation
<emeyer> …I think I prefer we either ignore the element if it’s fragmented, or we use the first fragment, I’d be okay with either
<emeyer> …Doing a union like element() does, we could try, but I really want to see how that pans out
<astearns> ack fantasai
<emeyer> fantasai: If we take option 2 and go back to Alan’s idea, we could apply the same transition to all the fragments that is being applied to the first fragment
<khush> q+
<vmpstr> q+
<emeyer> …In the future, we might be able to address each fragment individually, through something like :nth-fragment
<emeyer> …I think Alan’s proposal is the best going forward, but it’s not the easiest to implement
<emeyer> …We could start with first-fragment use, and leave the door open to individual-fragment later
<astearns> ack khush
<emeyer> khush: I think the problem is the View Transition API has a way to address fragments individually
<emeyer> fantasai: The selector would select all of them
<emeyer> JakeA: We could ask for the devloper to give view transition names to each fragment, and we could ignore in the meantime
<emeyer> fantasai: The whole element has a name, and we could use :nth-fragment
<astearns> ack vmpstr
<emeyer> ???: Would that mean the pseudo representing these elements is itself fragmented?
<JakeA> vmpstr:
<emeyer> fantasai: I’m not sure of the distrinction
<emeyer> vmpstr: So the new selector would select both, but they would be their own fragments?
<emeyer> JakeA: My understanding is if you give a name to an element, we’ll operate on the first fragment, but we can use :nth-fragment to address fragments?
<emeyer> fantasai: You can transition an <article> element, and then transition that element
<emeyer> …Each fragment gets snapshotted
<emeyer> JakeA: So what happens if you give an individual fragment a view transition name?
<emeyer> fantasai: I suppose you could do that
<emeyer> astearns: I like the idea of individually transitioning fragments, but don’t know if it’s useful to do before we have fragment pseudos
<emeyer> …That seems like the path forward
<flackr> I think the ideal would be if you have an element that has n fragments transition to another element with n fragments it pairs them up?
<emeyer> …Not transitioning fragmented elements is a way of saying we’ll address this in the future, but don’t want to have a suboptimal solution now
<emeyer> JakeA: I like it
<emeyer> astearns: Any arguments that it would be better to transition the first fragment, rather than none?
<emeyer> (silence)
<JakeA> +1
<khush> sounds good
<emeyer> astearns: Proposed resolution is that if an element if fragmented into more than one element, it does not participate in View Transitions
<emeyer> RESOLVED: if an element if fragmented into more than one element, it does not participate in View Transitions (plus note that we want to do such things in the future)
<khush> Next one is easier... I hope. :P